Now, he wants people all over the country to take their green spaces back, so both Britain's gardens and communities can bloom and flourish together…
When Jim Buttress first started gardening at school, after being asked out of class by the teacher, he never thought it would lead to him being one of the UK's leading experts on horticulture for decade after decade, and a starring role on the judging panel for the 'greatest flower show on Earth' at Chelsea.
"I can"t say it was written in the stars," says the 71-year-old London-born gardener when describing his passion for the great outdoors,
"I can't say I wasn't going to be fit for anything else, I was just fascinated by being outside, getting dirty, and then seeing a packet of seeds that I had sown actually germinate, and I'd done it all myself."
From there Jim went on to emulate his own gardening heroes, men like Percy Thrower and Jack Hamilton who "didn't have scripts, they just got out there and did it."
This idea of active gardening is something Jim is keen to encourage – and he sees the future of Britain"s green and pleasant spaces as being in the hands of the people, rather than the politicians.
"Because of the demise of a lot of councils, and cutbacks, the community are taking it on themselves," he explains.
"I've always been drawn towards community involvement. It says so much for the area, and the people, and it says so much that people who live next door to one another for ten years, never talked, got on with their own life, and then they find that the two of them are picking up litter on the local common.
"The community are finding waste bits of land that no-one else wants to know about, and they"re creating these community gardens. It"s crazy, isn't it?"
Despite appearing on BBC Two's The Big Allotment Challenge, Jim places more faith in a sharing of knowledge throughout the gardening community, rather than to make it a competition.
"At the end of the day, gardening is an extremely friendly profession," he beams.
"If you've got a real problem, there will be loads of people on hand to take a look at it and help you out."
Jim's dedication to his profession earned him a much-prized, and highly-coveted, Victoria Medal of Honour, alongside 62 other legendary names in the world of horticulture.
It is his biggest achievement to date in his expansive career, but he hopes that one day he"ll be able to fulfil another life-long dream.
"I would love to set up my own school, with my own training and help," he says.
"I do a lot with other people, but if I could say that Jim Buttress set up his own School of Horticulture for all those people that may have slipped off the wagon somewhere or another, and it will keep on helping them, that would be a marvellous thing."
The People's Gardener by Jim Buttress is out now via Sidgwick & Jackson, RRP £16.99