Those behind a community garden in the field are worried for the future of the project if the council goes ahead with the plans.