West Wiltshire DC paid £45,000 to have the house in Upton Scudamore and its adjoining property upgraded.
After seeing the new house, Philip Smith and his common-law wife, Gillian Carter, said the garden was too small and they needed the garden from the second house too.
The family has a history of disputes with West Wiltshire over keeping livestock and cars in the garden. It is understood that the council intends to keep the second garden free to act as a buffer for other houses nearby.
Mr Smith risks losing the new house unless he changes his mind before the work is finished in two weeks.