National Housing Federation Chief Executive David Orr issued the stark warning in Birmingham when he addressed the annual conference of the Federation, which represents England’s housing associations.
Mr Orr said that unless radical action is taken immediately only around 1.6 million homes will be delivered by 2020.
Based on those figures, the three million target figure may not be reached until 2029.
In a keynote address, Mr Orr said: “The Prime Minister was both brave and right to put the dire need for new housing centre-stage, and make it a national priority, when he took office last summer.
“He is rightly aware that too many people are living in cramped and unsuitable conditions and that more than five million people will soon be on waiting lists for social housing.
“However, with the global credit crunch worsening, and conditions getting tougher for all house builders, it is time to recognise that the very commendable 2020 target is now almost impossible.”
While the Government wants 240,000 to be built by all sectors every year from 2016, experts believe that this year the total number of new homes produced this year could be as low as 100,000. Last year, the figure was 167,577.