The Social Services Inspectorate's 11th annual report singled out this issue as an important and significant obstacle in the way of improvement.
Chief inspector Denise Platt said key factors affecting recruitment and retention rates were poor image, poor human resource management, low pay and poor job prospects.
The most severe problems are in London and the south-east. According to inspection evidence in children's services, agency staff were used mainly in this area.
'This is a unique opportunity to transform the status, image and position of social workers,' she said.
Owen Davies, Unison's national social services officer, said: 'The salaries being paid to all staff in social care are too low. The government has announced a 6% a year increase in funding, but it says this must not be spent on salaries.
'This is a service that depends overwhelmingly on labour. It has 80% labour costs. The idea that you can improve services without increasing labour costs is ridiculous.'