Many column inches have been given over to portraying the Troubled Families programme as a grand folly cooked up by headline-obsessed politicians and misguided civil servants at astronomical expense to the taxpayer.
Without taking time to digest the evaluation report commissioned by the Department for Communities & Local Government from a consortium led by Ecoryn, which sparked the furore, one could be forgiven for instantly lamenting the worst excesses of knee-jerk, top-down policy and valuable time wasted on achieving exactly nothing.
In a personal blog, one of the report’s authors, Jonathan Portes, put the boot in, blaming former prime minister David Cameron for a lack of judgement and chastising the DCLG’s Dame Louise Casey, who led the programme in the last parliament, for ignoring warnings from him and others about the programme’s flawed evidence base.
The key finding of the evaluation was that positive outcomes for families could not be directly attributed to the programme.
On the ground the programme has been mush more positive than has been indicated… It was a long-term challenge
Difficulties in collecting reliable data and the complexities of finding families to act as a comparable control group in the assessment process, the report said it was not possible to prove improvements seen by many participants were directly attributable to Troubled Families itself.
The report identified “statistically significant” evidence that families were now more confident about being able to cope in the future than comparable families.
It also found that, while levels of self-reported debt had not fallen, families were more confident in their ability to deal with their finances.
While these outcomes appear modest beside claims previously made by government that 120,000 families had been ”turned around” by the programme, they could be perceived as worthy of merit in the first phase of a complex and challenging process.
To evaluation the impact of the programme, researchers used two methodologies.
First, a “quasi-experimental” approach using data from linked national administrative datasets (outcomes for 15,482 adults and 13,946 children) and a survey of families to provide a comparison between outcomes for families on the programme and a matched comparison group.
Second, interviews were carried out with 495 families who joined the programme nine months earlier and a comparison group of 314 families who were about to start, or had just started on it.
The evaluation aimed to quantify the impact of the programme across a range of outcomes. These included employment, benefit receipt, school attendance, safeguarding and child welfare.
The report however acknowledged that for both the administrative data and survey data analyses, the Troubled Families group and the comparison group were not “automatically comparable”.
This was due to variations in the circumstances of families in the two groups, with those in the Troubled Families group having worse historical outcomes.
Dame Louise questioned the credibility of the impact evaluation during her appearance before the public accounts committee two weeks ago.
She said: “The research, after a lot of sorting out and correcting … was over a timescale which was really early in the programme.”
Dame Louise said the fact that improvement could not be directly attributed to the programme was “being used in isolation” and was “part of a bigger story”.
“No one is disputing the fact that families had problems and they now have less problems,” she added.
Association of Directors of Children’s Services president Dave Hill insisted the programme had made a “real difference”.
He told LGC councils had always believed resolving issues would take time and outcomes would not be easily measurable in the initial stages.
Mr Hill added: “On the ground [the programme] has been much more positive than has been indicated in the report.
“[The problems] were never going to be sorted out in one parliament as these were inter-generational issues – it was a long-term challenge.”
The programme’s use of a ”payment by results” and the government’s bold claim that 100% of the 120,000 families enrolled on it had been ”turned round” further bolstered critics and created an impression of councils exploiting a flawed framework to make unfounded claims for cash.
But the financial framework was not so straightforward and crucially involved councils spending their own cash (see box).
The framework offered councils up to 40% of the estimated costs of interventions under the programme, a proportion of which would be paid as an attachment fee when families entered it and the remainder when results could be shown in the form of reductions in truancy, antisocial behaviour, joblessness or reliance on benefits.
The framework gave central and local government a stake in the success of an ambitious attempt to radically reform approaches to long-standing and formidable social challenges that have impacts far beyond individual households.
The report says the “window of opportunity” created by the programme on its launch in 2012 was seized by most councils.
One of those, Hampshire CC, quickly grasped the opportunity for system change.
It established a cross-council approach beyond children’s services and adult social care to incorporate many areas of council operations.
Deputy leader Keith Mans (Con), who has led the programme since its inception, said the council also worked with the wider community to change perceptions of troubled families and convince sceptical frontline social workers and health visitors to buy into the programme.
Last year a council-commissioned report by Portsmouth University found a “positive” shift towards “increased partnership working and inter-agency co-ordination and co-operation”.
Using its own “comparison group”, the research found evidence that children supported by the Troubled Families programme were proportionately less likely to enter care or be assessed as in need by children’s services.
We were much more interested in turning families around than getting money from the government or making savings
Analysis of data on 219 families on the programme found there had been “substantial reductions” in the prevalence of families that had a persistently absent child and a child excluded from school.
It also noted: “The existing body of relevant research illustrates the difficulty in achieving positive outcomes once families face multiple adversities.”
Cllr Mans said: “To start off with, our figures [of families on the programme] were pretty low compared with other parts of the country and we were put under a bit of pressure by DCLG.
“We were much more interested in turning around families than getting money from the government or making savings – we wanted to look long term.”
Cllr Mans said the council estimated that over three years of the programme the council had saved £2.4m annually, without figures from health or housing being taken into account.
Giving evidence to the public accounts committee, DCLG permanent secretary Melanie Dawes said the payment by results framework had encouraged councils to genuinely work in a cross-agency way for the first time.
She added that data systems were being developed and improved, describing them as “a really good effort for something we had not done before”.
Ms Dawes said: “This is a good programme that has achieved some good results.
“We have adapted our approach to evaluation so we can make it stronger in the future.”
Ms Dawes’ evidence strongly suggests the government is to persist with Troubled Families, despite criticism which, at times, bordered on mockery.
The payment-by-results controversy
Central government is said to have originally planned a conventional payment-b- results model for Troubled Families, with proof of outcomes required before funding was received.
But LGC understands local government lobbied for initial central government investment to get an ambitious and complicated programme off the ground, or provide a necessary boost to existing efforts to support families on the margins, at a time when budgets were stretched.
This led to the introduction of an ‘attachment fee’, paid when identified families engaged with the programme, in a bid to create momentum at a time when funding was becoming severely limited.
The report says there is evidence that this financial framework had influenced some councils’ decisions on the number of families that were supported under the first phase.
LGC understands that in some areas families categorised as less in need were included in the scheme to draw down the lucrative attachment fee and increase the likelihood of securing the results payment.
This could be interpreted as councils unfairly manipulating the system, but the report said “these behaviours were entirely consistent with the intended purpose of the framework”.
The financial framework saw central government make available up to £4,000 per family.
This was 40% of DCLG’s estimated average cost of the intervention required to turn a family around. However, subsequent research has found the average cost to councils of delivering the programme was found to be £3,350 per family per year.
Councils were given flexibility to develop plans, with many adopting a cross-departmental ‘whole family’ approach.
A DCLG report on cost savings based on information from 67 local authorities in England found families were costing the taxpayer £26,700 per year through services used before Troubled Families.
The average reduction after the programme launch was estimated at £7,050 per family per year.