Mohamed Danesh claimed he put up with so much hatred he had no choice but to bring his family to London in the hope of finding a more pleasant place to live.
But after reviewing the case, the council argued that because Mr Danesh had links to South Wales, the only local authority which could house him was Swansea City Council.
At the Shoreditch County Court in March this year, the Royal Borough's stance was overruled by Judge Cotran, who said its officers should have taken into account Mr Danesh's fear of violence rather than the risk of him actually being harmed.
However Lord Justice Neuberger, sitting with Lord Justice Mummery and Lord Justice Jacob at the Court of Appeal, said Judge Cotran was 'plainly wrong' and his decision 'wholly misconceived'.
Lord Justice Neuberger added that, under a strict test in the Housing Act 1996, there has to be a real danger of someone suffering a physical assault because of their specific link to an area for them to be able to win their case.
The test is commonly used in cases of domestic violence, where sending someone back to their old neighbourhood may lead to them being attacked again by a violent spouse or partner.
The judge added that, although Mr Danesh had been attacked twice, neither appeared to be related to his status as an asylum seeker and were 'random' incidents that could have occurred anywhere in the country.
The court heard that after arriving in the UK in August 1993, Mr Danesh and his family went at first to Barnsley and then came to Townhill in Swansea.
Mr Danesh says he has been asked if he is member of Al Qaeda and someone on a bus refused to sit next to him because of his colour.
In January 2004 he was pushed from behind outside Townhill Community Centre and was taken to hospital because of his injuries, and in June 2004 was mugged in Swansea town centre.