The number of attacks on asylum seekers has led to families asking to be moved out of Scotland, according to the Scottish Refugee Council.
'The exact number of people wanting to leave is unclear, but we have recently seen a huge number of racist attacks and the next step is that people ask to be moved out,' the spokesman said.
The government's dispersal plan was recently criticised by the Welsh Assembly after no asylum seekers arrived in Wales, even though a number of houses had been earmarked for new arrivals.
The criticism comes at a time when the Conservatives have been under attack for their proposals for dealing with asylum seekers.
The Tories want all new arrivals housed in detention centres - preventing them from getting lost in the wider community.
But the policy has been criticised as 'wrong' by Conservative peer Lord Taylor of Warwick. Where such centres have already been planned or built by the current government there has been local opposition.
In Kent a government planning application for a detention centre was turned down by Conservative Ashford BC. The council went to court over the plans, but claims it was because the plans were unsustainable, not because it opposed the policy.
Kent CC leader Sandy Bruce-Lockhart (Con) said the council would accept a reception centre as long as other councils around the country followed suit.
He said: 'We support detention centres, in fact they would reduce the number of people heading towards the UK across continental Europe to seek asylum.'