He also called for a government review into the alternatives, saying that little had changed five years after he wrote a damning report on the conditions at HM Gateside Prison, at Greenock, which houses asylum seekers.
He said: 'How long must Scotland and its prison service go on bearing this ugly stain on its reputation?'
He called on the government to consider the alternatives to prison for the growing numbers of refugees who are held in custody in case they abscond if their asylum applications are rejected.
'Prison is not the place for these unfortunate individuals. With their numbers set to increase, surely it is time they were got into a proper detention centre as happens all over England and not sent to languish in jail,' said Mr Fairweather.
Figures from last week show there were 44 asylum seekers in Scottish prisons although numbers are expected to at least double under new legislation.
The previous Conservative government pledged to consider building a separate detention centre for refugees although at the time the small numbers made the cost difficult to justify. But it is believed the Labour administration has not investigated this further.
Mr Fairweather is planning another inspection at Gateside later this year, and added: 'I would welcome a review and maybe now is the time to carry it out since the slightly better arrangements in Longriggend have come to an end.'
He is concerned that asylum seekers are still being held in a volatile hall where a high number of suicides had taken place.
Critics say asylum seekers do not need to be in a prison with round-the-clock guards and high security fences and should be kept instead in an open detention centre. Alternatives, like Dungavel low security unit in Lanarkshire which has just closed, have been suggested.
A spokesman for the home office was unable to say if the government planned to investigate alternatives to prison in Scotland. He said: 'Although we are expanding our detention centres in England there are no proposals yet for a new detention centre in Scotland.'