Mr Elvidge will replace Muir Russell, who is leaving to take up a post as principal and vice chancellor of the University of Glasgow.
It was an open competition involving candidates from both public and private sectors. The recommendation has been agreed by the first minister and by the prime minister.
Mr Elvidge said:
'I will be proud to be the permanent secretary of the Scottish Executive. The people who work in the organisation have achieved a lot in the first four years of devolution and morale is high. I look forward to leading them in responding successfully to the aspirations of the people of Scotland in the years that lie ahead.'
Sir Muir, who has been the top civil servant in Scotland for more than five years and who announced his intention to join the university last October, commented that he had known and worked with John Elvidge for over 30 years, and had every confidence that he would be an effective and successful leader of the executive.
First minister Jack McConnell said:
'I congratulate John Elvidge on his appointment. He takes over at a time when the Scottish Executive faces the big challenge of delivering our plans for the next four years working closely with our stakeholders. I know that his skills and experience equip him well to lead the organisation in supporting ministers to achieve that. I look forward to working with him.'
John Elvidge was educated at Oxford University (BA English). He joined the Scottish Office in 1973, working first in education and subsequently in most other departments.
He was seconded to Scottish Homes (1988-89) and the Cabinet Offi ce (1998-99), where he worked on co-ordination of the UK government's legislative programme and social policies. He was head of the Executive's education department from May 1999 before being appointed head of FCSD in March 2002.