The results(2) show that in general the government's website provision has improved since 2002.
The report concludes with general commendations, but also questions 4) what the government means when it says it will make all its services available electronically by 2005, and questions how the government is quantifying, and qualifying, its claims to have two-thirds of all services 'e-enabled' already.
The 270-page report was commissioned by Interactive Bureau - the website strategy and design agency; and the research was carried out by Porter Research.
The report focuses on 31 government and quasi-governmental sites. The choice of sites - out of the 1,000-plus - was made principally on the grounds that these are the sites we would expect to be the best ??? 'the flagships' - of the drive to e-government.
Fifteen of the sites are of the principal public-facing ministries, including the prime minister and deputy prime minister's own sites - and the others are the sites of key public-facing 'citizen-focused' organisations ??? the ones where it is most likely the public is going to want to interact with its government.
The criteria used to assess all the sites are based on accepted best practice and the Office of the e-Envoy's own 'Guidelines for UK Government Web sites ??? Illustrated Handbook for Web management teams ??? May 2002'. (Further details in Notes)
1) The Good News
The report found:
* There has been a marked improvement in the government's website prov ision since last year
* The top six sites have come quite close to offering consistent and useful navigation, content, presentation, and design through their online presence
* The evidence is that government guidelines for web teams, which were published in May 2002, are beginning to have an effect, and that they are being used more widely and purposefully
* Levels of technical compliance appear to be increasing year on year
The report says that 'the overall quality of government websites has increased in the past year. The Guidelines, which were put in place by the Office of the e-Envoy in May 2002, to assist web teams in the implementation of their organisation's websites, have begun to take effect. Levels of technical compliance and performance have increased, we are seeing more considered designs, particularly when redesigns have been implemented, and useful navigational provision on government sites is extending into their lower levels. We have also noted that to a degree, and particularly on redesigned sites, thetone of the sites' content is beginning to be considered ??? and written for the web ??? and in places consistent methods of presenting information are being put in place'.
Adrian Porter, the report's author comments, 'we are really pleased to see a general improvement, it is evident that major steps have been taken in the right direction'. He mentions that he 'found it interesting that the sites we reviewed last year are generally better than the ones we added into the mix this year, we wondered whether we might have played some small part in this?'
Medwyn Jones, who was responsible for the technical assessment of the sites, commented ??? 'we noted that the sites, which had been redesigned this year, had a higher average rating than all of the others, suggesting that when considering new developments, higher standards are being put into practice.'
Mr Jones also commented that he was 'generally impressed with the technical quality of the Web sites and how they matched up against the criteria'. He went on to say that 'where they (the Web sites) fell short, we did not in most cases anticipate major problems in addressing the issues which he saw as shortcomings'.
2) The Results (For full table, criteria and criteria-specific results, see Notes)
The report found:
The best site examined this year was the Equal Opportunities Commission (80.75%)
The worst was found to be that of the Immigration and Nationality Directorate (46.45%).
Second place overall was awarded to the Police Complaints Authority (80.3%), followed by The Department of Health (75.6%), The Department for Culture Media and Sport (75.1%) and The Home Office (74.4%).
The second worst site was that of the Ministry of Defence (48.1%), followed by Her Majesty's Treasury (48.7%), The General Register Office (51.9%) and The Department for International Development (53.35%).
The report says, 'We suggested last year, and still maintain this year, that any site that scored 65% or less of the available mark in the analysis was in need of immediate attention in one area or another. Last year this amounted to three-quarters of the 20 sites, this year of the 31 sites considered, 58% fell below the 'cut off point'.
'Notable among those that improved were the DVLA (40% to 69.1%) and the Department for Culture Media & Sport (48.5% to 75.1%) ??? both redesigned sites, and DEFRA (46.25% to 68.7% ), which has consolidated its provision since its redesign, which occurred in the midst of last year's assessment. Of note also is the improvement made by the prime minister's website as a result of a redesign (40.75% to 64.1%), although the site is still below the 65% mark.
'On the other side of the coin are six sites, which have a lower rating than last year, for three of these the dip is negligible, but for Her Majesty's Treasury (60% to 46.7%), the Department for Education & Skills, last year's winner (78.5% to 71.15%), and the National Health Service Direct (63% to 57.5%) the situation is more serious. Each of these sites has in one way or another lost the consistency of approach for which they were commended last year, becoming somewhat sloppy in the maintenance of their offerings'.
'The Equal Opportunities Commission site ranked top of all the others because it is a considered and functional offering, which explains in an easy tone of voice, the purpose of the commission, the legislation regarding its remit, and how this applies to individuals, and companies'.
'The Immigration & Nationality Directorate fared badly in all but one area of the assessment. Given the high profile of immigration issues currently, it might well be that the most sensible option for the IND is to redevelop its site in order to reflect the government's current positioning with regard to the subject of immigration, and its commitment to standards of online provision'.
Mr Porter comments that 'although the trend is generally upwards, the fact that nearly 60% of the sites in our view are in need of some immediate attention, suggests that there are still a huge number of the other thousand, or so, Government sites out there that are in need of similar attention' He goes on to say that 'you must also remember that these are the government's 'flagship' sites, so we would expect them to be the exemplars'.
3) The Bad News
The report found:
Nearly 60% of 31 'Flagship' government websites surveyed in depth in the second independent report are in need of immediate attention in one area or another
Nearly 50% of all the sites surveyed failed to either provide a contact point, or reply in a satisfactory manner, to a simple email enquiry
Too often the sites assumed a level of knowledge, which in our view is unlikely to be evident in the 'average' citizen ??? the Office of the Deputy Prime Minister being one of the primary culprits
The overriding problems:
No real attempt is being made to present existing documentation in a manner which shows an appreciation of the Web as a different medium to print
No great increase was evident in the number of transactions that can be performed with government websites online
No evidence of a consistent approach being implemented which facilitates movement between government sites, either to information on satellite sites within the departmental remit, or other related Department sites ??? joining up services for a consistent user experience
With regard to the poor response to email enquiries the report says 'eleven sites did not score any marks in this category because they either did not provide a general email enquiry facility, did not supply the requested information, or because they insisted on us providing personal details, or case reference numbers, in order for them to generate a response'.
The report asks 'where do departments such as that of the deputy prime minister expect members of the public to direct queries relating to the workings of their departments ??? why do they not provide a general email enquiry point? And secondly, given the very benign nature of our enquiry why would Departments such as The Inland Revenue, Customs and Excise and Immigration & Nationality Directorate, need personal details and/or case numbers to be provided in order to generate a response?'
Mr Porter comments that 'this seems to represent a backward step. While we appreciate the resource implications the advent of email must have had on the government, surely this is a significant area where the government can connect with the people, and the people can feel that the government is listening to them'.
Porter cites the Foreign Office as a particular example: 'We sent an email to the FO from its website, asking for information, and they replied with an email containing nothing more than a link to their website home page. We found this amusing, but it was hardly courteous'.
The report also says that 'It is no surprise to note that the sites which rated highest in this c ategory were also the ones who did best overall, suggesting that those sites, which take the design, navigation and performance of their Web sites seriously are also those that have a commitment to this interactive elements of their service provision'.
The survey sees that the methods employed in the government's push towards the 2005 deadline as detrimental to the quality of much of its online provision, and in particular its publications.
It quotes UK Online's Annual Report 2002, which claimed that the government had 'transformed the provision of information. Instead of the mass of information only available on paper, information is now easily and freely available on the internet 24 hours a day, seven days a week'.
The survey goes on to say that 'In 2003 a similar claim has been made, which we do not dispute. However, is it OK for the government to tick the box for information provision by transforming a 'mass of information only available on paper', into a mass of electronic information, which is only suitable to be printed out on paper?'
Regarding online transactions with the government, the report fears that not much progress has been made saying that 'The evidence in this report is that very little progress has been made since last year in providing publicly available online transaction services. It is perhaps poignant to note that the Driving Standards Authority Web site, which is featured in the OeE 2003 Annual report as a success regarding its online driving theory test bookings system, was rated as the second worst citizen-focused site in this survey, due to its lack of; design consistency, browser compatibility and accessibility'.
Porter comments, 'We envisaged progress towards a time when we wouldn't have to go to the post office to get a form to send to the government, and were hoping to see some movement towards that this year. However, it would appear that this is a long way off. You might be able to download a form, but having the majority of government forms available for online completion and submission does not seem to be in the remit of the drive to 2005'
When commenting on the joining up of government services and how this relates to its Web sites the report says 'One of our major criticisms in this year's report, and last year's, is the government's inability to impose any kind of consistency regarding the way visitors move around its closely inter-related and dependent websites. No conventions, however basic, have been put in place, with each department providing a different solution. This we see as a major barrier towards joining up services and providing citizens with a consistent user-experience'.
Porter explains 'Visitors to government websites are rarely told where a link to 'related information' will take them, or what will happen when they click on it. A simple example of this is that many departmental websites offer publications for sale. Having found a particular document on the department's site visitors are invariably provided with a link to The Stationary Office website home page, from which they must begin their search for the document all over again'.
4) The Questions
In the light of the publication this week of the Office of the e-Envoy's, Annual Report 2003, the Interactive Bureau commissioned report of these 31 sites has added poignancy.
There are two aspects of the OeE report, which grabbed the attention of this report's author. The first is the claim by government that 'two-thirds of central government services have been e-enabled'
The report points out that in 2002 the government said that '54% of services are now available electronically'. It questions the government's definition of 'services', and how figures of 54 per cent, or two thirds, could be arrived at?
The second is the announcement of a new pilot scheme ??? a portal ??? aimed at providing a 'single point for delivery of government services' with 'more user-friendly services, which are based firmly around customer needs', presenting 'information in a consistent and useable way', and which will 'enable common customers of different departments to access services from a single location'.
Porter says 'we are not questioning the figures per se, but we do wonder what they mean. The evidence is that in transactional terms the government is now going to concentrate on key service delivery from a centrally controlled portal. Considering that this is a pilot scheme, and that 2005 is fast approaching, does this mean that 100% electronic service delivery was never meant to include any online transaction? Given the two-thirds claim, and the findings of our report, it would seem not'
Porter also asks 'whether delivering 100% of services electronically actually means nothing more than putting all of a Department's publications into an electronic format?' It is, he says, 'a quality over quantity issue', and points to the report, which found 'that pdfs are overwhelmingly the preferred delivery medium, presumably because it is quicker and cheaper to produce them than a HTML page'.
The report found that 'in most cases, where a HTML document has been produced, little, if anything, has been done to make its presentation suitable for the Web environment', and it goes on to ask if it is 'acceptable for the big Departments to provide all of their publications in pdf format? And even if they manage to get all of their documents onto their Web sites by 2005, which the evidence suggests will be a Herculean task, does this mean that they can 'tick the box'?
The report concludes by saying that the single point of delivery portal is an 'eminently sensible approach', and commends the OeE for 'taking the bull by the horns'. However, it also asks a series of questions, which attempt to get to the heart of the government's thinking:
Has the government admitted that the future for electronic service delivery cannot be left in the hands of the departments?
Will all future transactions with government be conducted through a centrally controlled portal?
Is prior knowledge of this scheme the reason we have seen so few Departments adding transactional capabilities to their websites?
In terms of 100% service delivery will the departments now only have to ensure that they have all their information online by 2005?
What of the departmental websites now, will they be solely focused on their professional audiences, with the citizen audience catered for elsewhere?
Porter finishes off by saying 'overall this report shows that the government's web site provision is improving. We wonder however, whether in terms of 2005 the goal posts might have moved. In truth we are not overly concerned if they have, since the plans that are being put in place appear basically sound. However, we would like to see the government come clean with what it considers to be its realistically achievable aims for 2005, based on a qualitative rather than quantative approach'.
'The Second Annual Report Into Key Government Web Sites' ??? is published by Interactive Bureau, London. Price £425 + VAT. Copies can be obtained from Interactive Bureau, 9 White Lion Street, London N1 9PD. Tel: 020 7278 4352.
Or email: email@example.com ??? or order online at: www.iablondon.com
The sites were analysed between 1 and 30 November 2003.
Table of Results:
Department / Organisation
1st Equal Opportunities Commission80.75
2nd Police Complaints Authority80.3
3rd Department of Health75.6
4th Culture Media & Sport75.1
5th The Home Office74.4
6th Commission for Racial Equality73.75
7th Office of the Rail Regulator73.4
< p/="">8th Department for Education & Skills71.15
9th Food Standards Agency70.05
10th Driver & Vehicle Licensing Agency69.1
12th Office of Fair Trading67.2
13th UK Passport Agency66.8
14th Prime Minister's Office ??? No.1064.1
15th Foreign & Commonwealth Office63.3
16th Her Majesty's Customs & Excise61.55
=17th Department of Trade & Industry60.9
=17th Department for Transport60.9
19th Department for Constitutional Affairs60.75
20th Welsh Office60
21th Child Support Agency59.75
22nd Inland Revenue59.45
23rd Job centre plus58.9
24th National Health Service Direct57.5
25th Office of the Deputy Prime Minister57.05
26th Driving Standards Agency53.9
27th Department for International Development53.35
28 General Register Office51.9
30 Ministry of Defence48.1
The criteria we used to assess all the sites are based on accepted best practice and the Office of the e-Envoy's own 'Guidelines for UK Government Web sites ??? Illustrated Handbook for Web management teams ??? May 2002', which was developed by The Government Internet Forum.
Broadly speaking the criteria were put in place to test the quality of design, navigation, interaction and performance of the sites. Wherever possible we tried to keep subjective assessment out of the equation.
For the Citizen-focussed sites we added an assessment of how well they are facilitating the public's quest for information, interaction and advice, since the provision of such facilities must be seen as the primary 'raison d'être' of these sites, and one of the principal aims of providing inclusive electronic government.
We believe that the criteria we used are correct. We did not expect too much of these sites, merely a reasonable standard, based on common sense and good practice.
Some further findings by Category
In general, it is noticeable that those sites, which rated highly in this report, did so consistently across all the criteria, and those that rated poorly, also did so consistently. This seems to show that those organisations, which are following good practice are doing so across the range of disciplines necessary to produce and maintain a good website, and those that are not, are generally either unaware of what is good practice, or do not have the necessary skills, to implement it.
It would appear that some government departments and organisations have a superior understanding of their website audiences' requirements and how their resources and information is relevant to those audiences.
We commented last year that none of the sites looked at could claim to be consistent throughout its entirety with regard to either its navigational, content, presentational, service or design provision. However this year we w ould suggest that the top six sites have come close to this nirvana.
The provision of Standard Elements from the Home Pages of government sites has improved since last year with the average mark rising from just over 11% to just under13.5%.
The elements, which were least likely to appear on these sites, were the glossary (4 out of 31), management structure (7 out of 31), FAQs (11 out of 31), and Site Help (13 out of 31). In the case of management structure, most sites provided this information in their 'About Us' sections, which we would suggest is not an unreasonable approach, although the government guidelines suggest that there should be a direct link to this information from the home page of sites. When considering the others, we find it rather remiss that these sites cannot offer visitors these simple elements, which are designed to assist them and which to an extent, make the sites more 'approachable' for less experienced users.
With regard to our email enquiries a staggering 14 sites failed to register any marks in the category either because they did not respond at all (two sites), failed to respond within five days (one site), or because they either did not provide a general email enquiry facility, did not supply the requested information, or insisted on us providing personal details, or case reference numbers, in order for them to generate a response. It is no surprise to note that the sites which rated highest in this category were also the ones who did best overall, suggesting that those sites, which take the design, navigation and performance of their Web sites seriously are also those that have a commitment to the interactive elements of their service provision.
Search engine registration does not appear to be a problem for any of these sites, although some of the sites could pay a little more attention to the results they are getting from some of the search engines.
With regard to design, we noticed that there is a general improvement in the standard of design, in particular in the depth to which design conventions and templates are being used on the sites. Last year we heavily criticised the lack of design consistency of the majority of the sites we looked at, however this year there has been a definite improvement in the level of consistency, with design parameters being implemented to a much deeper level of the sites. However, there are still some design horrors, which pay no heed to any form of design convention, or best practice.
Navigationally we saw a similar improvement, with organisations being more capable of providing consistent navigation to deeper levels of their sites. There are still many sites, which have problems presenting long documents on a single page, with very few being able to offer consistent internal navigation of the same, and few sites have resolved the problem of how to present a link to an external Web site, and what should happen when clicking on the link.
With regard to offering content and online functionality/services, which is audience specific and relevant to citizens, and professionals ??? 'e-government' ??? it would appear that there is still a long way to go, but that the gap is narrowing. If the Government is prepared to accede that there is a difference between the requirements of its citizen and professional audiences then surely more could be done to present information to these two groups in a way that is understandable, particularly to the citizen. Too often the sites assumed a level of knowledge, which in our view is unlikely to be evident in the 'average' citizen ??? the ODPM being one of the primary culprits.
Many sites provide some kind of online service by offering their forms and publications in pdf format for download from their sites. However, as with last year, we found that often:
The most important/popular forms were often not provided
The content of specifically designed citizen-focused offline publications was only available as a pdf download and no attempt had been made to make this very useful information available from sites in HTML format. So much so, that many sites did not bother to provide any citizen-focused information in any other format.
When considering the performance of the sites we noticed that improvements have been made, suggesting that attention is beginning to be paid to the government's Guidelines. However, two sites, The Treasury and Job Centre Plus actively discriminated against certain browser users.
With regard to accessibility, we were encouraged by what we found, since it would appear that most sites now take this as a serious issue. Interestingly we noted that the sites which, provide text only versions (8 out of 31), are likely to be some of the least accessible, suggesting that the trend towards providing one accessible site is preferable, and of course this is more in keeping with the government's own view.
As to the speed of loading of the Home Pages, in line with last year, we found that two thirds of the sites rated as either average, slow or very slow by NetMechanic. This fact suggests that few have taken on board the government's guideline, which states that the total file size of website home pages should not exceed 40kb.