Report on Departmental Open Data Commitments and adherence to Public Data Principles for the period between April and June 2014.

This is the eighthreport on departmental open data commitments and adherence to public data principles covering the first quarter of 2014/15.

Open data is not just about releasing datasets. The Public Data Principles (Annex A) are designed to encourage departments and government organisations to look at their processes and make improvements to enable the release of data in open and reusable formats.

During this quartersome of the highlights of the reported achievements are:

  1. Department for Communities and Local Government (DCLG) released the first set in a series of new Experimental Statistics on Energy Performance of Buildings Certificates as well as the first set of new statistics on the Help-to-Buy scheme, the latter in a fully open data format (measuring 5 stars). Both are available at: and
  1. Data files have been published from the Government's Property and Land asset database (Cabinet Office) containing details of location, tenure and other key attributes for each asset, including details about the buildings, any vacant space and occupiers. Data can be accessed at:
  1. Department for Transport (DfT) agencies published the complete record of bus registrations for Great Britain and the Rail Routing Guide that shows the routes over which rail fares are valid. National Rail Enquiries made access to the Application Programming Interface (API) for the Darwin service of real-time train running available for free, whereas previously this had been a commercially licensed product.
  1. Department for Education (DfE) published Destination Measures, showing progression of students into further education. This included a statistical first release, tables and underlying data. The education destinations of the 2010/11 Key Stage 4 cohort were also published on the performance tables’ website at:
  1. Within the Department for Business, Innovation and Skills (BIS), the Science and Technology and Facilities Council and the Big Innovation Centre organised a “Big Data - Open/free data for use by business” event, which brought together the Public Data Group Members and Research Councils to showcase the vast range of data that they make available.

Departmental Progress

Datasets

Corporate releases – Departmental Returns

Corporate releases support progress against Public Data Principles 2, 3, 4, 5, 6, 8, 9 and 11.

A list and explanation of the corporate commitments can be found at Annex A:

1.Nine departments were fully compliant compared to eight in this quarter. The fully compliant departments were: BIS, DfT, DCLG, Department for Energy and Climate Change (DECC), DfE, Home Office, Department for Work and Pensions (DWP) and HM Revenue and Customs (HMRC).

2.Quarterly Data Summaries are no longer reported in this statement, and are now published on behalf of departments by the Cabinet Office. The data can be found at:

3.The table below shows the progress departments have made on a range of data they committed to publish by the end of the reporting quarter. The measurement of ‘met’ or ‘not met’ denotes whether the expected publications were available at the end of the reporting quarter. A measurement of ‘not met’ merely indicates that the expected data was not publicly available at the end of June 2014 – the data may have been published in the time between the end of June and the publication date of this report. An explanation of each of the reporting categories can be found at Annex B.

4.Due to the way the data is collected, publication of gifts and hospitality and travel are published a quarter behind other datasets.

1

Expected period of data available at the end of reporting quarter / BIS / CO / DCLG / DECC / DEFRA / DCMS / DfiD / DfE / DfT / DH / DWP / FCO / HMRC / HMT / HO / MOD / MoJ
Spend data / May 2014 / Met / Not met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Not met
GPC data / April 2014 / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Not met
Organogram / March 2014 / Met / Met / Met / Met / Not met / Met / Met / Met / Met / Not met / Met / Met / Met / Met / Met / Met / Met
Exceptions to Moratoria (ICT, Property, Recruitment, Marketing & Consultancy) / Q4 2013/14 / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met
Workplace Management Information / May 2014 / Met / Met / Met / Met / Not met / Met / Met / Met / Met / Met / Met / Met / Met / Not met / Met / Met / Met
Senior Officials Travel Report / Q3 2013/14 / Met / Met / Met / Met / Met / Not met / Not met / Met / Met / Met / Met / N/A / Met / Met / Met / Not met / Met
Ministers’gifts, hospitality received, overseas travel, meetings with external organisations including media executives / Q3 2013/14 / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / N/A / Met / Met / Met / Met
Special Advisers’gifts and hospitality and meetings with media executives / Q3 2013/14 / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / N/A / Met / Met / Met / Met
Senior Officials’expenses, travel and hospitality / Q3 2013/14 / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Not met / Met
Permanent Secretaries’meetings with external organisations including media executives / Q3 2013/14 / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met
Contract Finder / Real time / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Not
met / Met / Met / Met / Met / Met
Real Time Energy User / Real time / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met / Met

1

Reasons for non-publication

  1. Cabinet Office reported thatpublication of spend data was delayed awaiting approval.
  1. DfT reported a delay to consultation on the release of Vehicle Registration details. It should now be issued in late Q3 or early Q4 2014/15.
  1. The Senior Officials Travel Reportincludes bookings made through central booking systems and follows a Ministerial commitment given last summer to publish this data across government. The Cabinet Office is working with the departments that did not meet this commitment to clarify the requirements of this new dataset, and to ensure timely delivery for the next return.
  1. A new requirement for the Foreign and Commonwealth Office (FCO) to publish contracts with overseas suppliers has led to a significant increase in workload. FCO is currently reviewing and updating existing work processes to accommodate this requirement.
  1. HM Treasury (HMT) reported that the publication of workforce management data was delayed and they are working to provide the return shortly.
  1. Department for Environment, Food and Rural Affairs(Defra) reported that there remain barriers to moving to a fully open by default position, including intellectual property, derived data issues with the combination of data from multiple sources, technical issues and resource availability. Defra is working to document the barriers and actively look to remove them where possible. This includes identifying priority datasets and then looking to identify resources to remove technical or other barriers.

Published Commitments

Published Commitments supports progress against Public Data Principles 1, 2, 3, 4, 5, 8, 9, 10, 11 and 12.

  1. DfE published the following datasets:

●Pupil Absence Statistics (open data format):

●Neighbourhood Statistics - Small Area Pupil Absence and Attainment by Pupil Characteristics in England 2012 to 2013:

●Cross Border Movement LA Tables and Matrices:

●Secondary and Primary School Applications and Offers 2014:

●National School Workforce April 2014 (England only):

●Schools in England:

●Outcomes for children looked after by local authorities; School workforce in England – November 2013

●Not in Education, Employment, or Training (NEET) Statistics Quarterly Brief January – March 2014; Children accommodated in secure children's homes – 31 March 2014; Schools pupils and their characteristics – January 2014; Participation in education, training and employment, age 16 to 18 – end 2013; and provision for children under 5 years of age – January 2014.

  1. BIS reported thatin December 2013, as part of the public consultation on tackling abuse in zero hour contracts, they released analysis on the extent of use of zero hour contracts in Great Britain. Further data was published in August this year, on estimates of numbers on zero hour contracts, as part of the government response to the consultation.
  1. Following a public consultation exercise, DECCpublished two sets of anonymised data from the National Energy Efficiency Dataset – Public Use File and End User Licence Fileavailable at:
  1. DWPreported that Stat-Xplore, the department’s online tabulation tool, now has over 2000 registered users – the number of registered users has doubled in the last six months. Over 21,000 bespoke tables across all the databases in Stat-Xplore have now been constructed by users. Stat-Xplore data releases during the period April to June 2014 include registration of National Insurance Numbers to adult overseas nationals; Personal Independence Payment, Jobseeker’s Allowance/Employment and Support Allowance (ESA) Sanctions data; Housing Benefit data; and Benefit Cap data. More information is available at In addition, the department released ESAand Work Capability Assessments data on data.gov.uk.
  1. DWPalso reported that Universal Credit statistics are being published monthly on the same day as the Office for National Statistics’ (ONS) Labour Market statistics. The statistics cover total caseload and starts on Universal Credit, with breakdowns by Jobcentre Plus, Local Authority, Parliamentary constituency, gender, age band and duration.
  1. As part of the transparency policy for publication of major projects data the Major Projects Authority within Cabinet Office has published its second annual report. It includes a set of combined data of the Government Major Projects Portfolio, progress and future priorities. See for more information.
  1. FCO has made significant progress in releasing project documentation, which can be linked to the Official Development Assistance data as part of its work towards international aid transparency index standards.
  1. HMRC’s releases includedthe HMRC Data Catalogue, a revised inventory of the data the Department holds.
  1. Home Office published data on UK entry clearance visas issued overseas for the period 2005-2013and Disclosure and Barring Service list of serious offences, i.e. those that will never be filtered from a criminal record check.
  1. In April, DCLG released the first in a series of new Experimental Statistics on Energy Performance of Buildings Certificates. In doing this DCLG released the data in fully open, accessible 5-star formats via its OpenDataCommunities service.
  1. In June, DCLG released the first set of new statistics on the Help-to-Buy scheme, again as fully open 5-star data via OpenDataCommunities, with a supporting application developed in-house.Also in June, DCLG commenced work on unlocking statistics about Local Government Financeand Council Tax, completing initial conversion of outputs at Local Authority and Parish Council level during July.

Arm’s Length Bodies (ALB) Progress

ALB reporting supports progress against Public Data Principles 1, 2, 3, 4, 5, 8, 9, 10, 11, 12

  1. Department of Health (DH) reported that its ALBs continue to engage with the transparency requirements in relation to Organogram and Exceptions to Moratoria (recruitment and procurement).
  1. DHhas been working with its ALBs to improve compliance with requirements on spend data, Government Procurement Card data and business expenses where most ALBs publish the data but do not then also make available through data.gov.uk. The current position is that seven of the department’s ALBs are fully compliant with the measures.
  1. DECC reported that:

●The Coal Authoritypublished the Coal Mining Reports dataset, spend data and the staff organogram on data.gov.uk. The Authority has also started a programme to identify additional datasets for inclusion in the National Information Infrastructure (NII).

●Committee on Climate Changepublished all required transparency data.

●The Nuclear Decommissioning Authority published monthly expenditure above £25,000;staff organogram; hospitality data; and a register of directors’ interests on their website. It also published contracts on Contracts Finder, again available on the website. The Authority is currently working with Cabinet Office and DECC on publishing on data.gov.uk.

  1. DWPhas been working with its ALBs to ensure their Annual Reports and Accounts are aware of, and are taking action to comply with, the requirement for all government Command and House of Commons Papers to be published on gov.uk.
  1. The items below highlight some of the other work being undertaken by non-departmental public bodies within the DWP:

●Health and Safety Executive has recently published statistics on fatal injuries to workers.

●The Industrial Injuries Advisory Council’s website was successfully transitioned to gov.uk. This is where the Council regularly publish itsdetailed scientific advice to Ministers in the form of Command and Position Papers.

●National Employment Savings Trust published an annual report and accounts for both the Corporation and Scheme.

  1. Cabinet Office reported that the Crown Commercial Service (CCS) was launched as an Executive Agency of Cabinet Office on 1 April 2014. CCS will be publishing a variety of information relating to its procurement activities and operational and business performance, including data formerly published by the Government Procurement Service.
  1. DfT reported that the main area of work has been the preparation for the transfer of Network Rail into the public sector from September and the transition of the Highways Agency from a DfT agency to a government company. All aspects of obligations and controls have been identified and actions agreed to migrate towards compliance or exceptions as appropriate.
  1. ALBswithin theFCO are fulfilling their obligations where it is appropriate and possible given national security constraints.
  1. The executive partner of HMRC, the Valuation Office Agency, has continued to publish Council Tax and Non-Domestic Rating List Band and released more information under the National Statistics programme.
  1. HMT has not yet discussed compliance with its agencies. However, as part of the Management Information Unit’s strategy HMT will be engaging with them in the near future.
  1. Home Office reported that the General Registry Office is exploring the feasibility of publishing data on places of worship. They have also had discussions with the Cabinet Office and representatives of the Open Data User Group regarding issues around the publication of registers of births, marriages and deaths.
  1. Department for Culture, Media and Sport(DCMS) reported that engagement through regular open data forums continues to reinforce the benefits of compliance.
  1. Defra has been focusing on a number of its ALBs and some core Defra teams with significant publications announced and completed by the Environment Agency and the Air Quality Team in Core Defra.
  1. Department for International Development (DfID) is in talks withthe Independent Commission for Aid Impact (ICAI) to discuss linking ICAI data to DfID data in order to improve compliance. DfID is also discussing adding the ICAI commissioners’ expenses along with their own corresponding datasets. ICAIs financial transactions with DfID continue to be routed via DfID’s publication commitments.
  1. Commonwealth Scholarship Commission (CSC) financial transactions with DfID are already routed via DfID’s publication commitments. The department also published monthly management information on its two ALBs (ICAI and CSC) on staff numbers, allowances and salary costs (published under the Workforce Management Information dataset).
  1. Ministry of Defence(MOD)reported that the National Museum of the Royal Navy, the National Army Museum, the United Kingdom Hydrographical Office and the Defence Science and Technology Laboratory have continued to encounter technical difficulties in publishing organograms.
  1. BISreported the following ALB compliance activities:

●In April, the Met Office hosted two of 95 International Space Apps Challenge hackathons held worldwide.

●In June, Ordnance Survey (OS) promoted the use of OS OpenData and OpenSpace at a hackathon in Wales. They also attended Association for Geographic Information Belfast, exploring the challenges and opportunities arising from Open Geospatial Information. Also in June, OS supported the Local Government Digital Makers Hackathon in Birmingham, which was organised by Nesta.

●The Met Office hosted two of 95 International Space Apps Challenge hackathons held worldwide - with the aim of contributing to space exploration missions and improving life on Earth.

National Information Infrastructure

  1. DfE reported that they have recently supported the Open Data Institute (ODI) as part of a series of open data challenges by providing safe and controlled access to the pupil records from the National Pupil Database, allowing software developers to test what the next generation of education software could look like.
  1. DH is working with its ALBs to understand and resolve barriers to publishing datasets identified as part of the NII. It is also working with them to identify an effective and efficient way to meet the requirement to identify a list of all unpublished datasetsand, where appropriate, on activity planned by ALBs as part of their work to reduce data burdens on the NHS.
  1. DWP reported that they were unable to publish some datasets,as they contain personal or sensitive data. The department is currently adding links to related (subset) data that it does publish, such as statistics or reports. A clear link has been established between DWP’s information asset inventory and the NII, so that the information asset community is able to easily identify new datasets.
  1. DfT reported that all existing commitments were delivered or are on track. The key task is the next iteration of the NII and an assessment of the transport content of the original NII that were agreed and published in 2013. Items under consideration may include rail fares, air data and accessible services.
  1. HMRC met NII commitments include:

●providing explanations on barriers for publication of datasets included in the first iteration of the NII in the Transparency Data Catalogue;