Overview & Scope

This paper presents emerging thinking, areas for discussion and some initial recommendations for the technical proposals for implementing the International Aid Transparency Initiative (IATI).

At the heart of the IATI proposals is the development of an aid information standard, including an agreement that donors, and other actors, will publish information about what they are funding in a consistent way.

There are four parts to the IATI standards: (1) agreement on what will be published, (2) common definitions for sharing information, (3) a common electronic data format, and (4) a code of conduct

The paper attempts to set out HOW data providers will publish data and in what format (part two of standard), and HOW users of information can find, access and use the data. It covers the technical architecture, the data format that should be used, how data should be licensed, and finally, briefly considers some possible solutions to some of the key challenges that IATI faces to meet its objectives. The purpose of the paper is not to present a set of proposals for agreement, but to start consultation and discussion around the technical proposals amongst the members of the TAG.

It should be noted, that this report focuses on the formatof data, but necessarily also mentions its meaning, since without semantic information, a format has no value; however, the primary IATI semantic work, including detailed data definitions, is addressed in an companion paper “Discussion paper on draft definitions and data formats” which looks at the first two parts of the standards (what will be published and definitions).

Recommendations & Discussion Points

We have set out a series of overview requirements on page 6 that provides the basis for technical design decisions for IATI. Do you agree with these requirements?
The design is based on ‘specialist users’ being the direct consumers and users of IATI data (whilst the primary goal is to serve end users through Infomediaries).
We recommend that data providers should publish machine-readable aid- data files in a standard format freely available on public websites. Please consider specifically:

The publication of files vs. the implementation of APIs
How data should be segmented – by activity? country? data provider?
How data should be updated and versioned?
Whole vs. Incremental updates
How important is keeping historical data

We recommend an IATI website will create a registry of links to these aid-data files. Is this the right approach? What is the role of the registry? How important is it, that it isn’t a single point of failure?
We suggest, initially data providers must inform the registry directly about their published aid data files. Do you thinkthis isbest approach?
Should we consider the more sustainable decentralised option of data providers also publishing index and summary files to enable the registry to retrieve this information directly?
There will be a notification mechanism based on standard pull RSS feed. We suggest this is implemented by the registry initially. Should providers also provide this?
We recommend using a standard XML format and developing a new schema. Do you agree?
We propose the adoption of one standard open licensing model. Is this feasible?

Which license model should we adopt? Public domain? attribution? share-alike?

We suggest laying foundations for a more linked data/semantic web approach by establishing URL based identifiers for activities as well as other main elements such as sectors and countries. Do you agree?
We have suggested some options for addressing geographic classifications in three ways: 1) international standard ISO codes 2) geodesy codes 3) text. Thoughts?
Finally, the IATI registry could potentially take on additional functions such as authenticating sources of data and hosting services and software tools to help donors create and host aid data files. Are there additional roles for registry or additional technical services we should provide centrally?

Technical Requirements

This report proposes the following 9 high-level technical requirements for IATI:

Openly licensed — any third party must be allowed to use any published IATI data under the consistent open terms, without requiring explicit permission from the donor who provided the data. Use of IATI information must not be subject to patent restrictions or licensing fees of any kind.

Machine-readable — it must be possible for computer programs to extract useful information from IATI data without manual intervention. Whenever possible, IATI information should use numbers or codes to aid in machine processing. IATI may use regular text when it is more appropriate, to provide background information for human readers.

Easily accessible — users must be able to obtain IATI information automatically and anonymously using existing public network infrastructure. IATI should use well-known and well-supported open networking formats and protocols.

Decentralized — the IATI must be capable of continuing to function without a central administrator or computer system (no single point of failure).

Comparable— while not all donors will supply all data specified by IATI, whenever two donors dosupply the same data, it must be possible for users to compare those data in a meaningful way (apples to apples, and oranges to oranges).

Flexible— data formats and publishing schedules must allow donors the flexibility to omit information that is not relevant or available at different points in the information's life cycle.

Extensible— it must be possible for donors to supply additional information not covered by IATI if desired; it must be possible for users to determine what information represents donor extensions; and it must be possible for a user to ignore the extended information without affecting the value of the remaining core IATI information.

Vendor- and platform-independent— IATI information publishing must not depend on software from a specific vendor, or on a specific hardware or software computing platform.

Multilingual— for human-readable text, the IATI must support all major world languages, and it must support multiple versions of the same text in different languages.

Technical Architecture

How to Publish and Find Aid Information

This section outlines some emerging thinking about IATI's architecture — the architecture describes how donors publish data and users will find and access the data, while the following section, “Data Formats” (page 14) describes what donors publish. The following section, “Licensing of data” (page 20), covers the legal terms under which IATI donors will publish their data.

There are already many different kinds of projects and initiatives that seek to collect and amalgamate or analyze aid data and make it accessible. IATI does not intend to replace any of these; instead, our goal is to provide them with new and valuable sources of aid information. A good starting point for the architecture discussion is to distinguish what the IATI isfrom what it is not:

What the IATI is / What the IATI is not

an agreed set of definitions for aid data
shared technical specifications for a data format
an agreed mechanism for how data will be published andmade available
shared code of conduct and commitment to data sharing and transparency

an aid software package
an aid web application
an aid search engine
an aid database
an aid management system
a central organization that collects aid information

Target Audience

It is useful to distinguish between two types of users:

Non-technical end users of aid data include stakeholders such as politicians, policy staff, and citizens. These are the people who typically want access to aid data through user friendly, interactive applicationsand searchable databases, and want access to reports, summaries and graphs.

Technical specialist users of aid data include people such as application developers, owners of aid databases and aid managements, statisticians, researchers, and analysts. These are the people who collect, summarize, or analyze raw aid data and make it accessible to the end users by re-purposing it and developing new applications and information services.

IATI aims to provide non-technical end users with access to better quality data and better quality services, websites and tools to help access it. IATI is designed to do this by serving the technicalspecialist users directly, supplying them with more and better raw data more quickly. Better raw data allows the specialist users to provide a better quality of information to the end users, who thus benefit from IATI indirectly.

These specialist users will be able to access the data both manually by searching directly on the registry website, or automatically by establishing systems to retrieve data directly. While we have striven to keep the architecture and data formats as simple as possible – and it will be possible for non-technical end users to find and access the raw data - it is likely to require a certain amount of technical knowledge and effort to use the data effectively. However, it would be relatively simple to create simple translation tools for both specialist and non-specialist users to open this data in desktop office applications such as Excel.

Overview

We recommend that IATI adopt a decentralised, web-based architecture with the following three components:

Machine-readable aid-data files freely available on public web sites for download or search-engine indexing
A discovery mechanism for aid-data files, based initially on a central IATI registry to collect and provide links to all the data files published, but with potential to be further decentralised to remove dependency on such a central function
A notification mechanism to let users know about new or modified aid-data files.

To keep implementation cost and effort as low as possible, to avoid a single point of failure, and to ensure that IATI adds value to existing initiatives such as aid databases (rather than duplicating their efforts), we propose a decentralisedweb-based architecture to allow donors to publish their own aid information directly to all interested specialized users with no intermediary.

Donors should use existing web and internet infrastructureas their channel to publish aid data in machine-readable files. Donors can publish those files on a new or existing web site as they would any other resource, such as a graphics file

Files or APIs?

An alternative approach to data files would have been to require donors to implement an API (application programming interface) to respond to aid-data requests dynamically over the web, but we have decided against this approach for several reasons:

APIs require considerable additional technical infrastructure and development effort, while downloadable files will work with any existing web infrastructure.
APIs provide weak support for publication-approval workflows, since it is difficult to be certain what a user will receive through an API, especially if it is connected to a constantly-changing database; files, on the other hand, can be exported and approved before release.
Files can be downloaded and processed offline, which may be especially valuable for users without persistent Internet connections.
API data are difficult to test and validate, while files can be validated against schemas and other batch processes to verify both structure and content.
API data are more difficult to digitally sign, while files can easily be signed with the donor’s hash key (though digital signatures are not part of our first planned implementation phase).

Note that donors are still free to generate aid files dynamically from their databases, just as many web sites generate web pages dynamically — that is an internal implementation detail outside the scope of IATI – but should publish the data in files in addition to the API. IATI may provide specifications for optional, supplemental APIs in the future for functions such as searching a donor’s data, and in the interim, we anticipate that some intermediaries (such as online aid databases) may amalgamate data from donors and provide access through their own web APIs.

Donors may choose to publish aid data files on their existing web sites, or to delegate the publishing to contractors, partners, or external services, but the choice must lie with them

Aid Data Files

The first of the three major IATI architectural components is the aid data files themselves. Using this approach, donors would publish aid data files simply by placing them in a publicly-accessible web directory (one that does not require registration or login). The following “Data Formats” section (page vi) describes the internal format of these machine-readable files, which will be based on XML (the Extensible Markup Language).

An IATI donor’s web site could — in the simplest case — simply add XML files containing aid data to the same location:

file1.xml — first file of aid data.
file2.xml — second file of aid data.

Any web content management system should allow these files to be added without customization or custom development. If the URL (web address) of the first aid data file could be “ and any member of the public could download and use it.

Segmenting Data

How should donors divide IATI aid data into files? There are three criteria that any segmentation should to consider:

Non-duplication: whether aid data for any single activity should appear in only one place at the donor’s site, so that there is an unambiguously authoritative source of information
Persistence: whether aid data for any single activity must continue to have the same web address through its entire lifecycle
Granularity: users must not be required to download unreasonably large amounts of data to obtain information about a single activity

There are many different ways donors could segment their aid-activity data:

one file for each activity (e.g. Clean water infrastructure project for Kenya in its own file)
one file for each partner country or region (e.g. all activities for Kenya in a single file)
one file for all activities of a specific type
one file for all of a donor’s activities

All of these approaches have advantages: for example, with the second approach, someone interested in aid for Haiti could download all Haiti-related information in a single file. However, the last three approaches all suffer from potentially significant disadvantages:

If an aid activity affects more than one sector or country (for example, an aid project targeting both Senegal and Guinea-Bissau), the activity cannot appear in both files without violating the non-duplication criterion.
If a donor initially allocates an activity without having chosen a specific target country or sector (for example, a grant), or changes the target country or sector for a grant, the donor cannot move the activity to a different country or sector file without violating the persistence criterion.
While a single file for all activities might be suitable for a small donor, a single file for a large donor might contain information about tens of thousands of activities, violating the granularity criterion.
If IATI decides on a code of conduct that encourages or requires donors to leave activity data online indefinitely after the activity’s completion, any of the last three files will continue to grow year to year. Moving old activity data to a different file would violate the persistence criterion, while allowing the files to continue to grow as new activities are added would eventually violate the granularity criterion.

For these reasons, the use of a single file for each aid activity seems like a good approach, so that each activity’s data will have a single, persistent address, and so that users can download data with arbitrarily small granularity. We will also need to consider how much data will be published for each activity – if we agree to publish transaction data, files with collections of activities could become increasingly large. We recognize, however, that there are disadvantages to this approach: for example, a partner country might have to download hundreds of files to obtain information about all activities targeting that country. We will aim to design a format that allows either multiple activities or a single project, but agreement of good practice in this areas would be useful for users of the data

A possible development for the future is to retain the one-file-per-activity as the canonical source of data for each activity, but also adding index and summary files. The index files will contain metadata and links for each activity, while the summary files will contain key information (including financial totals) for different groupings of data.

We look forward to views from IATI members about how the data should best be segmented and the potential for these additional index and summary files

Web Best Practices

IATI data providers may choose their own locations for publishing aid data, but IATI recommends that they design their aid-data URLs to be permanent and self-documenting, to make it easier for users and search engines to find. URL’s such as “ represent particularly bad design as it is not a permanent address, is subject to change and search engines will sometimes not pick it up.

A well-designed URL provides a clear path to the file, with no dependencies on a specific content-management system or portal, such as “

Updating and Versioning Data Files

Updating Data Files

There are two options for updating data files:

A full dataset update where the whole dataset is replaced regardless of how much of it has changed
Incremental updates where just the data that has changed since the previous publication

We recommend IATI implements a full dataset update approach. It is now easy to republish an entire electronic document and determine what has changed by comparing it to the previous version. It is often difficult (and error-prone) to apply incremental updates to an existing dataset, because of the huge risk of falling out of sync, and it's also likely to be much more expensive for data providers to design a system that isolates and publishes deltas, rather than just re-importing the entire dataset.