SDG Build is a Python package for converting data on the Sustainable Development Goals (SDG) from one format into another. This is mainly useful to an SDG reporting platform, by providing these benefits:
- Input of SDG data from various machine-readable formats, for human-friendly visualisation and display
- Output of SDG data to various machine-readable formats, for interoperability with other systems
- Validation of the data and metadata for quality control
- Python 3.7 or higher
SDG Build can input SDG data in the following formats:
- Data in CSV files (long/tidy format)
- Data (and minimal metadata) from SDMX-JSON and SDMX-ML
- Data from a CKAN instance
- Data in JSON-Stat format from an API
- Data from a PxWeb API instance
- Metadata in YAML files
- Metadata in CSV files
- Metadata in Excel files
- Metadata in SDMX files
- Metadata in Microsoft Word templates
SDG Build can output SDG data in the following formats:
- A particular JSON structure for data and metadata, expected by the Open SDG reporting platform.
- GeoJSON for mapping (there is not a global standard for SDG GeoJSON at this time, so this is our best guess at a useful structure)
- SDMX-ML output (data and metadata)
- Datapackages
- CSVW
Sometimes you may need to alter data and/or metadata before importing into this library. This can be done after instantiating the input objects, with add_data_alteration and add_meta_alteration to add callback functions.
In these callback functions, the thing to be altered is passed as the first parameter, followed by a "context" dict, containing other information. These "context" dicts currently contain:
- indicator_id : string - The id of the indicator, if known
- class : string - The name of the particular input class that is being altered
For example:
def my_data_alteration(df, context):
# Target a particular input class.
if context['class'] != 'InputCsvData':
return df
# Drop an unnecessary column in the data.
df = df.drop('unnecessary_column', axis='columns')
# Drop an unnecessary column for one particular indicator.
if context['indicator_id'] == '1-1-1':
df = df.drop('another_column', axis='columns')
return df
def my_meta_alteration(meta, context):
# Target a particular input class.
if context['class'] != 'InputYamlMeta':
return meta
# Drop an unecessary field in the metadata.
del meta['unnecessary_field']
# Set a metadata field for one particular indicator.
if context['indicator_id'] == '1-1-1':
meta['another_field'] = 'My hardcoded value for indicator 1.1.1'
return meta
my_data_input.add_data_alteration(my_data_alteration)
my_meta_input.add_meta_alteration(my_meta_alteration)
Sometime you may want to alter the metadata based on the data, or vice versa, to alter the data based on the metadata. The above-mentioned add_data_alteration and add_meta_alteration are not reliable for this, because they act on each individual input. So to accomplish this, there is an add_indicator_alteration method on all output classes, which can be used to set a callback function which runs on each indicator, after it has been fully compiled from all the inputs.
In these callback functions, the indicator to be altered is passed as the first parameter, and it is an Indicator object (see sdg/Indicator.py). The second parameter is a "context" dict, containing other information. These "context" dicts currently contain:
- indicator_id : string - The id of the indicator, if known
- class : string - The name of the particular output class containing the indicator
The callback function must return the Indicator object.
For example:
def my_indicator_alteration(indicator, context):
# If the number of years is 2 or less, use a bar chart.
if indicator.data.Year.nunique() <= 2:
indicator.meta['graph_type'] = 'bar'
return indicator
my_output.add_indicator_alteration(my_indicator_alteration)
SDG Build requires a schema for any metadata. Currently the following formats are supported:
- YAML schema intended for Prose.io
- SDMX MSD
A data schema can be provided to control the sorting and validation of data. Currently the following formats are supported:
- Table Schema (YAML)
- SDMX DSD
SDG Build can also import translations and use them to produce fully-translated builds. The translations can also be exported as well. Input formats include:
- SDMX DSD
- YAML local files
- YAML Git repository
The export formats include:
- JSON
Indicator metadata is expected to be simple key/value pairs, but translations can also be structured using a "subfolder" approach. With this approach, a full set of pre-translated key/value pairs can be placed in the metadata under the appropriate language code. For example, the following structure could be used to provide a Spanish translation of an indicator's name:
indicator_name: My English indicator name
es:
indicator_name: My Spanish indicator name
When using the InputYamlMdMeta class, this can be accomplished by creating subfolders for each language code, and adding pre-translated versions of the YAML files there.
Usage examples are available in docs/examples. In each of these examples, the output is generated in a _site folder. Before running these examples, make sure to run:
pip install -r docs/examples/requirements.txt
An example conversion from CSV data and YAML metadata into JSON suitable for the Open SDG platform. This example uses a configuration file and helper functions (open_sdg_check and open_sdg_build) for the greatest simplicity:
python docs/examples/open_sdg_simple.py
An example conversion from CSV data and YAML metadata into JSON suitable for the Open SDG platform. This example (in contrast to the "simple" example above) uses no helper functions, in order to demonstrate how to use the various Python classes:
python docs/examples/open_sdg.py
An example conversion from SDMX-JSON (from an API endpoint) into JSON suitable for the Open SDG platform:
python docs/examples/sdmx_json.py
An example conversion from SDMX-ML into JSON suitable for the Open SDG platform:
python docs/examples/sdmx_ml.py
An example conversion from CKAN data to JSON suitable for the Open SDG platform:
python docs/examples/ckan.py
An example conversion from CSV data to GeoJSON suitable for mapping:
python docs/examples/geojson.py
See the docs folder for additional documentation.
MIT © Office for National Statistics