Skip to content

Our data pipeline

Find all the information about our data pipeline powering the ONE Data knowledge graph.

The ETL pipeline is a core component of ONE Data, bringing together data from various sources into a unified knowledge graph from which users have structured access to the data and from which we build our analytical workflows. This pipeline extracts data from various providers of development data, transforms it into a unified schema aligned with the Data Commons knowledge graph model, validates its accuracy and consistency, and loads it into the ONE knowledge graph.

Broadly, the ETL pipeline consists of the following steps:

flowchart LR
B[Extract] --> C[Transform]
C --> D[Validate]
D --> E[Load]

  1. Extract: Data is extracted from various sources—including APIs, databases, and files—using bblocks data importers tools for structured data access. Read more about bblocks here

  2. Transform: The raw data is reshaped into a unified schema that aligns with the Data Commons knowledge graph model. This includes designing the data model for the dataset, mapping the data to the schema, and applying any necessary transformations to ensure consistency.

  3. Validate: The transformed data is checked for accuracy and completeness. This involved validating values, detecting missing or incorrect data, verifying metadata coverage, and ensuring that the data meets our quality standards.

  4. Load: Once validated, the data is loaded into the ONE knowledge graph, making it available for exploration and analysis though ONE Data.

How does the ETL pipeline fit into ONE Data?

The ETL pipeline is the backbone of the ONE Data platform. It enables users to access up-to-date structured and validated data stored in the ONE knowledge graph, through the ONE Data website or API. It enables us to build and automate our analytical workflows, which start by accessing the data through the knowledge graph.


flowchart

SA[World Bank] --> ETL
SB[IMF] --> ETL
SC[OECD] --> ETL

ETL --> KG[(ONE knowledge graph)]

KG --> DA[ONE Data website]
KG --> API[ONE Data API]
KG --> AW[Analytical workflows]
AW --> DA