Skip to content

dcp-tools

Prepare and load data for Data Commons Platform instances.

dcp-tools is a Python package for building the config, metadata, and data files a Data Commons Platform instance needs, then uploading them and triggering ingestion. Use it as a library or through the dcp-tools command-line tool.

A Data Commons Platform instance lets an organization combine its own datasets with the public knowledge graph at datacommons.org, while reusing the platform's search and visualization tools. The official Custom Data Commons documentation covers how the platform itself works. This site covers the tooling that prepares and loads data into it.

Note

This package was published as bblocks-datacommons-tools until version 0.1.1, and imported as bblocks.datacommons_tools. Installing the old distribution now pulls in dcp-tools and redirects those imports with a DeprecationWarning, so existing code keeps working. Update imports to dcp_tools when convenient.

Key features

  • Build and edit config.json files programmatically
  • Register single- and multi-entity observations from CSV data
  • Declare custom schema nodes (entity types, event types, properties, units, measurement methods, StatVars, and StatVar groups) with typed builders
  • Upload prepared files to Google Cloud Storage and trigger the DCP (Data Commons Platform) ingestion job
  • Usable as a Python API or through the dcp-tools CLI

Where to go next

  • Getting started: install dcp-tools and build a first import, from an empty config to files ready to upload.
  • Preparing data: task recipes covering the full CustomDataManager surface, from input files and custom dimensions to schema nodes and config merging.
  • Why config.json and MCF: the reasoning behind the config.json/MCF split, dcid minting rules, and how multi-entity observations use custom dimensions.
  • Loading data: upload prepared files to Cloud Storage and trigger the job that loads them into your instance.
  • CLI tools: reference for the dcp-tools command and its subcommands.