Skip to content

dcp-tools CLI

dcp-tools is the command-line entry point for the functions documented under Preparing data and Loading data. It converts a CSV into MCF, uploads a prepared export directory to Google Cloud Storage, and triggers the DCP ingestion job. Each subcommand parses its arguments and calls straight into the Python API. It adds no behavior of its own.

There are four subcommands and no subcommand groups: csv2mcf, upload, dataload, and pipeline. Run any of them as dcp-tools <command> ... or python -m dcp_tools <command> .... Every command, including dcp-tools itself, accepts -h/--help for its full option list.

Global options

upload, dataload, and pipeline connect to Google Cloud through a KGSettings object (see Loading data). Two flags control where those settings come from:

  • --settings-file PATH — a JSON file using the settings' alias keys, all uppercase (LOCAL_PATH, GCP_PROJECT_ID, GCS_BUCKET_NAME, and so on).
  • --env-file PATH — a .env file using the same keys.

If both are given, --settings-file wins and --env-file is ignored. If neither is given, settings are read from a .env file in the current directory.

csv2mcf takes neither flag. It doesn't talk to Google Cloud, so it has no settings to load.

customDC.json
{
  "LOCAL_PATH": "./export",
  "GCP_PROJECT_ID": "one-climate-finance",
  "GCS_BUCKET_NAME": "one-dcp-import",
  "GCS_INPUT_FOLDER_PATH": "climate-finance",
  "GCS_OUTPUT_FOLDER_PATH": "climate-finance/output",
  "LOAD_JOB_REGION": "us-central1",
  "LOAD_JOB_NAME": "dcp-prep-job",
  "INGESTION_WORKFLOW_NAME": "dcp-prep-workflow"
}
customDC.env
LOCAL_PATH=./export
GCP_PROJECT_ID=one-climate-finance
GCS_BUCKET_NAME=one-dcp-import
GCS_INPUT_FOLDER_PATH=climate-finance
GCS_OUTPUT_FOLDER_PATH=climate-finance/output
LOAD_JOB_REGION=us-central1
LOAD_JOB_NAME=dcp-prep-job
INGESTION_WORKFLOW_NAME=dcp-prep-workflow

Note

GCP_CREDENTIALS and LOAD_JOB_SERVICE_ACCOUNT are optional and omitted above. Without GCP_CREDENTIALS, GCP calls fall back to Application Default Credentials (gcloud auth application-default login).

Exit status

All four subcommands return 0 on success. Argument errors, such as a missing required argument or an invalid --node-type choice, print a usage message to stderr and exit non-zero before any subcommand code runs. Errors raised by the underlying Python API (a pydantic validation error from a malformed CSV, a Google Cloud auth failure) propagate as an uncaught exception and also exit non-zero. The CLI does not catch or reformat them.

dcp-tools csv2mcf

Converts a CSV of node metadata into an .mcf file, wrapping csv_metadata_to_mcf_file.

dcp-tools csv2mcf <CSV> <MCF>
                  [--node-type {Node,StatVar,StatVarGroup,Topic,StatVarPeerGroup}]
                  [--column-mapping CSV_COL=MCF_PROP]
                  [--csv-option KEY=VALUE]
                  [--ignore-column COLUMN]
                  [--append]

Arguments

  • <CSV> — path to the input CSV file. Required.
  • <MCF> — path to write the generated MCF file. Required, must end in .mcf.

Options

  • --node-type {Node,StatVar,StatVarGroup,Topic,StatVarPeerGroup} (default: Node) — the MCF node type to build, one per CSV row. StatVar, StatVarGroup, Topic, and StatVarPeerGroup map to the typed node models under dcp_tools.custom_data.models. Node builds the untyped base model and accepts any column as an extra property.
  • --column-mapping CSV_COL=MCF_PROP (repeatable) — rename a CSV column to an MCF property before building nodes, e.g. --column-mapping identifier=Node. Pass the flag once per column.
  • --csv-option KEY=VALUE (repeatable) — an extra keyword argument forwarded to pandas.read_csv, e.g. --csv-option delimiter=";".
  • --ignore-column COLUMN (repeatable) — drop a CSV column before building nodes. Pass the flag once per column.
  • --append — append to <MCF> if it already exists instead of overwriting it. Without this flag, csv2mcf overwrites the file (or creates it if it doesn't exist).

CSV values that map to a dcid:-typed field (Node, populationType, measuredProperty, measurementQualifier, measurementDenominator) must already carry the dcid: prefix. csv2mcf builds the node objects directly from the CSV, and unlike CustomDataManager's add_* methods it does not mint the prefix for you. A bare value such as climateFinanceProvidedCommitments in the Node column fails with a pydantic string_pattern_mismatch error.

typeOf isn't in that list: for --node-type Node it's a required column, but a bare value there is minted to dcid:<value> automatically; for the other four node types it defaults to a fixed value (dcid:StatisticalVariable, dcid:StatVarGroup, dcid:Topic, dcid:StatVarPeerGroup) that a CSV typeOf column doesn't need to repeat.

Example

variables.csv
Node,name,populationType,measuredProperty,statType,description
dcid:climateFinanceProvidedCommitments,Climate finance provided (commitments),dcid:Thing,dcid:amount,dcid:measuredValue,"Bilateral climate finance commitments reported by the provider country, in current USD."

$ dcp-tools csv2mcf variables.csv custom_nodes.mcf --node-type StatVar
$ cat custom_nodes.mcf
Node: dcid:climateFinanceProvidedCommitments
name: "Climate finance provided (commitments)"
typeOf: dcid:StatisticalVariable
description: "Bilateral climate finance commitments reported by the provider country, in current USD."
statType: dcid:measuredValue
populationType: dcid:Thing
measuredProperty: dcid:amount

--column-mapping, --csv-option, and --ignore-column combine to read a differently-shaped source file:

$ dcp-tools csv2mcf variables_raw.csv custom_nodes.mcf \
    --node-type StatVar \
    --column-mapping node_id=Node \
    --column-mapping label=name \
    --column-mapping pop_type=populationType \
    --column-mapping measured_prop=measuredProperty \
    --csv-option delimiter=";" \
    --ignore-column source_note

That reads a semicolon-delimited variables_raw.csv, drops its source_note column, and renames the rest to the properties named on the right of each =.

dcp-tools upload

Uploads every .csv, .json, and .mcf file under a directory to the configured GCS bucket, wrapping upload_to_cloud_storage.

dcp-tools upload [--settings-file PATH] [--env-file PATH] [--directory PATH] [--sync]

Options

  • --settings-file PATH, --env-file PATH — see Global options.
  • --directory PATH (default: the directory configured as LOCAL_PATH in settings) — local directory to upload. Files are matched recursively. Anything under it that isn't .csv, .json, or .mcf is skipped with a warning. The local directory structure is preserved under the bucket's input folder.
  • --sync (default: off) — after uploading, delete remote blobs that no longer have a local counterpart. Deletion is scoped to import subdirectories that have local files present. An import missing from --directory entirely, or present but empty, is left untouched remotely.

Example

$ dcp-tools upload --settings-file customDC.json --directory export/
2026-07-21 09:14:02 | INFO     | dcp-tools | Uploaded export/config.json to climate-finance/config.json
2026-07-21 09:14:02 | INFO     | dcp-tools | Uploaded export/provenance.mcf to climate-finance/provenance.mcf
2026-07-21 09:14:03 | INFO     | dcp-tools | Uploaded export/custom_nodes.mcf to climate-finance/custom_nodes.mcf
2026-07-21 09:14:03 | INFO     | dcp-tools | Uploaded 3 files to climate-finance in GCS bucket one-dcp-import

dcp-tools dataload

Triggers the DCP ingestion job that loads uploaded files into the knowledge graph, wrapping run_data_load.

dcp-tools dataload [--settings-file PATH] [--env-file PATH] [--imports a,b,c]

Options

  • --settings-file PATH, --env-file PATH — see Global options.
  • --imports a,b,c (default: every import) — comma-separated import names to scope the run to. Omit it to load everything under the configured GCS input folder.

Example

$ dcp-tools dataload --settings-file customDC.json --imports climate-finance
2026-07-21 09:15:10 | INFO     | dcp-tools | Starting data load job 'dcp-prep-job'
2026-07-21 09:15:11 | INFO     | dcp-tools | Started job 'projects/one-climate-finance/locations/us-central1/jobs/dcp-prep-job'

Note

dataload only triggers the ingestion job. It doesn't upload anything first, so the files it loads must already be in Cloud Storage (run upload beforehand, or use pipeline). Once the job finishes, the platform ingests and serves the new data on its own. There's no separate redeploy or restart command to run.

dcp-tools pipeline

Runs upload followed by dataload for every import in one call, wrapping upload_to_cloud_storage then run_data_load.

dcp-tools pipeline [--settings-file PATH] [--env-file PATH] [--directory PATH] [--sync]

Options

  • --settings-file PATH, --env-file PATH, --directory PATH, --sync — same flags and defaults as upload.

Heads up

pipeline has no --imports flag. It always calls dataload with no import filter, loading everything. To load a subset of imports, run upload and dataload --imports=... as two separate commands instead.

Example

$ dcp-tools pipeline --settings-file customDC.json --directory export/
2026-07-21 09:16:40 | INFO     | dcp-tools | Uploaded export/config.json to climate-finance/config.json
2026-07-21 09:16:40 | INFO     | dcp-tools | Uploaded export/provenance.mcf to climate-finance/provenance.mcf
2026-07-21 09:16:41 | INFO     | dcp-tools | Uploaded export/custom_nodes.mcf to climate-finance/custom_nodes.mcf
2026-07-21 09:16:41 | INFO     | dcp-tools | Uploaded 3 files to climate-finance in GCS bucket one-dcp-import
2026-07-21 09:16:41 | INFO     | dcp-tools | Starting data load job 'dcp-prep-job'
2026-07-21 09:16:42 | INFO     | dcp-tools | Started job 'projects/one-climate-finance/locations/us-central1/jobs/dcp-prep-job'
  • Preparing data — build the config.json, CSVs, and .mcf files that csv2mcf, upload, and pipeline operate on.
  • Loading data — the KGSettings model and the Python functions these commands wrap.