Documentation · Bulk data

Downloading and keeping data fresh

The bulk files sit at fixed addresses and need no login. This page explains the address pattern, how to tell when a new build is out, how to check a download, and how to keep your copy in step.

The bulk files are free to download, with no account. Non-commercial use is covered by our data licence; commercial use needs a licence. To check names one at a time instead, use the screening API.

Download addresses

Every file of every collection has one address, which always serves the latest build:

/downloads/collections/<collection>/<file>

# examples
/downloads/collections/default/entities.ftm.json
/downloads/collections/sanctions/targets.simple.csv
/downloads/collections/maritime/index.json

Collections: default, sanctions, peps, crime, maritime. Files are listed in Data formats. The catalogue of all collections is /downloads/collections/index.json.

BASE=https://crownintelligence.net     # the public download host; see the note below
curl -O "$BASE/downloads/collections/sanctions/targets.simple.csv"

Historical data

Each address is overwritten by the next build. We do not yet keep dated copies of past builds; the figures of every past build are kept in each collection's statistics.json, and the changes of recent builds in its deltas.json (see Delta files).

How often the data changes

Two clocks matter:

• Registers. Each source register is collected on its own schedule, matched to how often its publisher updates it. The refresh interval of every register in a collection is in its sources.json (cadence_hours); Freshness summarises them.
• Bulk builds. The files are rebuilt from the register as a batch. The current builds are below.

“Data last processed” is the latest confirmation date of any record in the build. A new build of unchanged data gets a new version but the same data date.

Keeping your copy in sync

Pick one of three methods, from simplest to most efficient.

1. Download everything again

Fetch the files you use on a schedule and replace your copy. Simple and robust; costs bandwidth on the large collections (default entities.ftm.json: —).

2. Poll the index, download on change

Fetch the collection's small index.json, compare its version (or a file's sha256) with what you stored, and download only when it differs.

#!/bin/sh
# download targets.simple.csv when a new build is out
BASE=https://crownintelligence.net; C=sanctions; F=targets.simple.csv
NEW=$(curl -s "$BASE/downloads/collections/$C/index.json" | jq -r --arg f "$F" '.files[] | select(.name==$f) | .sha256')
OLD=$(cat "$F.sha256" 2>/dev/null)
if [ "$NEW" != "$OLD" ]; then
  curl -s -o "$F.part" "$BASE/downloads/collections/$C/$F"
  echo "$NEW  $F.part" | shasum -a 256 -c - && mv "$F.part" "$F" && echo "$NEW" > "$F.sha256"
fi

3. Apply deltas

Download once, then apply each build's changes from deltas.json. Best for large collections and long-running stores. See Delta files.

Checking a download

Every file's SHA-256 checksum, size and row count are in its collection's index.json. Compare before you replace your copy:

shasum -a 256 targets.simple.csv          # macOS, Linux
certutil -hashfile targets.simple.csv SHA256   # Windows

Files are written to a temporary name and swapped in when complete, so a download never catches a half-written file. If you fetch index.json and a data file across a build boundary, the checksum will not match: fetch the index again and retry.

Deletions, merges and id changes

• Track records by id (crown-<number>). A record missing from a new build has left the collection: delete it from your copy. Delta files give these as DEL operations.
• Records we have matched as one party share a canonical_id. It can appear, change or disappear between builds as matching improves, so do not use it as a permanent key.
• referents is always empty today; it will list ids merged into a record once we publish merged entities.

Last updated 2026-10-03