Process IDMC Internal Displacement Events Into Country-Week Panels

Reads IDMC Internal Displacement Updates (IDU) event records and expands them into complete country-week panels of displacement-occurrence covariates, mirroring process_EMDAT_data so the two hazard families can be joined onto the same ISO-week grid in compile_suitability_data().

Usage

process_IDMC_data(PATHS, source_dir = NULL, panel_start = "2000-01-03")

Arguments

PATHS

A list of paths generated by get_paths(). Must include:

DATA_IDMC_RAW: Path to MOSAIC-data/raw/IDMC.
DATA_IDMC: Path to MOSAIC-data/processed/IDMC/weekly.

source_dir

Optional directory of IDU event CSVs to read. When NULL (default), the NEWEST hdx_<date>/ snapshot directory under PATHS$DATA_IDMC_RAW is selected automatically (the convention written by download_IDMC_data); if no such snapshot exists, PATHS$DATA_IDMC_RAW itself is read, so a flat directory of CSVs still works. Pass an explicit path to reprocess an older snapshot. All *.csv files in the chosen directory are read and row-bound; duplicate event ids are dropped, so per-country mirrors and a global export can be mixed safely.

panel_start

First Monday of the panel grid. Defaults to "2000-01-03" to match the EM-DAT panels exactly. Cells before the first IDU event are structural zeros, NOT observed absences – see Details.

Value

Invisibly, a named character vector of the two written CSV paths (conflict, disaster). Each CSV has one row per (iso_code, year, week):

iso_code: ISO3 country code.
year, week: ISO-8601 year and week (1-53).
date_start, date_stop: Monday and Sunday of the ISO-week.
idmc_<kind>_active: 1 if any event of that kind overlaps the week.
idmc_<kind>_new: Number of events of that kind starting in the week.
idmc_<kind>_displaced: log1p of summed figure (persons displaced) across overlapping events; NAs treated as 0.

Details

Two panels are written to PATHS$DATA_IDMC: displacement_conflict_country_weekly.csv and displacement_disaster_country_weekly.csv. The split follows IDU's own displacement_type field ("Conflict" / "Disaster") and is deliberate: the two have different causal relationships to cholera. Disaster-driven displacement is largely flood/storm-mediated and therefore partly collinear with the EM-DAT hazard covariates, whereas conflict-driven displacement is an independent signal (WASH collapse, camp formation, care-seeking disruption) with no EM-DAT analogue.

Only rows with role == "Recommended figure" are used; IDU "Triangulation" rows are alternative estimates of displacement the recommended rows already count (IDMC totals use recommended figures only), so including them would double-count. Rows from extracts without a role column are kept.

Each event is expanded across the ISO-weeks its displacement_start_date:displacement_end_date range touches. Records missing a start date are dropped; a missing or earlier end date is set to the start date (single-day event).

Coverage caveat – read before modelling. IDU is a rolling near-real-time product and the accessible history depends on the route used. Most HDX per-country mirrors begin 2025-01-01, but coverage is NOT uniform – KEN reaches back to 2011, while several series cover only a few weeks. Zero cells before a country's first observed event therefore mean "not reported by this extract", not "no displacement occurred". Always check the per-country first-event dates printed by this function (and stored in the coverage attribute) before using the panel as a covariate or as a trigger denominator. This is a materially weaker coverage guarantee than EM-DAT's, which runs from 2000.

Collinearity caveat. idmc_disaster_* is largely flood/storm-driven and will correlate with emdat_flood_*; do not enter both into a feature set without checking. idmc_conflict_* is the genuinely additional signal.

Data source and access

IDU are preliminary estimates of new displacements, updated daily; the curated annual counterpart is the Global Internal Displacement Database (GIDD), which is too coarse for event-timing work. Two acquisition routes exist:

HDX per-country mirrors (RECOMMENDED – open, no credentials, refreshed daily): <iso>-idmc-idu-events datasets on the Humanitarian Data Exchange. Use download_IDMC_data, which resolves the current CSV resource URLs from the HDX CKAN API and archives a dated snapshot. 38 of the 40 MOSAIC countries have a dataset; ERI and TGO do not. Measured 2026-09-17: all 38 refreshed that day.
IDMC external API via the CRAN idmc package (idmc::idmc_get_data()), which reads an endpoint URL from the IDMC_API environment variable. This route is credential gated – unauthenticated calls to helix-tools-api.idmcdb.org/external-api/idus/all/ return HTTP 403 "Client is not registered." – and the client id must be requested from IDMC. It buys pre-2025 history; it is not otherwise needed. Note idmc_get_data() parses a standard_popup_text column that the HDX CSVs do not carry, so it cannot be used to read HDX downloads.

Note the URL advertised on the HDX resource page (backend.idmcdb.org) no longer resolves (DNS failure confirmed 2026-09-17); the live API host is helix-tools-api.idmcdb.org.