Skip to content

feat(data): refresh country carbon intensity from Our World in Data - #1468

Open
dilipna wants to merge 1 commit into
mlco2:masterfrom
dilipna:refresh-global-energy-mix
Open

dilipna wants to merge 1 commit into
mlco2:masterfrom
dilipna:refresh-global-energy-mix

Conversation

@dilipna

@dilipna dilipna commented Oct 28, 2026

Copy link
Copy Markdown

Description

Refreshes the country carbon intensities in global_energy_mix.json from Our World in Data's maintained energy dataset (Ember's electricity data and the Energy Institute's Statistical Review), and replaces the manual notebook run with a script.

  • codecarbon/data/private_infra/update_global_energy_mix.py: rebuilds the file from OWID's owid_energy.csv (default: OWID's catalog URL, or a CSV file or URL given as argument). It keeps each country's latest year with a carbon intensity, refuses to run if a country would disappear, and writes a carbon intensity that is not positive as null (CodeCarbon then computes it from the electricity mix), and prints the dataset's sha256 plus a short list of entries to check (empty mix, data older than two years, a year going backwards).
  • codecarbon/data/private_infra/global_energy_mix.json: regenerated. Same 213 countries and 19 keys, values only, except that Central African Republic now has a null carbon intensity instead of 0.0 (see below). 210 countries move to a newer year: 90 now use 2025 and 105 use 2024.
  • tests/test_update_global_energy_mix.py: new tests for the script, including that the committed file rebuilds exactly from the script's logic.
  • tests/test_emissions.py: the France and Canada tests hardcoded the old values; they now read the expected value from the data, so a refresh does not break them.
  • docs/explanation/methodology.md: describes the source (Ember via OWID, lifecycle basis) and the refresh command, and corrects China's national figure (582 → 525).

No logic in emissions.py changes.

Related Issue

Refs #503. Part of #947 (process for updating reference data).

Motivation and Context

The country values came from 2022 (130 countries) and 2023 (78), and refreshing them meant re-running a notebook by hand. What changes (carbon intensity in gCO₂eq/kWh, year in brackets):

Country before (year) now (year) change
France 56.0 (2023) 41.4 (2025) -26%
Germany 380.9 (2023) 329.6 (2025) -13%
China 582.3 (2023) 525.3 (2025) -10%
India 713.4 (2023) 670.1 (2025) -6%
United States 369.5 (2023) 384.4 (2025) +4%
Brazil 98.3 (2023) 110.0 (2025) +12%
Canada 170.0 (2023) 190.7 (2025) +12%

Across the 209 countries that gain a newer year and had a non-zero value, the median change is -2% and 40% of them move by more than 10%. Regional datasets (US states, Canadian provinces, Nordic zones, Chinese provinces) still take precedence where they apply; these are the national fallback values.

Source. The notebook reads OWID's GitHub copy of the dataset, which OWID now marks as "no longer updated" in favour of its catalog. I compared the two: on the 5,691 rows they share, the carbon intensity is identical, the country names are identical, and OWID publishes a column mapping. The script reads the catalog.

Central African Republic. The source reports a carbon intensity of 0.0 for CAF for every year from 2013 to 2023. Following your answer on #503 ("try with null instead of 0.0"), the script writes a non-positive value as null, so CAF falls back to the intensity computed from its electricity mix. The existing handling of an explicit numeric zero (#1441) is not touched. The docs paragraph describes this rule.

Other things you may want to decide

  • Reported emissions change for most countries (table above), so you may want to ship this with a release note.
  • Western Sahara (2009), Lesotho and Ukraine (2022) have old data in the source; the script lists them.

Dataset used: owid_energy.csv (9,081,888 bytes, served 2026-09-30), sha256 9d8efc4a9b1cc2ffdfa7f7f6f1e171b2922e0be2d97c0edf2982e874357ed6de. Running the script on it rebuilds the committed JSON byte for byte.

How Has This Been Tested?

  • The script run on the dataset above rebuilds the committed JSON byte for byte, and prints the sha256 and four checks (CAF, ESH, LSO, UKR).
  • New tests/test_update_global_energy_mix.py (20 tests): latest year with a carbon intensity, aggregates and countries without an intensity left out, missing sources, rounding and integer years, missing columns, each of the four checks, reading a file and a URL, the committed file rebuilding exactly, every country having a positive total (the viz code divides by it), and the command line (including refusing to drop a country). Each check was also confirmed to fail when the script is deliberately broken.
  • Full suite on Linux as a normal user, Python 3.12, on an earlier master (bcb446f2): 823 passed, 3 skipped. After rebasing on current master, on Windows (Python 3.12, tests/test_viz_data.py excluded because dash is not installed): 824 passed, 2 failed, 32 skipped. The 2 failures (test_expired_cache_entry_is_refetched, test_task_energy_with_live_update_interference) are the Windows/NVIDIA test issues from Unit tests on Windows: 13 failures besides the CLI ones in #1459 (one also on any machine with an NVIDIA GPU) #1460 and do not involve this change.
  • The new tests, tests/test_emissions.py and tests/test_zero_carbon_intensity.py also pass on Python 3.13 and 3.14 (pandas 3.0.5).
  • The open PR Add water consumption from electricity mix #1407 (water consumption) reads the same file: its 24 tests pass on current and on refreshed data.
  • Formatting hooks (autoflake, isort, black, flake8) pass.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

  • 🟥 AI-vibecoded
  • 🟠 AI-generated
  • ⭐ AI-assisted
  • ♻️ No AI used

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the docs/how-to/contributing.md document.
  • I have added tests to cover my changes.
  • All new and existing tests passed. (2 unrelated Windows failures, see above.)

🤖 Generated with Claude Code

Refs mlco2#503.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant