GridSignal.
Great Britain electricity intelligence
From fragmented electricity feeds to a system people can investigate. GridSignal combines reproducible ingestion, SQL analytics, geospatial comparisons and a deployed API in one connected analytical workspace.
One grid. Several different signals.
Electricity data arrives through separate providers, publication schedules and measurement definitions. Carbon intensity describes emissions per unit of electricity; generation shares describe composition; metered outturn and demand describe power. Connecting them requires preserving those distinctions throughout the data pipeline.
I built GridSignal to make that investigation practical: move from a national overview to a time period, fuel or region, inspect the source evidence, and export the observations behind the result.
The grid, in perspective
Six streams, explicit contracts
| Stream | Provider | Purpose |
|---|---|---|
| National carbon intensity | NESO Carbon Intensity API | Published forecast and estimated actual intensity |
| Regional intensity | NESO Carbon Intensity API | Regional forecasts and generation composition |
| National generation mix | NESO Carbon Intensity API | Fuel shares at half-hour intervals |
| Metered fuel outturn | Elexon Insights / FUELHH | MW by original fuel category and publication time |
| Electricity demand | NESO Data Portal | National and transmission demand by settlement period |
| Published carbon factors | NESO Carbon Intensity API | Reference values with acquisition lineage |
Source-specific clients handle request pacing, retries, pagination and historical resource discovery. Response-contract checks detect structural changes, while Pydantic models validate individual records. Original HTTP responses remain available alongside the normalised facts.
A reliable path from source to screen
- 01Official APIsNESO and Elexon source clients
- 02Acquire & validatePython · contracts · Pydantic
- 03PostgreSQL warehouseRaw evidence · facts · revisions · operations
- 04SQL analytical martsTime-series, regional and quality metrics
- 05Read-only APIFastAPI · validated filters · connection pool
Collection that can recover and explain itself
Each collection run records its extraction window and obtains a source-specific database lock. Raw responses are committed before the fact-processing transaction. Failed processing preserves the acquired evidence while rolling back incomplete fact changes.
- Batched, hash-sensitive upserts insert new facts, skip unchanged observations and preserve meaningful revisions.
- Elexon publication ordering prevents an older release from replacing a newer stored observation.
- Progress checkpoints and overlapping extraction windows support bounded recovery after missed runs.
- A continuous reporting calendar supplies expected intervals independently of received data.
- UTC storage and Europe/London settlement mapping handle 46-, 48- and 50-period reporting days.
Ask a question, then follow the evidence
Carbon intensity through time
PostgreSQL computes period comparisons, distributions, hourly and weekday profiles, volatility, rankings and paired forecast–estimate differences. The browser consumes these analytical results rather than maintaining a separate set of metric definitions.
Generation and demand, with units intact
Spatial context without losing the national picture
Regional intelligence
The map uses licensed NESO distribution-network boundaries, transformed from British National Grid to WGS84 and simplified for the browser. Documented region mapping connects the geometry to source identifiers; the national series remains separate from regional forecasts.
A workspace, not a collection of notebooks
- Seven connected screens: overview, carbon, generation and demand, regions, forecast performance, operations and methodology.
- URL-backed date, region, fuel, resolution and comparison filters make analytical views shareable.
- Chart selections and regional comparisons connect summary views to underlying observations.
- Accessible data tables, keyboard navigation, responsive layouts and reduced-motion support accompany the visualisations.
- CSV exports return the full filtered dataset, independently of visible table pagination.
Deployed from ingestion to interface
The public workspace runs on Vercel, with FastAPI on Render and PostgreSQL 18 on Neon. GitHub Actions collects official observations every six hours. Docker Compose reproduces the complete database, migration, API, worker and frontend topology for development.
Collection status distinguishes a normal wait for the next scheduled update from failed or overdue work. Observation age remains visible separately, including cases where collection succeeds but an upstream provider has not published newer data.
Source timestamps, run counters, validation results and calendar completeness are part of the product. They help a user decide whether a pattern is ready to interpret, rather than asking them to trust a dashboard simply because it has rendered.
Correctness checked at each boundary
| Layer | Evidence |
|---|---|
| Python, API and database | 51 automated checks, including isolated PostgreSQL integration tests |
| Frontend utilities | 5 unit tests for reporting dates, missing values and request behaviour |
| Browser experience | 36 checks across desktop, tablet and mobile Chromium |
| Containers | Fresh Compose startup, migrations, all six adapters and a database-backed overview |
| Hosted application | Anonymous access, HTTPS, exact-origin CORS and 38 analytical routes verified |
Regression checks cover idempotency, revision ordering, interrupted collection, reader isolation, forecast-pair null handling, daylight-saving boundaries and completeness denominators. Collection-state tests include schedule grace, failed attempts, stalled work and delayed upstream observations.
Keep the interpretation as precise as the system
National actual intensity is an estimate; regional series are forecasts. Forecast comparisons describe published forecast–estimate differences, not retrospectively reconstructed issue-time skill. Metered outturn does not provide complete embedded-generation coverage, and interval-average fuel shares are not consumption-weighted energy shares.
The hosted edition refreshes every six hours and can take about a minute to resume after inactivity. The next engineering priorities are independent backup automation, controlled cold-start measurement and longer historical comparisons within a clear retention policy.