Area Intelligence Map — Data Documentation

Last updated: 2026-03-30 Map URL: http://cathaldev3.fcrweb.ie/ Data location: data/map/ (local) + fcrwatcher S3 bucket (roamworkflow/map-data/)

1. Data Sources

1.1 Census 2022 Small Areas (CSO)

Raw fields used:

1.2 CSO Small Area Boundaries (GeoJSON)

1.3 Pobal HP Deprivation Index (2022)

1.4 Commercial Properties — Tailte Éireann Valuation Office

1.5 Companies Registration Office (CRO)

1.6 New Dwelling Completions (CSO NDQ07)

1.7 Commencement Notices (CSO HSM14)

1.8 Property Price Register (NOT YET INTEGRATED)

2. BigQuery Storage

Project: listingmanager-1529856313699 Dataset: fcr_operations

2.1 Tables

2.2 cso_small_areas_v2 Schema

CREATE TABLE fcr_operations.cso_small_areas_v2 (
  sa_geogid        STRING,     -- CSO small area ID (e.g. "A017010016")
  sa_pub           STRING,     -- Public ID (e.g. "017010016")
  county           STRING,     -- County name (UPPERCASE, e.g. "WATERFORD")
  ed               STRING,     -- Electoral Division name
  urban_flag       INT64,      -- 1=urban, 0=rural
  urban_area_name  STRING,     -- Town/city name (blank for rural)
  total_households INT64,      -- Occupied dwellings (Census 2022)
  total_population INT64,      -- Resident population
  houses           INT64,      -- Houses/bungalows
  flats            INT64,      -- Flats/apartments
  centroid_lat     FLOAT64,    -- WGS84 latitude
  centroid_lng     FLOAT64,    -- WGS84 longitude
  -- Purchasing power (derived)
  pct_professional FLOAT64,    -- % professional/managerial social class
  pct_degree_plus  FLOAT64,    -- % honours degree or higher
  pct_homeowner    FLOAT64,    -- % owner-occupied households
  pct_multicar     FLOAT64,    -- % households with 2+ cars
  pct_employed     FLOAT64,    -- % at work (of 15+ population)
  pp_score         FLOAT64,    -- Composite purchasing power 0–71
  -- Pobal deprivation (joined from ED level)
  pobal_index      FLOAT64,    -- Pobal relative deprivation index
  pobal_category   INT64,      -- 1–6 scale
  pobal_label      STRING      -- Human-readable label
);

3. Transformations

3.1 Purchasing Power Score (pp_score)

Script: data/cso/build_purchasing_power.py

Input: SAPS 2022 CSV (793 columns) → 5 percentage fields → 1 composite score

Composite: pp_score = Σ(component × weight) / Σ(weights for non-null components)

Distribution:

Important: This is our own derived metric, not an official index. It uses real census counts but the weighting is ours. Pobal's deprivation index uses similar inputs but different methodology (factor analysis).

3.2 Pobal Deprivation Join

Script: data/cso/build_purchasing_power.py

Join key: cso_small_areas.ed (Electoral Division name) → pobal_deprivation.ED_ENGLISH (case-insensitive UPPER match)

Match rate: 18,853 of 18,919 small areas matched (99.65%). 66 small areas across 3 EDs unmatched due to naming discrepancies:

• CASTLETOWN (NORTH)
• DUNDALK RURAL (SOUTH)
• ST. MARY'S (EAST)

Granularity mismatch: Pobal is at ED level (3,418 EDs), small areas are finer (18,919). All small areas within an ED get the same Pobal score.

3.3 Coordinate Conversion (ITM → WGS84)

Script: data/cso/fix_coords.py

CSO GeoJSON boundaries use Irish Transverse Mercator (ITM). Centroids computed as average of boundary polygon vertices, then converted to WGS84 lat/lng using GRS80 ellipsoid parameters:

• Semi-major axis: 6,378,137.0m
• Origin: 53.5°N, 8°W
• False easting: 600,000m, false northing: 750,000m

Sanity check: all 18,919 centroids fall within 51–56°N, 5–11°W (Ireland bounding box).

3.4 Eircode → County Mapping (Growth Data)

Script: data/map/build_growth_data.py

CSO NDQ07 reports completions by 3-character Eircode routing key (e.g. "X91"). We map these to county names using a hardcoded lookup table (139 Eircodes → 26 counties).

Mapping rate: 98,724 of 98,917 completions mapped (99.8%). 193 completions in Eircodes not in our lookup.

Known limitation: Dublin Eircodes (D01–D24, A94, A96, K78, etc.) all map to "DUBLIN". The map's county selector splits Dublin into Dublin City, Fingal, South Dublin, Dún Laoghaire-Rathdown. Growth data shows the combined Dublin figure for any Dublin sub-county. Same applies to Cork/Cork City, Galway/Galway City, Limerick/Limerick City, Waterford/Waterford City.

3.5 Local Authority → County Mapping (Commencements)

Script: data/map/build_growth_data.py

CSO HSM14 uses Local Authority names (e.g. "Dún Laoghaire-Rathdown"). We map these to our county names using a string-match lookup. 29 of 31 LAs mapped.

4. JSON Data Files (Map Layer Data)

Local path: data/map/ S3 path: s3://fcrwatcher/roamworkflow/map-data/ Public URL: https://fcrwatcher.s3.eu-west-1.amazonaws.com/roamworkflow/map-data/

Total: 29 files, 16 MB

4.1 small_areas.json Record Format

{
  "id": "A017010016",
  "ed": "CARLOW RURAL",
  "county": "CARLOW",
  "area": "Carlow",
  "lat": 52.844252,
  "lng": -6.918703,
  "hh": 114,
  "pop": 242,
  "houses": 105,
  "flats": 9,
  "pp": 36.87,
  "pProf": 14.91,
  "pDeg": 25.95,
  "pHome": 70.18,
  "pCar": 36.84,
  "pEmp": 47.27,
  "pi": -2.21,
  "pc": 4,
  "pl": "Marginally Below Average"
}

4.2 growth_data.json Structure

{
  "meta": { "sources": [...], "updated": "2026-03-30" },
  "completions_eircode": { "X91": { "2023": 560, "2024": 739, "2025": 742, "total": 2041 }, ... },
  "completions_county": { "WATERFORD": { "2023": 560, ... }, ... },
  "commencements_county": { "WATERFORD": { "2023": 700, ... }, ... },
  "completions_lea": { "Dungarvan, Waterford": { "2023": 48, ... }, ... }
}

5. Data Pipeline

5.1 Initial Load (one-time)

1. Download SAPS 2022 CSV from CSO
2. Download GeoJSON boundaries from OSi
3. Run fix_coords.py — converts ITM→WGS84, joins SAPS, outputs cso_small_areas_centroids.ndjson
4. Run build_purchasing_power.py — adds PP fields + Pobal join, outputs cso_small_areas_enriched.ndjson + BQ batch SQL files
5. Create cso_small_areas_v2 table in BQ (manual — CREATE TABLE DDL)
6. Load batches via n8n workflow cso-bq-insert (48 batches of 400 rows)

5.2 Map Data Generation

1. Run data/map/pull_all_data.py — pulls from BQ via worker API into local JSON files
2. Run data/map/build_growth_data.py — processes CSO NDQ07 + HSM14 CSVs into growth_data.json
3. Upload JSON files to S3 via n8n workflow map-data-upload (webhook: POST /webhook/map-data-upload)
4. Upload JSON files to server: scp to admin@52.48.145.217:/var/www/cathaldev3/data/
5. Run data/map/generate_national_map_s3.py — generates Ireland_Area_Intelligence.html
6. Deploy HTML: scp to /var/www/cathaldev3/index.html

5.3 Refresh Schedule

5.4 n8n Workflows

6. Known Limitations & Caveats

1. Census 2022 is 4 years old. New housing estates built 2023–2026 have zero population in census data. Growth data (NDQ07/HSM14) partially compensates but doesn't have demographic detail.

2. Purchasing power score is derived, not official. Our weighting (25/25/20/15/15) is reasonable but arbitrary. Pobal uses factor analysis with different weights. Both use the same underlying census data.

3. Pobal at ED level, PP at small area level. Pobal assigns one score to ~5-6 small areas. Our PP score varies within an ED. They may appear to contradict — e.g. a small area with PP 40 inside an ED labelled "Marginally Below Average". This is expected: the PP score is more granular.

4. CRO data is county-level only. No coordinates for individual companies. Registered address ≠ trading address. Useful for macro density but not for "where exactly are the accountants".

5. Growth data eircode→county mapping loses Dublin granularity. All Dublin Eircodes map to one "DUBLIN" entry. Can't split completions between Dublin City, Fingal, South Dublin, DLR.

6. Valuation properties may be stale. Tailte Éireann updates periodically but we have a point-in-time snapshot. New builds won't appear until rated.

7. Property Price Register not integrated. No public API. Form-based download only (Lotus Domino). Would provide trailing 12-month transaction volumes and median prices by Eircode — valuable for showing where money is being spent.

8. 3 Electoral Divisions unmatched in Pobal join (66 small areas). CASTLETOWN (NORTH), DUNDALK RURAL (SOUTH), ST. MARY'S (EAST) — naming differences between CSO and Pobal datasets.

7. Verification Queries

Business Planning can verify key numbers using these BQ queries via the worker API:

API_KEY=$(grep VITE_N8N_API_KEY .env.local | cut -d= -f2)
BASE="https://fcr-dashboard-api.cathaldempsey.workers.dev/dashboard-bq-execute"

# Total small areas and population
curl -s -X POST -H "x-api-key: $API_KEY" -H "Content-Type: application/json" "$BASE" \
  -d '{"sql": "SELECT COUNT(*) as areas, SUM(total_population) as pop, SUM(total_households) as hh, ROUND(AVG(pp_score),1) as avg_pp FROM `listingmanager-1529856313699.fcr_operations.cso_small_areas_v2`"}'

# Purchasing power by county
curl -s -X POST -H "x-api-key: $API_KEY" -H "Content-Type: application/json" "$BASE" \
  -d '{"sql": "SELECT county, COUNT(*) as areas, ROUND(AVG(pp_score),1) as avg_pp, SUM(total_households) as hh FROM `listingmanager-1529856313699.fcr_operations.cso_small_areas_v2` GROUP BY county ORDER BY avg_pp DESC"}'

# Pobal match rate
curl -s -X POST -H "x-api-key: $API_KEY" -H "Content-Type: application/json" "$BASE" \
  -d '{"sql": "SELECT COUNTIF(pobal_index IS NOT NULL) as matched, COUNTIF(pobal_index IS NULL) as unmatched FROM `listingmanager-1529856313699.fcr_operations.cso_small_areas_v2`"}'

# Commercial properties by county
curl -s -X POST -H "x-api-key: $API_KEY" -H "Content-Type: application/json" "$BASE" \
  -d '{"sql": "SELECT county, COUNT(*) as properties FROM `listingmanager-1529856313699.fcr_operations.valuation_properties` WHERE centroid_lat IS NOT NULL GROUP BY county ORDER BY properties DESC", "limit": 50}'

FCR Dashboard documentation · generated from docs/ · keep counts verified, not guessed.