How this project works

The full picture, in plain words. What the app does, where the data comes from, how the pipeline and hosting fit together, and how to set it up from scratch. Deep per-field notes live in data.md; the deployment checklist is SETUP.md.

1. What this is

London Buses is a website for exploring every bus route in London. You can search routes and see them on a map. You can watch live arrivals and live bus positions. You can browse operators, garages and stops. You can also dig into contracts: who runs each route, what it cost, and when it comes up for tender again.

It is an independent project. It is not affiliated with Transport for London. All data comes from public sources.

1b. Orientation — the whole system in one screen

If you are reading this page to understand the project quickly (as a person or as an automated agent), this section is the map; the rest of the page is the territory.

• System of record: the Git repository. Every dataset is a committed JSON file. The served data is data/api/*.json (the "faux-API"); its machine-readable index is data/api/manifest.json — one entry per dataset with source, fetchedAt, rows, cadence and files.
• Runtimes (four): GitHub Actions runs the pipeline on schedules (§9); a static host serves main (§11); two server endpoints handle live GPS and the street-works push feed; the browser fetches arrivals and service status straight from TfL (§10).
• Data flow in one line: public source → scripts/fetch-*.js (writes data/source/ caches and data/*.json primaries) → scripts/build-*.js (joins into data/route_classifications.json and the overview geometry) → scripts/build-api.js (reshapes into data/api/) → scripts/audit-data.js (gate) → commit → host → index.html.
• Sources, by what they provide: TfL Unified API (routes, stops, timetables, status, arrivals, TIMS roadworks); TfL open-data files (route geometry ZIP, iBus vehicle list, QSI performance PDFs, tendering programme PDFs, tender award pages); DVLA (make, fuel, age per registration); DfT (BODS live GPS, Street Manager works, D-TRO orders, STATS19 collisions); ONS (CPI); londonbusroutes.net and bustimes.org (operator, garage, PVR, vehicle type — community references used to fill blanks); london-bus-routes.fandom.com (provisional awards); DVSA operator licences (garage capacity, hand-curated in capacity/licences.json); fixed snapshots for crowding and low bridges. Full table in §5; field-level lineage in data.md.
• Processing that changes meaning: fallback chains fill blanks (API first, scrape second, last-known-good third); data/route-overrides.json wins over everything; fleet make/age are a mode/mean over a route's recurring vehicles (one-off cover buses are dropped); operator names are normalised to parent brands via data/operator-aliases.json; contract value is a sum of accepted annual bids (an estimate); scheduled mileage is timetable × geometry (an estimate); awards from the community wiki are served flagged provisional; roadworks, orders and collisions are "corridor-joined" to routes by distance from the route line; every string is sanitised (no HTML, no control characters, no email addresses).
• Secrets: five Actions secrets for the pipeline and three host variables for the endpoints; the browser needs none. Names and where they go are in SETUP.md.
• Where to look: scripts/refresh.js for pipeline order; scripts/build-api.js for the shape of every served file; data.md for every field; tests/verify-*.mjs for what the app promises; SETUP.md to deploy.

2. The big idea

The data platform is one Git repository, and the site is a folder of static files served from it. Four parts do all the work:

• GitHub Actions is the scheduler and the computer. Workflows run on a timetable. They fetch data, check it, and commit the results.
• Git is the database. Every dataset is a JSON file in the repo. Every past state of the data is a commit. History comes free.
• A static host serves the repo. The JSON files under data/api/ act as the site's API. We call this the faux-API. Two small server endpoints sit beside the files: a live-GPS proxy and the street-works receiver.
• Live things stay live. Arrivals, bus positions and service status are never stored. The browser fetches them fresh each time.

Why build it this way? Almost nothing has to stay online: the pipeline runs on a schedule and the site is plain files. And every data change can be reviewed in Git history.

3. The codebase — where everything lives

The repository is the single source of truth. Code, data, docs, tests and automation all live in it.

How GitHub is used

• Branches: main is production; the host serves whatever is on it. Two side branches belong to the street-works feed (streetworks-inbox, streetworks-archive) and are managed by the code, never by hand.
• Secrets: the pipeline API keys (BUS_API_KEY, DVLA_API_KEY, and the D-TRO credentials DTRO_CLIENT_ID / DTRO_API_KEY / DTRO_CLIENT_SECRET) are stored as Actions secrets. They never appear in code or in the browser.
• Actions minutes: the repo is public, so Actions minutes are unlimited and free.
• Data commits: automated commits are plain commits with clear messages, like data: status refresh 2026-09-08T20:16, so any two states of the data can be diffed.

4. The app

The app is a single-page app in plain JavaScript. It all lives in index.html. There is no framework and no build step. You can read the file top to bottom.

The views

How it's built

• Routing: the URL hash picks the view. A hashchange listener calls navigate(), which renders into <main>.
• Data layer: one small object wraps fetch() for each faux-API file. Each file downloads once per visit.
• Async safety: each navigation bumps a token. Renderers check it after every await. A slow fetch from an old view can never overwrite the current one. Timers are cleared and maps destroyed on every navigation.
• Maps: Leaflet draws every layer. In the light theme the basemap is OpenFreeMap "Bright" (free vector tiles, no key, no usage cap), rendered by MapLibre GL through the maplibre-gl-leaflet bridge. It is split into two layers so place and road names sit above the route lines. The dark theme, and any vector failure (library blocked, no WebGL, tiles unreachable), use CARTO raster tiles instead. The MapLibre files load lazily, so pages without a map never download them. Optional layers: stops, live buses, garages, low bridges, collision heatmap, street works, traffic orders.
• Theming: CSS variables define both palettes. A tiny script sets the theme before first paint, from the saved choice or the system setting.
• Safety: every remote string that reaches innerHTML goes through one escape helper. CSV exports guard against formula injection. The Leaflet and MapLibre CDN files are pinned with integrity hashes.

5. Data — where it comes from

Keys go in a local .env file (copy .env.example) or in GitHub secrets. The frontend needs no keys at all. It runs entirely from the committed JSON.

6. Data — how it's made

npm run refresh runs the steps below in order (the historical "17-step" numbering kept its shape as steps were added — 9b, 12b, 12c). Fetch steps come first, then the builds, and the quality gate closes the run. A fetch step may fail without stopping the run — the last good data stays. A build step must succeed. If the audit fails, nothing is committed.

Four more steps run only in the 2-hourly status workflow, because they accumulate history from feeds that publish no archive: build-diversion-history.js (folds each diversion snapshot into the rolling history), fetch-roadworks.js (TIMS snapshot, corridor-joined to routes, accumulated), drain-streetworks.js (folds Street Manager events pushed to the streetworks-inbox branch into the archive, converting coordinates to WGS84), and fetch-dtro.js (statutory traffic regulation orders from DfT's D-TRO service — a full Greater London search per run while the store is small, paced under the service's rate limit, switching to incremental once volumes grow).

7. Data — where it's stored

Everything is a file in Git. There is no database, no object store, no cache server. Three layers of files:

• data/source/ — raw material. What the fetchers pulled down, plus long-lived caches. Example: the DVLA cache holds every vehicle lookup for 90 days, so most runs only query a handful of new registrations.
• data/ — built datasets. What the build steps produce. The most important file is route_classifications.json: one complete record per route.
• data/api/ — served files. The reshaped, app-facing JSON. This is what the browser actually downloads.

Some things are deliberately not stored: live feeds (always fetched fresh), API keys (env vars and secrets only), and node_modules (ignored).

Storage cost is a non-issue. The whole data set is a few tens of megabytes of JSON. Git compresses the daily diffs well.

8. The faux-API

The app reads these files from data/api/. They are plain JSON, committed to the repo, served as static files. They are public: no key, and the host serves them with access-control-allow-origin: *, so other sites can read them straight from the URL. Each row below says which script writes the file, what it is built from, how often it moves, and its top-level shape. Sizes are approximate and uncompressed.

Besides data/api/, two things are fetched by URL: data/routes/{id}.geojson (full-fidelity geometry per route, written by fetch-data.js) and data/route_destinations.json (used by the /v3 concept page).

8b. Data conventions

• Route ids are TfL line ids in upper case (25, N25, SL7, EL1). Night routes are N-prefixed; a 24-hour route is the day id with type: "twentyfour". Joint tender awards may key on a slash-joined id (1/N1); byRoute splits them.
• Envelope: almost every served file is {generatedAt, source, count, …} with the payload under one key (routes, entries, byId…). generatedAt is the build time; manifest.json's fetchedAt is when the upstream was last read, which is the honest "data as of".
• Dates are ISO 8601 strings (2026-10-05, or with time and Z). Money is pounds as a number; costPerMile is £ per live mile at award. Distances are km in the pipeline and metres in join thresholds; coordinates are WGS84 and GeoJSON order ([lon, lat]).
• TfL periods are 4-week reporting periods named P01 24/25 … P13 24/25; a financial year runs April to March.
• Blanks: a missing value is null, never an empty string or 0. When a source fails, the previous value is kept (last-known-good) and the manifest row still shows the older fetchedAt.
• Estimates are labelled in their file's source/method/note field: scheduled mileage, contract value and corridor joins are derived, not published figures. Provisional tender rows carry provisional: true.
• Sanitisation: every string that enters a cache goes through scripts/_lib/sanitize.js — tags stripped, entities decoded, control characters removed, email addresses redacted. The frontend escapes again before innerHTML.
• Stability: writers sort keys and skip rewriting when only timestamps changed, so a run with nothing new produces no diff and no commit.

9. Automations

Four GitHub Actions workflows keep the data fresh. Times are UTC. The app's About page shows the same schedule in London time.

Built-in safety:

• One at a time. All four share one concurrency group, so runs never overlap.
• Commit only on change. A run that finds nothing new commits nothing. The site does not redeploy for no reason.
• Cron slips. GitHub's scheduler can run a slot late, or drop it. The tender check also fires after every other workflow finishes, which is reliable. That is what guarantees same-day pickup of new awards.
• Safe pushes. If another commit lands mid-run, the workflow retries. The nightly, tender and fleet workflows reset to the new head, keep their freshly fetched source data, and re-run the API build so generated files are rebuilt rather than merged (rebasing regenerated JSON conflicts by construction). The status workflow, whose files rarely collide, simply pulls, rebases and retries up to three times.

10. Live data

Three things are live and never stored:

• Arrivals boards. The browser calls TfL's API directly. Boards count down every second and re-sync about every 30 seconds.
• Route status. Fetched from TfL on view, then re-polled every 60 seconds. If the live call fails, the committed snapshot fills in.
• Live bus positions. The GPS feed needs a secret key, so the browser can't call it directly. It calls /api/live/vehicles?line=25 instead. That is a small server endpoint — see the next section.

Every live surface survives a bad feed. Last-good data stays on screen with a quiet "reconnecting…" note. Polling backs off and recovers on its own.

11. Hosting

The site is a folder of static files plus two small server endpoints. Any host that can do the following will run it; today that host is Cloudflare Pages, and SETUP.md walks through both Cloudflare and a plain Linux server.

• Serve the repo as static files from the main branch, with index.html at the root, and pick up each data commit (Cloudflare redeploys on push; a server does a git pull on a timer).
• Headers. _headers lists them: HSTS, X-Frame-Options, nosniff, a referrer policy, and Cache-Control: max-age=0, must-revalidate for HTML, JSON, JS and CSS. There are no fingerprinted filenames, so revalidation is what keeps returning visitors on the current code. The faux-API also needs access-control-allow-origin: *.
• Redirects. _redirects keeps the old /v2 and /changelog.html URLs working (301s).
• The live-GPS endpoint. functions/api/live/vehicles.js serves /api/live/vehicles?line=25. It adds the secret BODS key, fetches the national GPS feed, filters it to the route's bounding box (read from data/api/route-bboxes.json), and caches the result for 10 seconds per line so heavy use never hammers the upstream feed. It needs the BODS_API_KEY environment variable. Written for the Cloudflare Pages Functions runtime; on another host, run the same logic as a small Node service.
• The street-works receiver. functions/api/streetworks.js serves /api/streetworks. It accepts DfT Street Manager push notifications (AWS SNS), verifies their signatures, filters to London's highway authorities by SWA org code, and commits each event to the streetworks-inbox branch via the GitHub API — git is the queue. It needs GITHUB_REPO (owner/repo) and GITHUB_TOKEN (a fine-grained token for that repo only, Contents read/write). The status workflow folds queued events into streetworks-history.json. Same runtime note as above.

12. Deployment

• What triggers a deploy: any commit on main. That includes code changes and the automated data commits. Data commit → host picks it up → fresh JSON on the site. On Cloudflare that is a build (a minute or two); on a server it is the next git pull.
• Commit budget: the schedules produce about 12 data commits a day. That matters only on hosts that rebuild per commit (Cloudflare's free tier allows 500 builds a month); a server pulling on a timer does not care.
• Caching: HTML, JSON, JS and CSS all revalidate on every load. Browsers send a quick check and get "not modified" or the new file. Nobody runs stale code against new data.
• Rolling back: revert the commit and push. The host serves the previous state at its next deploy or pull. Because data lives in Git, this works for data mistakes too.

13. Keeping the data honest

• Last-known-good everywhere. If a source is down, every field keeps its previous value. A flaky scrape never blanks the site.
• Fetch only what's new. Facts that never change (DVLA records, published awards) are cached. Watermarks skip work when an upstream hasn't moved.
• Manual overrides. data/route-overrides.json pins hand-checked corrections. They beat any scraped value and survive every refresh. Example: route 660's vehicle type, which the upstream reference still lists wrongly.
• Cross-checks. A separate audit compares our per-route vehicle verdicts against independent sources and writes disagreement reports to data/audit/.
• A hard gate. The last pipeline step checks every record for plausibility. Critical failures abort the run, so broken data is never committed.

14. Testing

The suites in tests/ run the real app in a headless browser. Every external feed is mocked, so they pass offline and never depend on TfL being up. They cover: mobile layout and overflow, dark and light themes, the tender tables and their analysis numbers, the CPI maths (checked against ONS reference values), the map hazard layers, live-feed outage and recovery, and the promise that no retired hosts are ever called. Run one with node tests/verify-<name>.mjs.

15. Set it up from scratch

The short version. SETUP.md in the repository is the full checklist, in order, with every secret, environment variable and registration spelled out.

1. Get the code. It ships with working data, so the site runs immediately with no keys.
2. Serve it locally. npm install, then any static server (npx serve .) from the repo root. Everything works from the committed JSON.
3. Get keys (only needed to refresh data): a TfL API key, a DVLA key, and a BODS key. All free. Copy .env.example to .env and fill them in.
4. Run the pipeline. npm run refresh. Watch the steps run. The audit at the end must pass. Each step can also run alone — see package.json.
5. Host it. Serve main as static files with the headers and redirects from section 11, and run the two /api endpoints with BODS_API_KEY set.
6. Automate it. Add BUS_API_KEY and DVLA_API_KEY as GitHub Actions secrets. The four workflows then keep the data fresh on their own. Public repos get unlimited Actions minutes.
7. Optional — street works. The Street Manager feed is push-based, so it needs your own registration: give the receiver GITHUB_REPO and a fine-grained GITHUB_TOKEN, GET /api/streetworks until it reports ready, then register that URL at DfT's open-data onboarding. Everything else works without it.
8. Optional — traffic orders. Register (free) at d-tro.dft.gov.uk, then add DTRO_CLIENT_ID, DTRO_API_KEY and DTRO_CLIENT_SECRET as Actions secrets. Without them the fetcher skips gracefully and the rest of the pipeline is unaffected.
9. Fix data by hand when needed. Add an entry to data/route-overrides.json. Rebuild: npm run build-classifications, then build-overview, then build-api. Commit. Overrides survive every future refresh.
10. Verify. npm run test:fixtures once (it downloads the pinned map libraries the suites serve offline), then run the suites in tests/. They need a Chromium and mock every feed.

16. Glossary