Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start grabbing GTFS data from FGB files on the web #7

Merged
merged 7 commits into from
Jul 6, 2024
Merged

Conversation

dabreegster
Copy link
Contributor

Part of #4. The approach is:

  1. Offline, assemble one massive GTFS feed covering ideally the world. Right now, it's just the UK, thanks to https://data.bus-data.dft.gov.uk/.
  2. Run a tool that extracts relevant info from the GTFS and expresses as a flatgeobuf file
  3. When the user imports an area on the web (dynamically from OSM data, thanks to the Overpass API), make the browser also query relevant areas of the FGB file and build the public transit graph too!

The "performance" is comically bad right now. Just the Elephant & Castle area triggers about 770 requests and sucks down 18MB. But not bad for a low-effort simple start!

The main missing thing is clipping; I'll try to add that now. Any trip intersecting the boundary area will have all stops included. So for part of central London, we wind up with stops very very far away:
image
This also messes up the routing.

Also before merging, I'll upload the FGB file and hook it up to the GH Pages demo... but maybe only if I can get the performance to be better, so I'm not burning through bandwidth ridiculously.

Encoding

What's in the FGB file exactly? Per "route variant", a linestring, with one point per stop. (The full route shape isn't included yet.) What's a "route variant"? In GTFS, "route 5" has many trips, some of them with potentially different sequences of stops (like an express / weekend / full service). To express a route geometrically, we need to decide the stops included, so we use the sequence of stops as the primary key.

For each of these route variants with a linestring, some JSON-encoded data:

  • per stop, its original GTFS ID and its human-friendly name
  • the list of trips. Each trip is a list of arrival times at the stops. Right now encoded as strings like "03:53:00"
  • some route metadata: GTFS ID, its long and short name, description

Also speed up FGB writing by not using geojson. Not sure results
readable yet, though. Also, breaking main.rs cause of async
No clipping yet, but otherwise things seem to work!
@dabreegster
Copy link
Contributor Author

Perf update: after using the most popular Rust compression crate on the properties, the full FGB file goes from 270MB to 100MB. Loading the same area now takes 935 requests (!!) but only 6.5MB instead of 18.

@dabreegster
Copy link
Contributor Author

dabreegster commented Jul 6, 2024

OK, I implemented clipping. Some of the routes near the map boundary are still odd, but I'll merge now. I've uploaded the UK GTFS to https://assets.od2net.org/severance_pbfs/gtfs.fgb (you can download the full thing and open in QGIS if you're curious) and wired up the web demo to use it.

Loads of work to do, but proof-of-concept of PT routing "anywhere" entirely within a browser!

@dabreegster dabreegster merged commit 8dd0a04 into main Jul 6, 2024
@dabreegster dabreegster deleted the fgb branch July 6, 2024 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant