SLSA Marine Data Pipeline
The Challenge
A beach safety website is only as good as the data behind it. Telling someone the surf is dangerous at Cottesloe requires knowing the swell, the wind, the tide, the water temperature, the UV index and the forecast — for that beach, right now.
None of that data starts life in a usable shape. It arrives from the Bureau of Meteorology as gridded forecast products, FTP file drops, RSS feeds and radar imagery, all organised around meteorological geography — forecast grid points, marine zones, tide ports, observation stations — none of which is "a beach".
Surf Life Saving Australia needed every beach in the country continuously matched to the right data sources, refreshed on a schedule, and degrading gracefully when an upstream feed went quiet. Thousands of beaches. Every day. Without anyone touching it.
The Solution
The answer sits underneath SLSA BeachSafe: an ingestion and resolution layer that acquires government meteorological data on a schedule, normalises it, and fans it out to individual beaches.
Acquiring the data
- Gridded forecasts from the Bureau's Australian Digital Forecast Database, decoded into per-location swell, temperature, wind speed and wind direction series
- FTP acquisition from the Bureau's file service, with the transfer layer isolated behind its own service so upstream changes don't ripple through the app
- Tides imported from annual tide tables per port
- Observations pulled from weather observation stations
- Marine forecasts and warnings consumed from RSS
- UV index and water temperature on their own schedules
- Radar imagery downloaded and stored as time-ordered frames
Each source has a dedicated importer, so one failing feed never takes the pipeline down with it.
Resolving data to beaches
The harder half of the problem. Every beach has to be matched to its nearest tide port, its observation station, its forecast grid point, its marine zone and its radar — and so does every postcode, because visitors search by where they are, not by beach name.
That resolution runs as its own set of maintainable linking tasks over spatial data in MySQL, so when the Bureau adds a station or moves a grid point, the associations can be recomputed rather than hand-corrected.
Keeping it honest
Data-checking routines validate what arrived, retention tasks clear down old records, and the whole pipeline runs on a schedule with no manual step. When a feed is late or missing, the site falls back rather than showing something wrong — in beach safety, a stale reading presented as current is worse than no reading.
The Results
- More than 12,400 Australian beaches each resolved to their own forecast grid point, tide port, observation station, marine zone and radar
- A fully automated daily pipeline across roughly twenty distinct government data sources, running unattended
- Isolated failure domains — one dead upstream feed degrades one feature, not the site
- Still running today, more than a decade after the first version, through successive Bureau product changes
What that means commercially
Twelve thousand beaches is the number that makes this a data-engineering problem rather than a website. Any one beach could be wired up by hand in an afternoon. Twelve thousand of them, each needing five different upstream sources correctly matched and refreshed daily, is only tractable if the matching is computed and re-computable — which is exactly why the resolution logic is built as maintainable tasks rather than a migration someone ran once in 2013.
Technology
Laravel with a scheduled command per data source, service classes wrapping each Bureau product (FTP transfer, forecasts, observations, swell, UV, annual tides), spatial queries in MySQL 8 for beach-to-source resolution, and queue workers for the heavier import runs. Deployed and operated by Nano Solutions on infrastructure we also maintain.
See also: SLSA BeachSafe — the public platform this pipeline feeds, and SLSA Bluebottle Forecast — the predictive layer built on top of it.
Related Nano Solutions services: Systems Integration · Cloud Maintenance · Software Development. WA government agencies can engage us directly through our CUAICTS2021 panel listing.