Rush — Peer Review Manual¶

Rush answers one question for HSLU students who work between Zurich and the campus: should I leave the office now, or wait? It looks up the average traffic for the hour and route you care about, adjusts for the current precipitation forecast, and prints a short verdict.

This site is the peer review manual for the DENG course project. Each section maps to a graded requirement from the project description. Use the sidebar to navigate.

Grading Overview¶

Section	Points	Page
Dataset and Use Case	5	1. Use Case
Ingestion Pipeline	10	2. Ingestion
Local Storage + Dockerized Environment	10	3. Storage and Docker
Data Transformation	5	4. Transformation
Workflow Orchestration	10	5. Orchestration
Repository Requirements	5	6. Repository
Peer Review Total	45
Final Presentation (IaC + Data Lake + DWH + Repo)	40	7. Infrastructure

Quick Start¶

One command sets up everything on macOS, Linux, or Windows (WSL):

bash <(curl -fsSL https://raw.githubusercontent.com/javihslu/rush/main/install.sh)

This clones the repository, checks for required tools (installing anything missing), and starts the full Docker stack. If you already have the repo cloned, run ./setup.sh from inside it.

Windows

Open PowerShell as Administrator and run wsl --install
Restart your computer
Install Docker Desktop (enable WSL 2 backend)
Open your WSL terminal (Ubuntu) and run the command above

What the Setup Does¶

When you run the install command, two scripts execute in sequence:

Step 1 -- Bootstrap (install.sh)

This is the script that curl downloads. It runs on your machine before the repo exists.

Check	Action on failure
Git installed?	Prints install link and exits
Docker installed?	Prints install link and exits
Docker daemon running?	Prints "start Docker Desktop" and exits
Repo directory exists?	Asks to reuse or abort

If all checks pass, it clones the repository and hands off to setup.sh.

Step 2 -- Local setup (setup.sh)

This runs inside the cloned repo. It reads all settings from config.yaml.

Step	What happens
Tool check	Verifies Git, Docker, Docker Compose are present
gcloud CLI	If missing, offers to install via Homebrew (macOS) or apt/dnf (Linux)
Terraform	If missing, offers to install via Homebrew or apt/dnf
Generate `.env`	Reads `config.yaml` and writes `.env` with database, pgAdmin, and Airflow credentials
Docker stack	Runs `docker compose up -d --build` -- builds the image and starts all 7 services
GCP onboarding	If gcloud is available, runs `scripts/setup-gcp.sh` (auth, project, billing, APIs, Terraform)
Done	Prints service URLs

Here is what a successful setup looks like in the terminal:

$ ./setup.sh
rush -- project setup
=====================

[ok] config.yaml loaded

[ok] git
[ok] docker
[ok] docker compose
[ok] gcloud CLI
[ok] Terraform

[ok] .env generated from config.yaml

Starting docker compose stack ...
[+] Building 20.7s (25/25) FINISHED
[+] up 15/15
 ✔ Network rush_default               Created
 ✔ Volume rush_postgres_data           Created
 ✔ Volume rush_airflow_postgres_data   Created
 ✔ Container rush-pgdatabase-1         Healthy
 ✔ Container rush-airflow-postgres-1   Healthy
 ✔ Container rush-pgadmin-1            Started
 ✔ Container rush-dev-1                Started
 ✔ Container rush-airflow-init-1       Exited
 ✔ Container rush-airflow-scheduler-1  Started
 ✔ Container rush-airflow-webserver-1  Started

Services:
NAME                          STATUS
rush-pgdatabase-1             Up (healthy)
rush-airflow-postgres-1       Up (healthy)
rush-pgadmin-1                Up
rush-airflow-webserver-1      Up
rush-airflow-scheduler-1      Up

=====================
Setup complete.

  pgAdmin:    http://localhost:8085
  Airflow:    http://localhost:8080
  PostgreSQL: localhost:5432
  Stop:       docker compose down

Once running:

Service	URL	Credentials
Airflow	http://localhost:8080	`airflow` / `airflow`
pgAdmin	http://localhost:8085	`admin@admin.com` / `root`
PostgreSQL	`localhost:5432`	`root` / `root` / database `rush`

If you have port conflicts, stop the other containers before running the setup.

To stop: docker compose down. To fully clean up: ./teardown.sh.

Architecture¶

flowchart TD
    SZ["Stadt Zürich MIV counts"] -->|dlt backfill| PG["PostgreSQL"]
    OMA["Open-Meteo archive"] -->|dlt backfill| PG
    OMF["Open-Meteo forecast"] -->|dlt daily| PG

    AF["Airflow"] -->|rush_backfill| PG
    AF -->|rush_daily| PG
    AF -->|dbt build| DBT["dbt staging + marts"]
    DBT --> PG
    PG --> PGA["pgAdmin"]

    OSRM["OSRM (public demo)"] --> REC["recommender CLI"]
    OMF2["Open-Meteo forecast"] --> REC
    PG --> REC
    REC --> OUT["verdict + expected minutes"]

    PG -->|dlt to BigQuery| BQ["BigQuery"]
    BQ --> DBTC["dbt --target prod"]
    DBTC --> BQ
    BQ --> UI["Streamlit on Cloud Run"]

    subgraph local ["Docker Compose (local)"]
        AF
        DBT
        PG
        PGA
        REC
    end

    subgraph cloud ["GCP (Terraform)"]
        BQ
        DBTC
        UI
    end

Tech Stack¶

Component	Tool	Purpose
Backfill + ingestion	dlt + Python	2-year traffic + weather load, daily refresh
Storage	PostgreSQL 18	Local warehouse
Cloud storage	GCS + BigQuery	Data lake + cloud warehouse
Admin	pgAdmin	DB UI
Transformation	dbt-core	staging views + mart tables (Postgres + BigQuery)
Routing	OSRM (public demo)	driving route between two coordinates
Orchestration	Airflow 2.10.5	`rush_backfill` (manual), `rush_daily` (02:00 UTC)
Cloud orchestration	Cloud Run Jobs + Cloud Scheduler	`rush-ingest` (02:00 UTC), `rush-dbt` (03:00 UTC)
UI	Streamlit on Cloud Run	live recommender at the demo URL
Containerization	Docker Compose	reproducible environment
Infrastructure	Terraform	GCS bucket, BigQuery datasets, Cloud Run jobs + service, Scheduler
Dependencies	uv + pyproject.toml	fast Python package management

Data Sources¶

Stadt Zürich MIV hourly counts — data.stadt-zuerich.ch. One CSV per year, 215+ counter stations across the city. Free, no key.

Weather — Open-Meteo. Archive endpoint for 2 years of hourly history at four bounding-box corners, forecast endpoint for the "right now" precipitation lookup. Free, no key.

Driving route — OSRM public demo. Returns the driving polyline and duration between two coordinates. No key.