Skip to content

6. Repository Requirements

Points: 5 — Source code, Docker configuration, orchestrator setup, README documentation, and professional engineering practices.

Repository Structure

rush/
  .github/
    workflows/
      docs.yml              # GitHub Pages deployment (this site)
  dags/
    rush_pipeline.py        # rush_backfill + rush_daily
  docs/                     # Peer review manual (MkDocs source)
  pipelines/
    backfill/
      backfill_traffic.py   # Stadt Zürich MIV counts (dlt -> PostgreSQL / BigQuery)
      backfill_weather.py   # Open-Meteo archive (dlt -> PostgreSQL / BigQuery)
    ingestion/
      weather.py            # Open-Meteo forecast (dlt -> PostgreSQL / BigQuery)
    recommend/
      route.py              # OSRM call + haversine fallback
      snap.py               # LV95 -> WGS84, counters near the polyline
      lookup.py             # read marts from Postgres or BigQuery
      timeline.py           # hour-by-hour expected counts for the UI chart
      weather.py            # forecast precip at a point
      recommend.py          # glue + CLI
    transformation/
      dbt/
        macros/
          day_of_week.sql       # portable dow extraction
          median.sql            # portable median
        models/
          staging/
            sources.yml                # source definitions
            stg_traffic__counts.sql    # add hour/dow/month keys
            stg_traffic__counters.sql  # distinct counter dim
            stg_weather__history.sql   # bucket precipitation
          marts/
            mart_traffic_baseline.sql  # avg per counter, h, dow, month
            mart_weather_effect.sql    # weather coefficient
        dbt_project.yml
        profiles.yml
    ui/
      app.py                # Streamlit recommender (Cloud Run service)
  scripts/
    setup-gcp.sh            # GCP project setup + Terraform
  terraform/
    main.tf                 # GCS bucket + BigQuery datasets
    cloud_run.tf            # Artifact Registry, Cloud Run jobs/service, Scheduler
    variables.tf
    outputs.tf
  config.yaml               # single source of truth
  config.py                 # loader
  Dockerfile                # local dev container (Python 3.12 + uv + Airflow)
  Dockerfile.ingest         # Cloud Run Job image for ingest
  Dockerfile.dbt            # Cloud Run Job image for dbt
  Dockerfile.ui             # Cloud Run Service image for Streamlit
  docker-compose.yaml       # 7-service stack
  pyproject.toml
  setup.sh
  teardown.sh
  README.md

Checklist

Requirement Status Location
Source code Present pipelines/, dags/, config.py
Docker configuration Present Dockerfile, docker-compose.yaml
Orchestrator setup Present dags/rush_pipeline.py, Airflow services in docker-compose
Cloud pipeline Present Cloud Run Jobs + Cloud Scheduler, provisioned by terraform/cloud_run.tf
README documentation Present README.md with Quick Start, Tech Stack, Project Structure
Terraform IaC Present terraform/ with main.tf, variables.tf, outputs.tf
.env not committed Correct .env is in .gitignore; .env.example is provided
No hardcoded secrets Correct All credentials in config.yaml (local dev defaults)
Dependencies declared Present pyproject.toml with pinned dependency groups

Engineering Practices

Single source of truth for configuration. All settings live in config.yaml. The setup script generates .env for Docker Compose. Python code reads the YAML directly via config.py.

Separation of concerns. Ingestion, transformation, and orchestration are in separate directories with no circular dependencies. The DAG file imports nothing from the pipeline code — it runs scripts via BashOperator.

Dual-target dbt models. All SQL uses ANSI-compatible syntax and dbt macros so the same models run on both PostgreSQL (local) and BigQuery (cloud) without modification. See Transformation.

Reproducible environment. setup.sh handles everything from tool installation to Docker stack startup. A fresh clone on a machine with only Git and Docker can be fully operational in one command.

Clean teardown. teardown.sh removes all containers, volumes, images, and generated files. It optionally destroys GCP resources via Terraform.

README

The README.md covers:

  • Project description and motivation
  • One-liner quick start (bash <(curl ...))
  • Manual setup for macOS, Linux, and Windows (WSL)
  • What the setup script does
  • Service URLs after startup
  • Tech stack overview
  • Data sources with links
  • Full project structure tree
  • Configuration system
  • Development workflow (VS Code Dev Container and terminal-only)
  • Teardown instructions