GCP: Demo of testing Airflow, MLflow, XGBoost

Training a machine-learning model on a laptop is the easy part. The hard part starts the day after: who reruns the training when new data arrives? Where are the experiments recorded? How does a better model reach production without someone copying files around, and how do you know it is actually better? Those questions are what MLOps is about.

In this tutorial we will build, step by step, a small but complete MLOps platform on Google Cloud. The machine-learning model itself (an XGBoost classifier on the public UCI Bank Marketing dataset) is deliberately boring: it is only a vehicle. The real subject is everything around it: orchestrating, tracking, deploying, monitoring and reproducing an ML workflow.

The context: what we are building and why

Imagine a bank that wants to predict whether a customer will subscribe to a term deposit after a marketing call. A data scientist can train such a model in ten minutes. A team that has to retrain it every week, compare each new model with the one in production, serve predictions through an API, and let colleagues query it safely needs a platform. That platform is our project.

Objectives

By the end of this tutorial you will have:

An automated training pipeline (Airflow) that extracts data, validates it, preprocesses it, trains 3 XGBoost models and evaluates them.
An experiment tracker and model registry (MLflow) storing every run, metric and artifact.
A promotion rule that only puts a model into production if it is good enough and better than the current one.
A private prediction API (FastAPI on Cloud Run) and a dashboard (Streamlit) with Google sign-in and user / admin roles.
Infrastructure as code (Terraform) and a CI/CD pipeline (GitHub Actions) with no long-lived secret keys.
Observability through structured logs in Cloud Logging.

The big picture

Before typing any command, look at the whole map once. Do not try to memorize it: we will come back to each part when we build it.

Figure 1. Global architecture. Solid arrows are runtime calls, dashed arrows are deployments or artifact storage.

Read it from top to bottom. On the top row, code travels from a developer to GitHub Actions to Artifact Registry. In the middle row, three Cloud Run workloads run our code: the dashboard, the API and a Job that does the actual machine learning. Composer (managed Airflow) only gives orders to the Job. On the bottom row sit the state stores: MLflow (backed by Cloud SQL and Cloud Storage), the data bucket and BigQuery.

Five design decisions worth understanding first

Every choice below solves a real problem you would otherwise hit later. Knowing the why will make the many commands that follow feel logical instead of magical.

Airflow orchestrates, it does not compute. Each task of the DAG simply triggers the same Cloud Run Job with a different sub-command (python -m src.cli <step>). Result: Composer needs no ML libraries, and you can run any step by hand without Airflow.
Deploying code is not deploying a model. GitHub Actions ships application code. A new model goes live when the pipeline registers a better version and moves the MLflow alias champion; the DAG then calls POST /reload on the API.
A promotion guard. The best of 3 runs (by ROC-AUC) must reach MIN_ROC_AUC (default 0.75) and beat the current champion.
Private services and no long-lived keys. MLflow and the API are private Cloud Run services: callers send Google ID tokens and only the right service accounts get roles/run.invoker. GitHub Actions authenticates through Workload Identity Federation instead of a JSON key.
Role-based dashboard. Visitors sign in with Google. Normal users only see the Predictions form; admins also see the full dashboard and a Pipeline admin tab.

The repository layout

This is the project we will work in. Keep it in mind: each step touches one or two of these folders.

.
├── airflow/dags/xgboost_pipeline.py   # DAG xgboost_training_pipeline
├── src/                               # pipeline code (data, features, models, utils) + CLI
├── training/Dockerfile                # image of the Cloud Run Job
├── mlflow_server/                     # MLflow tracking server image (Cloud SQL + GCS)
├── api/                               # FastAPI prediction service
├── dashboard/                         # Streamlit app
├── infrastructure/terraform/          # APIs, GCS, BigQuery, AR, SAs/IAM, Cloud SQL, WIF
├── tests/  scripts/  .github/workflows/{ci,cd}.yml

The plan: a progressive route

We will not jump into the cloud straight away. The tutorial is ordered so that every stage is validated before the next one, and so that the expensive components are created as late as possible and destroyed as soon as we are done.

Figure 2. The route. Each stage ends with a verification so you never build on something broken.

Cost warning. Cloud Composer costs roughly $10+/day and Cloud SQL roughly $10/month while they exist, even when idle. Cloud Run scales to zero. That is why Composer is created only in Step 11 and why cleanup (Step 15) is part of the tutorial, not an afterthought. Set a budget alert now in Billing → Budgets & alerts.

Contents

Prepare your workstation
Prove it works locally (free)
One file for all variables
Create the GCP project
Provision infrastructure with Terraform
Upload the dataset
Build and push the Docker images
Deploy MLflow
Create the pipeline job
Milestone 1: run the pipeline by hand
Milestone 2: API and dashboard
Google sign-in and roles
Automate with Airflow on Composer
CI/CD with GitHub Actions
Observability with Cloud Logging
The final check
Cleanup
Troubleshooting

Step 0. Prepare your workstation

Everything in this tutorial is written for Windows PowerShell. You need a Google account with billing enabled, plus a handful of tools:

Tool	Check with
Google Cloud CLI (`gcloud`, includes `bq`)	`gcloud --version`
Terraform ≥ 1.5	`terraform -version`
Docker Desktop (WSL 2 backend)	`docker --version`
Python 3.11 (3.13 is not supported by the pinned versions)	`py -3.11 --version`
Git	`git --version`
GitHub CLI	`gh --version`

The quickest way to install all of them is winget. We pass --source winget to avoid the Microsoft Store source, which sometimes fails with a certificate error.

winget install --id Python.Python.3.11 --source winget
winget install --id Google.CloudSDK --source winget
winget install --id Hashicorp.Terraform --source winget
winget install --id Docker.DockerDesktop --source winget
winget install --id Git.Git --source winget
winget install --id GitHub.cli --source winget

Close and reopen PowerShell afterwards. If you already have Python 3.13, keep it: the py launcher lets you pick 3.11 per project (py -0 lists both).

Four habits that will save you hours

Run commands from the project root (the folder containing requirements.txt) unless a step says otherwise.
New PowerShell window? Run . .\env.ps1 first (dot, space, dot-backslash). Variables vanish when a window closes. If a command fails with expected one argument or an empty value, this is almost always the reason.
A backtick ` at the very end of a line continues the command. Nothing, not even a space, may follow it.
In PowerShell curl is an alias of another command, so we use curl.exe.

Certificate errors? If pip, gcloud, terraform, git or docker build fail with unable to get local issuer certificate, your antivirus or network is re-signing HTTPS traffic. Jump to the first row of the Troubleshooting table before continuing.

Step 1. Prove it works locally first (free)

A golden rule of cloud work: never debug in the cloud what you can debug on your laptop. The cloud is slower, costs money, and gives worse error messages. So we begin with a local virtual environment, the linter and the tests.

py -3.11 -m venv .venv
.venv\Scripts\Activate.ps1
python --version          # must print 3.11.x
python -m pip install --upgrade pip
pip install -r requirements-dev.txt

If PowerShell refuses to run Activate.ps1, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned once and retry.

Our project has a folder named airflow/, which confuses the linter (ruff) into thinking the Airflow library is local code and reporting an import-order error. We tell ruff that airflow is third-party, then lint and test:

Add-Content pyproject.toml "`n[tool.ruff.lint.isort]`nknown-third-party = [`"airflow`"]"
ruff check . --fix
ruff check .
pytest -q

You should see All checks passed! and 13 passed. The tests cover data validation, preprocessing (leakage column removed, reproducible stratified split), a tiny XGBoost training with MLflow logging, the threshold gate and champion promotion, and the API (health, predict, input validation, no-model case).

Optional but recommended: the smoke test

This runs the entire ML flow (clean, split, train 3 XGBoost runs, evaluate, register) against a throw-away SQLite MLflow. It is the cheapest way to see what the cloud pipeline will do later.

python scripts\download_data.py
$env:MIN_ROC_AUC = "0.5"
python scripts\local_smoke_test.py

It must end with OK. To browse the 3 runs, go into the folder it created and use a relative path (absolute SQLite paths are easy to get wrong on Windows):

$SMOKE = Get-ChildItem $env:TEMP -Directory -Filter "mlops-smoke-*" | Sort-Object LastWriteTime -Descending | Select-Object -First 1
cd $SMOKE.FullName
mlflow ui --backend-store-uri sqlite:///mlflow.db

Open http://localhost:5000, look around, stop with Ctrl+C, then cd back to your project folder. Without data\bank-additional-full.csv the smoke test uses synthetic data.

A line-ending trap to fix now

Our MLflow container starts from a shell script. If Windows saved it with CRLF line endings, Linux will fail with the cryptic exec /entrypoint.sh: no such file or directory. Fix it before you forget:

Set-Content .gitattributes "*.sh text eol=lf" -Encoding ascii
(Get-Content mlflow_server\entrypoint.sh -Raw) -replace "`r`n","`n" | Set-Content -NoNewline mlflow_server\entrypoint.sh

Step 2. One file for all variables

We are about to type dozens of commands that mention the project ID, the region, the bucket, the registry and several service accounts. Typing these by hand is how typos are born. Instead, every value lives in one file, env.ps1, and every command uses variables. A typo can then happen in exactly one place.

Variable	Meaning	Value
`$PROJECT_ID`	GCP project ID	`mlops-demo-gcp`
`$REGION`	GCP region	`europe-west1`
`$GITHUB_REPO`	GitHub repository (`OWNER/REPO`)	`8-chems/gcp-mlops-demo`
`$BILLING`	Your billing account ID	the only value you must fill in yourself
`$MY_EMAIL`	Your Google account	detected automatically
`$BUCKET`	Data bucket created by Terraform	`mlops-demo-gcp-bucket`
`$REGISTRY`	Artifact Registry path	`europe-west1-docker.pkg.dev/mlops-demo-gcp/mlops`
`$SQL_CONN`	Cloud SQL connection name	`mlops-demo-gcp:europe-west1:mlflow-pg`

First, look up your billing account ID:

gcloud billing accounts list

Now create the file. Replace XXXXXX-XXXXXX-XXXXXX by your billing account ID (the only value to edit) and run the block. Notice the helper Check-Env: it tells you which variables are missing, which is precisely the problem you will meet each time you open a new window.

@'
# ---- edit only $BILLING ----
$BILLING      = "XXXXXX-XXXXXX-XXXXXX"

# ---- project settings ----
$PROJECT_ID   = "mlops-demo-gcp"
$REGION       = "europe-west1"
$GITHUB_REPO  = "8-chems/gcp-mlops-demo"

# ---- derived names (do not edit) ----
$MY_EMAIL     = gcloud config get-value account 2>$null
$BUCKET       = "$PROJECT_ID-bucket"
$REGISTRY     = "$REGION-docker.pkg.dev/$PROJECT_ID/mlops"
$SQL_CONN     = "${PROJECT_ID}:${REGION}:mlflow-pg"
$JOB          = "mlops-pipeline-job"
$COMPOSER_ENV = "mlops-demo"
$SA_MLFLOW    = "sa-mlflow@$PROJECT_ID.iam.gserviceaccount.com"
$SA_TRAINING  = "sa-training@$PROJECT_ID.iam.gserviceaccount.com"
$SA_API       = "sa-api@$PROJECT_ID.iam.gserviceaccount.com"
$SA_DASHBOARD = "sa-dashboard@$PROJECT_ID.iam.gserviceaccount.com"
$SA_AIRFLOW   = "sa-airflow@$PROJECT_ID.iam.gserviceaccount.com"
$CICD_SA      = "sa-cicd@$PROJECT_ID.iam.gserviceaccount.com"

function Check-Env([string[]]$Names) {
  $missing = $Names | Where-Object { -not (Get-Variable $_ -ValueOnly -ErrorAction SilentlyContinue) }
  if ($missing) { Write-Host "MISSING: $($missing -join ', ')  -> run: . .\env.ps1" -ForegroundColor Red }
  else { Write-Host "OK: all variables set" -ForegroundColor Green }
}
'@ | Set-Content env.ps1 -Encoding ascii

Add-Content .gitignore "env.ps1"
Add-Content .gitignore "payload.json"

. .\env.ps1
Check-Env PROJECT_ID,REGION,GITHUB_REPO,BILLING,MY_EMAIL,BUCKET,REGISTRY,SQL_CONN
$PROJECT_ID; $REGION; $GITHUB_REPO; $MY_EMAIL; $BUCKET; $REGISTRY; $SQL_CONN

$MY_EMAIL is only filled after gcloud auth login (next step); if it is empty now, run . .\env.ps1 again after logging in. Later steps will append WIF_PROVIDER, MLFLOW_URL and API_URL to this same file, so it grows with your platform.

Step 3. Create the GCP project and link billing

Everything in Google Cloud lives inside a project. We log in twice: once for the gcloud CLI and once for "application default credentials", which Terraform and Python libraries use.

. .\env.ps1
gcloud auth login
gcloud auth application-default login
. .\env.ps1                      # refresh $MY_EMAIL now that you are logged in
gcloud projects create $PROJECT_ID
gcloud config set project $PROJECT_ID
gcloud billing projects link $PROJECT_ID --billing-account=$BILLING
gcloud auth application-default set-quota-project $PROJECT_ID
gcloud projects list

If projects create says the project already exists (and it is yours), just continue.
If the ID is taken globally, change $PROJECT_ID once in env.ps1 and run . .\env.ps1: every other name derives from it.
If linking fails with a quota error, your billing account has too many linked projects (see Troubleshooting).

Step 4. Provision the infrastructure with Terraform

We could click through the Cloud Console for an hour, or describe everything in code and let Terraform do it in ten minutes, reproducibly. That is infrastructure as code. Our Terraform creates:

Figure 3. What terraform apply creates in one go.

The terraform.tfvars file is generated from your variables, so it always matches env.ps1. One small adjustment is needed: the Terraform code names the bucket <project_id>-mlops-demo, but our convention is <project_id>-bucket, so we rename it with a text replacement and verify it.

. .\env.ps1
cd infrastructure\terraform
@"
project_id  = "$PROJECT_ID"
region      = "$REGION"
github_repo = "$GITHUB_REPO"
"@ | Set-Content terraform.tfvars -Encoding ascii
Get-Content terraform.tfvars

# The Terraform code names the bucket "<project_id>-mlops-demo". Rename it to "<project_id>-bucket":
(Get-Content main.tf -Raw).Replace('${var.project_id}-mlops-demo','${var.project_id}-bucket') | Set-Content main.tf -NoNewline -Encoding ascii
Select-String -Path main.tf -Pattern "bucket_name ="

terraform init
terraform apply

Select-String must show bucket_name = "${var.project_id}-bucket". Type yes when asked. Expect 8 to 12 minutes (Cloud SQL is the slow part). If it fails with API not enabled, wait a minute and run terraform apply again: API activation takes a moment to propagate.

Already ran terraform apply before? With the old bucket name, Terraform will replace the empty bucket mlops-demo-gcp-mlops-demo by mlops-demo-gcp-bucket; re-upload the dataset (Step 5) if you had already done so. With a different repository name, fix $GITHUB_REPO in env.ps1, regenerate terraform.tfvars and apply again: the authorized GitHub repository is updated in place, nothing is recreated.

Now read the outputs, return to the project root, and save the Workload Identity provider name, which GitHub Actions will need in Step 12:

$WIF_PROVIDER = terraform output -raw wif_provider
terraform output -raw bucket
terraform output -raw cloudsql_connection_name
cd ..\..
Add-Content env.ps1 "`$WIF_PROVIDER = '$WIF_PROVIDER'"
$WIF_PROVIDER

Verify: the printed bucket must equal $BUCKET, the connection name must equal $SQL_CONN, and $WIF_PROVIDER must look like projects/123456789/locations/global/workloadIdentityPools/github-pool/providers/github-provider.

Step 5. Upload the dataset to Cloud Storage

The first task of the pipeline, extract, reads the raw CSV from the bucket. Let us put it there. (Skip the first line if data\bank-additional-full.csv already exists.)

. .\env.ps1
python scripts\download_data.py
gcloud storage cp data\bank-additional-full.csv "gs://$BUCKET/raw/bank-additional-full.csv"
gcloud storage ls "gs://$BUCKET/raw/"

The pipeline samples SAMPLE_ROWS=15000 of the roughly 41,000 rows with a fixed seed (42), which keeps runs fast, cheap and reproducible.

Step 6. Build and push the Docker images

Cloud Run runs containers, so each of our four components needs an image: mlflow, training, api and dashboard. Start Docker Desktop, wait until the engine is running, and check that docker info prints server information instead of an error. We build with --platform linux/amd64 because Cloud Run runs on that architecture, whatever your laptop is.

. .\env.ps1
Check-Env REGISTRY,REGION
gcloud auth configure-docker "$REGION-docker.pkg.dev" --quiet

docker build --platform linux/amd64 -t "$REGISTRY/mlflow:latest" mlflow_server
docker build --platform linux/amd64 -t "$REGISTRY/training:latest" -f training/Dockerfile .
docker build --platform linux/amd64 -t "$REGISTRY/api:latest" -f api/Dockerfile .
docker build --platform linux/amd64 -t "$REGISTRY/dashboard:latest" -f dashboard/Dockerfile .

foreach ($i in "mlflow","training","api","dashboard") {
  docker push "$REGISTRY/${i}:latest"
}

gcloud artifacts docker images list $REGISTRY

Verify: the last command must list the four images.

Step 7. Deploy the MLflow tracking server

MLflow is the memory of our platform. It records every training run (parameters, metrics, plots) and holds the model registry where the champion alias lives. Metadata goes to Cloud SQL; big files (models, plots) go to Cloud Storage. The service itself runs on Cloud Run, privately.

Figure 4. MLflow separates metadata (SQL) from artifacts (object storage).

Deploy it. Note the secret: the database password was created by Terraform in Secret Manager, and Cloud Run injects it as DB_PASSWORD. We never write it down.

. .\env.ps1
Check-Env PROJECT_ID,REGION,REGISTRY,BUCKET,SQL_CONN,SA_MLFLOW

$MLFLOW_ENV = "DB_USER=mlflow,DB_NAME=mlflow,CLOUDSQL_CONNECTION_NAME=${SQL_CONN},ARTIFACT_ROOT=gs://${BUCKET}/mlflow-artifacts"

gcloud run deploy mlflow --region $REGION --image "$REGISTRY/mlflow:latest" `
  --service-account $SA_MLFLOW `
  --no-allow-unauthenticated `
  --add-cloudsql-instances $SQL_CONN `
  --set-env-vars $MLFLOW_ENV `
  --set-secrets "DB_PASSWORD=mlflow-db-password:latest" `
  --memory 1Gi --cpu 1 --min-instances 0 --max-instances 2

Only after the deployment succeeded, read the service URL and save it for later steps:

$MLFLOW_URL = gcloud run services describe mlflow --region $REGION --format "value(status.url)"
Add-Content env.ps1 "`$MLFLOW_URL = '$MLFLOW_URL'"
$MLFLOW_URL

Because the service is private, opening its URL in a browser gives 403. To look at it yourself, grant your account the invoker role and use the authenticated proxy, then open http://localhost:5000 (Ctrl+C to stop):

gcloud run services add-iam-policy-binding mlflow --region $REGION --member "user:$MY_EMAIL" --role roles/run.invoker
gcloud run services proxy mlflow --region $REGION --port 5000

Step 8. Create the pipeline Cloud Run Job and wire the permissions

Now the worker. A Cloud Run Job is a container that runs to completion and stops, the perfect fit for "run this step of the pipeline". We create it once; every step (and later, every Airflow task) will execute it with different arguments. Its environment variables tell it where the bucket, BigQuery dataset and MLflow live, and what quality bar a model must clear (MIN_ROC_AUC=0.75).

. .\env.ps1
Check-Env PROJECT_ID,REGION,REGISTRY,BUCKET,MLFLOW_URL,SA_TRAINING,JOB

$JOB_ENV = "GCP_PROJECT_ID=${PROJECT_ID},GCP_REGION=${REGION},GCS_BUCKET=${BUCKET},BQ_DATASET=mlops_demo,MLFLOW_TRACKING_URI=${MLFLOW_URL},MIN_ROC_AUC=0.75,SAMPLE_ROWS=15000"

gcloud run jobs create $JOB --region $REGION `
  --image "$REGISTRY/training:latest" `
  --service-account $SA_TRAINING `
  --set-env-vars $JOB_ENV `
  --memory 2Gi --cpu 1 --max-retries 0 --task-timeout 1800

Since MLflow is private, every service that talks to it needs roles/run.invoker. We grant it to the training, API and dashboard service accounts:

foreach ($sa in $SA_TRAINING,$SA_API,$SA_DASHBOARD) {
  gcloud run services add-iam-policy-binding mlflow --region $REGION `
    --member "serviceAccount:$sa" --role roles/run.invoker
}

Step 9. Milestone 1: run the pipeline by hand

Before adding Airflow, we prove the pipeline itself works by driving it from PowerShell. This is a deliberate strategy: if the manual run works, any later failure in Airflow is an orchestration problem, not an ML problem. Each step is one execution of the Job:

Figure 5. The seven steps executed by the Cloud Run Job.

. .\env.ps1
$RUN_ID = "manual1"
foreach ($s in "extract","validate","preprocess","load","train","evaluate","register") {
  Write-Host "=== $s ==="
  gcloud run jobs execute $JOB --region $REGION "--args=${s},--run-id,${RUN_ID}" --wait
  if ($LASTEXITCODE -ne 0) { Write-Host "Step $s failed, stopping." -ForegroundColor Red; break }
}

What success looks like: processed/clean.csv in the bucket; table mlops_demo.training_data in BigQuery; 3 runs in the MLflow experiment bank-marketing-xgboost; and model bank-marketing-xgb version 1 with alias champion. Check the data split in BigQuery:

bq query --use_legacy_sql=false "SELECT split, COUNT(*) AS n, AVG(target) AS rate FROM mlops_demo.training_data GROUP BY split"

A failure can be a feature. If evaluate fails with ThresholdNotMet, the model scored below MIN_ROC_AUC and was deliberately not registered. If any other step fails, read the job logs (Step 13): the failing step logs pipeline_failure with the error.

Step 10. Milestone 2: deploy the API and the dashboard

We now have a champion model in the registry. Let us serve it. The API is private (only authorized callers), loads the champion model from MLflow, and optionally logs every prediction to BigQuery.

. .\env.ps1
Check-Env PROJECT_ID,REGION,REGISTRY,MLFLOW_URL,SA_API,SA_DASHBOARD,SA_AIRFLOW

$API_ENV = "GCP_PROJECT_ID=${PROJECT_ID},GCP_REGION=${REGION},MLFLOW_TRACKING_URI=${MLFLOW_URL},LOG_PREDICTIONS_TO_BQ=true"

gcloud run deploy api --region $REGION --image "$REGISTRY/api:latest" `
  --service-account $SA_API `
  --no-allow-unauthenticated --min-instances 0 --memory 1Gi `
  --set-env-vars $API_ENV

$API_URL = gcloud run services describe api --region $REGION --format "value(status.url)"
Add-Content env.ps1 "`$API_URL = '$API_URL'"
$API_URL

Three callers need to reach this private API: the dashboard, Airflow (to trigger /reload) and you.

foreach ($sa in $SA_DASHBOARD,$SA_AIRFLOW) {
  gcloud run services add-iam-policy-binding api --region $REGION `
    --member "serviceAccount:$sa" --role roles/run.invoker
}
gcloud run services add-iam-policy-binding api --region $REGION --member "user:$MY_EMAIL" --role roles/run.invoker

Now deploy the dashboard. Unlike the API it is publicly reachable at the Cloud Run level (the app itself will handle login in Step 10b), and it mounts a secret as /app/.streamlit/secrets.toml (we create that secret in Step 10b §3, before this first deploy works end to end):

$DASH_ENV = "GCP_PROJECT_ID=${PROJECT_ID},GCP_REGION=${REGION},MLFLOW_TRACKING_URI=${MLFLOW_URL},API_URL=${API_URL}"

gcloud run deploy dashboard --region $REGION --image "$REGISTRY/dashboard:latest" `
  --service-account $SA_DASHBOARD `
  --allow-unauthenticated --min-instances 0 --memory 1Gi `
  --set-env-vars $DASH_ENV `
  --update-secrets /app/.streamlit/secrets.toml=dashboard-auth-secrets:latest

gcloud run services describe dashboard --region $REGION --format "value(status.url)"

Talk to the API from the command line

The private API expects a Google identity token. We ask the model to load the champion, then check health:

$TOKEN = gcloud auth print-identity-token
curl.exe -s -X POST "$API_URL/reload" -H "Authorization: Bearer $TOKEN" -d "{}"
curl.exe -s "$API_URL/health" -H "Authorization: Bearer $TOKEN"

(The -d "{}" is not decoration: a POST without a body has no Content-Length header and Google's front end answers 411 Length Required.) Then create a request file and ask for a prediction:

@'
{
  "age": 40, "job": "admin.", "marital": "married", "education": "high.school",
  "default": "no", "housing": "yes", "loan": "no", "contact": "cellular", "month": "may",
  "day_of_week": "mon", "campaign": 2, "pdays": 999, "previous": 0, "poutcome": "nonexistent",
  "emp_var_rate": 1.1, "cons_price_idx": 93.9, "cons_conf_idx": -36.4, "euribor3m": 4.8,
  "nr_employed": 5191
}
'@ | Set-Content payload.json -Encoding ascii

curl.exe -s -X POST "$API_URL/predict" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" --data "@payload.json"

The token lasts about an hour; rerun $TOKEN = gcloud auth print-identity-token to refresh it.

Security note. At this stage the dashboard is open to anyone who has the URL. Do not share the link yet: the next step adds Google sign-in and roles. The API and MLflow stay private regardless.

The basic dashboard has four tabs: Data overview (BigQuery), Model experiments (MLflow), Model performance (champion artifacts) and Predictions (calls the API).

Step 10b. Add Google sign-in and roles

A dashboard that lets anybody trigger pipelines is a liability. We want three outcomes depending on who shows up:

Figure 6. Who sees what. Roles come from e-mail allow-lists.

The rules are simple: e-mails in ADMIN_EMAILS are admins; if ALLOWED_EMAILS is empty, any verified Google account is a normal user, otherwise only the listed e-mails are. One principle is crucial: hiding tabs is not security. Every admin function re-checks the role in Python (require_admin()), so even a crafted request cannot bypass it. Every prediction is stored in BigQuery with the submitter's e-mail (submitted_by).

1. Add the `submitted_by` column (Terraform)

We patch the BigQuery schema in main.tf by inserting one nullable column, then apply. Because the column is nullable, Terraform updates the table in place.

. .\env.ps1
cd infrastructure\terraform
$old = '{ name = "features", type = "STRING", mode = "NULLABLE" },'
$new = $old + "`n    " + '{ name = "submitted_by", type = "STRING", mode = "NULLABLE" },'
(Get-Content main.tf -Raw).Replace($old, $new) | Set-Content main.tf -NoNewline -Encoding ascii
Select-String -Path main.tf -Pattern "submitted_by"
terraform apply
cd ..\..

Select-String must show exactly one line.

2. Create the Google OAuth client (in the console)

APIs & Services → OAuth consent screen: choose External, fill in the app name and your support e-mail.
Set the publishing status to In production. While it is "Testing", only listed test users can sign in. The default scopes (openid, email, profile) are the ones Streamlit requests.
APIs & Services → Credentials → Create credentials → OAuth client ID → Web application.
Under Authorized redirect URIs, add the exact output of this command:

$DASH_URL = gcloud run services describe dashboard --region $REGION --format "value(status.url)"
"$DASH_URL/oauth2callback"

Copy the client ID and client secret.

3. Store the sign-in settings in Secret Manager

The client secret must never sit in the repository or in an environment variable. A helper script generates a random cookie secret, writes a temporary file, uploads it to Secret Manager, deletes the local file and grants the dashboard service account access, all in one go:

. .\env.ps1
$DASH_URL = gcloud run services describe dashboard --region $REGION --format "value(status.url)"

.\scripts\setup_auth_secret.ps1 `
    -GoogleClientId     "PASTE_CLIENT_ID" `
    -GoogleClientSecret "PASTE_CLIENT_SECRET" `
    -DashUrl            $DASH_URL `
    -SaDashboard        $SA_DASHBOARD

The script creates (or updates) the secret dashboard-auth-secrets with this structure:

[auth]
redirect_uri  = "https://<dash-url>/oauth2callback"
cookie_secret = "<random-32-byte-base64>"

[auth.default]
client_id           = "<your-client-id>"
client_secret       = "<your-client-secret>"
server_metadata_url = "https://accounts.google.com/.well-known/openid-configuration"

Important. The section must be [auth.default], not [auth.google]. st.login() called without arguments looks for the default provider; any other structure produces a 500 or "missing keys" error. To rotate credentials later, simply rerun the script: it adds a new secret version.

4. Update the code

First the dependencies. Streamlit's login needs Authlib, and the admin tab needs the Cloud Run client. Once auth is configured, Streamlit enables CORS/XSRF protection by itself, so we remove the two flags from the Dockerfile.

@'
streamlit>=1.42.0
Authlib>=1.3.2
google-cloud-run>=0.10.0
pandas==2.2.3
numpy==1.26.4
pyarrow==18.1.0
db-dtypes>=1.3.0
mlflow-skinny==2.22.0
google-cloud-bigquery>=3.25.0
google-auth>=2.30.0
requests>=2.32.0
'@ | Set-Content dashboard\requirements.txt -Encoding ascii

(Get-Content dashboard\Dockerfile -Raw).Replace(' --server.enableCORS=false --server.enableXsrfProtection=false', '') | Set-Content dashboard\Dockerfile -NoNewline -Encoding ascii
New-Item dashboard\__init__.py -ItemType File -Force | Out-Null

API: replace api\main.py by the new main.py from the project; it reads the X-Submitted-By header and stores it in submitted_by.

Then three dashboard files. Read them in this order: auth.py answers "who are you and what may you do?", admin.py contains the privileged actions, and app.py assembles the interface.

dashboard\auth.py (new): sign-in and roles

"""Google sign-in (Streamlit OIDC) and role resolution (user / admin)."""
import os

import streamlit as st


def _emails(var: str) -> set[str]:
    return {e.strip().lower() for e in os.getenv(var, "").split(",") if e.strip()}


def require_login() -> str:
    """Show the sign-in page until the visitor is signed in; return their verified e-mail."""
    if not st.user.is_logged_in:
        st.title("MLOps demo - Bank Marketing")
        st.write("Sign in with your Google account to continue.")
        st.button("Sign in with Google", on_click=st.login)
        st.stop()
    email = (getattr(st.user, "email", "") or "").lower()
    if not email or not getattr(st.user, "email_verified", False):
        st.error("Your Google e-mail is missing or not verified.")
        st.button("Sign out", on_click=st.logout)
        st.stop()
    return email


def role_of(email: str) -> str | None:
    """'admin' if listed in ADMIN_EMAILS; 'user' if allowed; None if not allowed.

    ALLOWED_EMAILS empty means: any verified Google account is a normal user."""
    if email in _emails("ADMIN_EMAILS"):
        return "admin"
    allowed = _emails("ALLOWED_EMAILS")
    return "user" if not allowed or email in allowed else None


def require_admin() -> str:
    """Call this INSIDE every admin function, not only when drawing the UI."""
    email = (getattr(st.user, "email", "") or "").lower()
    if not st.user.is_logged_in or role_of(email) != "admin":
        raise PermissionError("Admin only")
    return email

dashboard\admin.py (new): run jobs and control Airflow

"""Admin actions: run the Cloud Run Job and control the Airflow DAG (admins only)."""
import os
from datetime import datetime, timezone

import google.auth
from google.auth.transport.requests import AuthorizedSession
from google.cloud import run_v2

from dashboard.auth import require_admin
from src.config import get_settings
from src.utils.logging import log_event

DAG_ID = "xgboost_training_pipeline"


def run_job_step(step: str, run_id: str) -> str:
    """Start one execution of the pipeline job for a single step; returns the operation name."""
    admin = require_admin()
    s = get_settings()
    job = os.getenv("PIPELINE_JOB_NAME", "mlops-pipeline-job")
    name = f"projects/{s.project_id}/locations/{s.region}/jobs/{job}"
    override = run_v2.RunJobRequest.Overrides.ContainerOverride(args=[step, "--run-id", run_id])
    request = run_v2.RunJobRequest(
        name=name, overrides=run_v2.RunJobRequest.Overrides(container_overrides=[override])
    )
    operation = run_v2.JobsClient().run_job(request=request)
    log_event("admin_job_triggered", component="dashboard", user=admin, step=step, run_id=run_id)
    return operation.operation.name


def _airflow():
    base = os.getenv("AIRFLOW_URL", "").rstrip("/")
    if not base:
        raise RuntimeError("AIRFLOW_URL is not configured (create Composer first, see Step 11).")
    creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
    return AuthorizedSession(creds), base, os.getenv("AIRFLOW_API", "v1")


def trigger_dag() -> dict:
    admin = require_admin()
    session, base, version = _airflow()
    body: dict = {"conf": {}}
    if version == "v2":  # Airflow 3 requires a logical date
        body["logical_date"] = datetime.now(timezone.utc).isoformat()
    response = session.post(f"{base}/api/{version}/dags/{DAG_ID}/dagRuns", json=body, timeout=90)
    response.raise_for_status()
    log_event("admin_dag_triggered", component="dashboard", user=admin)
    return response.json()


def set_dag_paused(paused: bool) -> None:
    admin = require_admin()
    session, base, version = _airflow()
    response = session.patch(
        f"{base}/api/{version}/dags/{DAG_ID}",
        params={"update_mask": "is_paused"},
        json={"is_paused": paused},
        timeout=90,
    )
    response.raise_for_status()
    log_event("admin_dag_paused", component="dashboard", user=admin, paused=paused)

dashboard\app.py (replaces the existing file): the interface

"""Streamlit dashboard: Google sign-in, role-based tabs (user / admin)."""
import os
import tempfile

import pandas as pd
import requests
import streamlit as st
from mlflow.tracking import MlflowClient

from dashboard.auth import require_login, role_of
from src.config import get_settings
from src.features.preprocessing import FEATURES, TARGET
from src.utils.auth import auth_headers, configure_mlflow_auth
from src.utils.logging import log_event

st.set_page_config(page_title="MLOps demo - Bank Marketing", layout="wide")
S = get_settings()
API_URL = os.getenv("API_URL", "http://localhost:8080").rstrip("/")
AIRFLOW_URL = os.getenv("AIRFLOW_URL", "").rstrip("/")

# ---- sign-in and role --------------------------------------------------------------------
email = require_login()
role = role_of(email)
if role is None:
    st.error("Access restricted. Ask an admin to add your e-mail.")
    st.button("Sign out", on_click=st.logout)
    st.stop()
st.sidebar.write(f"Signed in as **{email}** ({role})")
st.sidebar.button("Sign out", on_click=st.logout)

CHOICES = {
    "job": ["admin.", "blue-collar", "entrepreneur", "housemaid", "management", "retired",
            "self-employed", "services", "student", "technician", "unemployed", "unknown"],
    "marital": ["married", "single", "divorced", "unknown"],
    "education": ["basic.4y", "basic.6y", "basic.9y", "high.school", "illiterate",
                  "professional.course", "university.degree", "unknown"],
    "default": ["no", "yes", "unknown"], "housing": ["no", "yes", "unknown"],
    "loan": ["no", "yes", "unknown"], "contact": ["cellular", "telephone"],
    "month": ["jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec"],
    "day_of_week": ["mon", "tue", "wed", "thu", "fri"],
    "poutcome": ["nonexistent", "failure", "success"],
}


@st.cache_data(ttl=600)
def load_training_data() -> pd.DataFrame:
    from src.utils.bigquery import read_table

    return read_table(S.training_table_id, S.project_id)


def mlflow_client() -> MlflowClient:
    configure_mlflow_auth()
    return MlflowClient(tracking_uri=S.mlflow_tracking_uri)


@st.cache_data(ttl=120)
def load_runs() -> pd.DataFrame:
    client = mlflow_client()
    exp = client.get_experiment_by_name(S.experiment_name)
    if exp is None:
        return pd.DataFrame()
    runs = client.search_runs([exp.experiment_id], order_by=["attributes.start_time DESC"],
                              max_results=100)
    return pd.DataFrame([{
        "run": r.info.run_name, "run_id": r.info.run_id[:8],
        "pipeline_run_id": r.data.tags.get("pipeline_run_id"),
        **{k: r.data.params.get(k) for k in ("max_depth", "learning_rate", "n_estimators")},
        **{k: round(v, 4) for k, v in r.data.metrics.items()},
    } for r in runs])


st.title("MLOps demo - Bank Marketing (XGBoost)")
names = ["Predictions"] if role == "user" else [
    "Data overview", "Model experiments", "Model performance", "Predictions", "Pipeline admin"]
T = dict(zip(names, st.tabs(names), strict=True))

if "Data overview" in T:
    with T["Data overview"]:
        try:
            df = load_training_data()
            c1, c2, c3, c4 = st.columns(4)
            c1.metric("Rows", f"{len(df):,}")
            c2.metric("Features", len(FEATURES))
            c3.metric("Missing values", int(df.isna().sum().sum()))
            c4.metric("Positive rate", f"{df[TARGET].mean():.1%}")
            st.subheader("Target distribution")
            st.bar_chart(df[TARGET].value_counts().rename({0: "no", 1: "yes"}))
            st.subheader("Data quality")
            st.write(f"Duplicate rows: {int(df.drop(columns=['split']).duplicated().sum())}")
            st.write(df["split"].value_counts().rename("rows per split"))
            st.dataframe(df.head(50))
        except Exception as exc:  # noqa: BLE001
            log_event("dashboard_error", severity="ERROR", component="dashboard", tab="data",
                      error=repr(exc))
            st.error(f"Could not load BigQuery data: {exc}")

if "Model experiments" in T:
    with T["Model experiments"]:
        try:
            runs = load_runs()
            if runs.empty:
                st.info("No MLflow runs yet. Execute the pipeline first.")
            else:
                st.dataframe(runs, use_container_width=True)
                st.bar_chart(runs.set_index("run")[["roc_auc", "f1", "accuracy"]])
        except Exception as exc:  # noqa: BLE001
            log_event("dashboard_error", severity="ERROR", component="dashboard",
                      tab="experiments", error=repr(exc))
            st.error(f"Could not reach MLflow: {exc}")

if "Model performance" in T:
    with T["Model performance"]:
        try:
            client = mlflow_client()
            mv = client.get_model_version_by_alias(S.model_name, S.model_alias)
            run = client.get_run(mv.run_id)
            st.write(f"Champion: **{S.model_name} v{mv.version}** (run `{mv.run_id[:8]}`)")
            metric_names = ["accuracy", "precision", "recall", "f1", "roc_auc"]
            for col, m in zip(st.columns(5), metric_names, strict=True):
                col.metric(m, f"{run.data.metrics.get(m, float('nan')):.3f}")
            with tempfile.TemporaryDirectory() as tmp:
                images = ["confusion_matrix", "roc_curve", "feature_importance"]
                for col, name in zip(st.columns(3), images, strict=True):
                    path = client.download_artifacts(mv.run_id, f"evaluation/{name}.png", tmp)
                    col.image(path, caption=name.replace("_", " "))
        except Exception as exc:  # noqa: BLE001
            log_event("dashboard_error", severity="ERROR", component="dashboard",
                      tab="performance", error=repr(exc))
            st.warning(f"No champion model available yet: {exc}")

with T["Predictions"]:
    st.write("Will this customer subscribe to a term deposit?")
    with st.form("predict"):
        c1, c2, c3 = st.columns(3)
        payload = {}
        for i, (name, opts) in enumerate(CHOICES.items()):
            payload[name] = [c1, c2, c3][i % 3].selectbox(name, opts)
        payload["age"] = c1.number_input("age", 16, 110, 40)
        payload["campaign"] = c2.number_input("campaign", 1, 60, 2)
        payload["pdays"] = c3.number_input("pdays (999 = never)", 0, 999, 999)
        payload["previous"] = c1.number_input("previous", 0, 20, 0)
        payload["emp_var_rate"] = c2.number_input("emp_var_rate", -4.0, 2.0, 1.1)
        payload["cons_price_idx"] = c3.number_input("cons_price_idx", 90.0, 96.0, 93.9)
        payload["cons_conf_idx"] = c1.number_input("cons_conf_idx", -55.0, -25.0, -36.4)
        payload["euribor3m"] = c2.number_input("euribor3m", 0.5, 5.5, 4.8)
        payload["nr_employed"] = c3.number_input("nr_employed", 4900.0, 5300.0, 5191.0)
        submitted = st.form_submit_button("Predict")
    if submitted:
        try:
            r = requests.post(
                f"{API_URL}/predict", json=payload,
                headers={**auth_headers(API_URL), "X-Submitted-By": email}, timeout=60)
            r.raise_for_status()
            out = r.json()
            label = "WILL subscribe" if out["prediction"] else "will NOT subscribe"
            st.success(f"Prediction: customer {label} (probability {out['probability']:.1%}, "
                       f"model v{out['model_version']}). Your submission was recorded.")
        except Exception as exc:  # noqa: BLE001
            log_event("api_connection_failure", severity="ERROR", component="dashboard",
                      error=repr(exc))
            st.error(f"API call failed: {exc}")

if "Pipeline admin" in T:
    with T["Pipeline admin"]:
        from dashboard import admin

        st.subheader("Cloud Run Job")
        step = st.selectbox("Job step", ["extract", "validate", "preprocess", "load",
                                          "train", "evaluate", "register"])
        run_id = st.text_input("Run id", value="manual")
        if st.button("Run this step"):
            try:
                st.success(f"Started: {admin.run_job_step(step, run_id)}")
            except Exception as exc:  # noqa: BLE001
                log_event("dashboard_error", severity="ERROR", component="dashboard",
                          tab="admin", error=repr(exc))
                st.error(f"Could not start the job: {exc}")

        st.subheader("Airflow")
        if AIRFLOW_URL:
            st.markdown(f"[Open the Airflow UI]({AIRFLOW_URL})")
        a, b, c = st.columns(3)
        try:
            if a.button("Trigger DAG run"):
                st.json(admin.trigger_dag())
            if b.button("Pause DAG"):
                admin.set_dag_paused(True)
                st.success("DAG paused")
            if c.button("Unpause DAG"):
                admin.set_dag_paused(False)
                st.success("DAG unpaused")
        except Exception as exc:  # noqa: BLE001
            log_event("dashboard_error", severity="ERROR", component="dashboard", tab="admin",
                      error=repr(exc))
            st.error(f"Airflow call failed: {exc}")

Before building, run ruff check .; it must pass, because CI runs it too.

5. Permissions for the admin features

The dashboard's service account needs two things: the right to run this job (scoped to the job, not the whole project) and the right to act as the job's own service account.

# run the Cloud Run Job (scoped to this job) and act as the job's service account
gcloud run jobs add-iam-policy-binding $JOB --region $REGION --member "serviceAccount:$SA_DASHBOARD" --role roles/run.developer
gcloud iam service-accounts add-iam-policy-binding $SA_TRAINING --member "serviceAccount:$SA_DASHBOARD" --role roles/iam.serviceAccountUser

6. Build, push and redeploy

We record who the admins are, rebuild the two changed images and redeploy. A subtle point: gcloud splits --set-env-vars on commas, so a list of e-mails would be torn apart. The prefix ^;^ changes the separator to a semicolon. We leave AIRFLOW_URL empty until Composer exists (Step 11 fills it).

. .\env.ps1
Add-Content env.ps1 "`$ADMIN_EMAILS = '$MY_EMAIL'"
. .\env.ps1                    # ADMIN_EMAILS: one e-mail, or "a@gmail.com,b@gmail.com"

docker build --platform linux/amd64 -t "$REGISTRY/api:latest" -f api/Dockerfile .
docker build --platform linux/amd64 -t "$REGISTRY/dashboard:latest" -f dashboard/Dockerfile .
docker push "$REGISTRY/api:latest"
docker push "$REGISTRY/dashboard:latest"

gcloud run deploy api --region $REGION --image "$REGISTRY/api:latest"

$AIRFLOW_URL = ""
$AIRFLOW_API = "v1"
$DASH_ENV = "^;^GCP_PROJECT_ID=${PROJECT_ID};GCP_REGION=${REGION};MLFLOW_TRACKING_URI=${MLFLOW_URL};API_URL=${API_URL};ADMIN_EMAILS=${ADMIN_EMAILS};AIRFLOW_URL=${AIRFLOW_URL};AIRFLOW_API=${AIRFLOW_API};PIPELINE_JOB_NAME=${JOB}"

gcloud run deploy dashboard --region $REGION --image "$REGISTRY/dashboard:latest" `
  --service-account $SA_DASHBOARD --allow-unauthenticated --min-instances 0 --memory 1Gi `
  --set-env-vars $DASH_ENV `
  --set-secrets "/app/.streamlit/secrets.toml=dashboard-auth-secrets:latest"

The service stays public at the Cloud Run level on purpose: the application itself does the Google sign-in. To restrict normal users to invited people, add ;ALLOWED_EMAILS=a@gmail.com,b@gmail.com to $DASH_ENV.

7. Keep CD from erasing these settings

This one bites everybody once. The CD workflow deploys the dashboard with --set-env-vars, which replaces all environment variables and would wipe ADMIN_EMAILS on the next push. Switch it to --update-env-vars (which only touches the variables it names; the secret mount is left alone):

$old = '--set-env-vars GCP_PROJECT_ID=${PROJECT_ID},GCP_REGION=${REGION},MLFLOW_TRACKING_URI=${MLFLOW_URL},API_URL=${API_URL}'
$new = $old.Replace('--set-env-vars', '--update-env-vars')
(Get-Content .github\workflows\cd.yml -Raw).Replace($old, $new) | Set-Content .github\workflows\cd.yml -NoNewline -Encoding ascii
Select-String -Path .github\workflows\cd.yml -Pattern "update-env-vars"

8. Test it like a user, then like an admin

Open the dashboard URL and click Sign in with Google with your admin account: all five tabs appear.
Open a private window and sign in with a second Google account: only Predictions appears.
Submit a prediction as the second account, then check who submitted it:

bq query --use_legacy_sql=false "SELECT timestamp, submitted_by, prediction FROM mlops_demo.prediction_results ORDER BY timestamp DESC LIMIT 5"

As admin, run a job step from Pipeline admin and look for admin_job_triggered in Logs Explorer (Step 13).

Step 11. Automate with Airflow on Cloud Composer

So far we were the orchestrator: a PowerShell loop calling the job seven times. In production nobody wants that. Apache Airflow models the workflow as a DAG (directed acyclic graph) of tasks with dependencies, retries, a schedule and a UI. On Google Cloud it is offered as Cloud Composer. Remember the cost warning: this is the expensive component, so we create it now and delete it in Step 15.

. .\env.ps1
Check-Env PROJECT_ID,REGION,API_URL,JOB,SA_AIRFLOW,COMPOSER_ENV

$COMPOSER_VARS = "GCP_PROJECT_ID=${PROJECT_ID},GCP_REGION=${REGION},PIPELINE_JOB_NAME=${JOB},API_URL=${API_URL}"

gcloud composer environments create $COMPOSER_ENV --location $REGION `
  --environment-size small `
  --service-account $SA_AIRFLOW `
  --env-variables $COMPOSER_VARS

This takes 20 to 30 minutes, a good moment for a coffee. Then import the DAG file and trigger a run:

gcloud composer environments storage dags import --environment $COMPOSER_ENV --location $REGION --source airflow\dags\xgboost_pipeline.py
Start-Sleep -Seconds 120
gcloud composer environments run $COMPOSER_ENV --location $REGION dags list
gcloud composer environments run $COMPOSER_ENV --location $REGION dags unpause "--" xgboost_training_pipeline
gcloud composer environments run $COMPOSER_ENV --location $REGION dags trigger "--" xgboost_training_pipeline
gcloud composer environments describe $COMPOSER_ENV --location $REGION --format "value(config.airflowUri)"

(The quoted "--" is passed to gcloud untouched; PowerShell would otherwise interpret it.) Open the Airflow UI URL printed by the last command to watch the DAG, which has 9 tasks:

Figure 7. The Airflow DAG xgboost_training_pipeline.

Trigger it a second time. This is where the promotion guard we designed at the start proves its worth:

Figure 8. The promotion guard: a model must be good enough and better than what is live.

This rule is what turns "automatic retraining" from a risk into a safety net: a bad training run can never silently replace a good model.

Connect the dashboard's Pipeline admin tab to Airflow

Check your Airflow major version first: Airflow 2 exposes REST API v1, Airflow 3 exposes v2. The commands below read the version, store the URL, allow the dashboard's service account to use Composer, and update the dashboard's environment.

gcloud composer environments describe $COMPOSER_ENV --location $REGION --format "value(config.softwareConfig.imageVersion)"
$AIRFLOW_URL = gcloud composer environments describe $COMPOSER_ENV --location $REGION --format "value(config.airflowUri)"
$AIRFLOW_API = "v1"          # use "v2" if the image version above starts with composer-3-airflow-3

gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:$SA_DASHBOARD" --role roles/composer.user
gcloud run services update dashboard --region $REGION --update-env-vars "AIRFLOW_URL=${AIRFLOW_URL},AIRFLOW_API=${AIRFLOW_API}"

The first API call creates an Airflow user for the dashboard's service account (default Airflow role Op). Detailed Airflow configuration (variables, connections, schedules) stays in the Airflow UI: give your admins roles/composer.user on the project so they can open it with their Google account. A DAG's schedule lives in the DAG file, so changing it means re-importing the file. For a simple recurring run without Composer, Cloud Scheduler can run the Cloud Run Job directly.

Step 12. CI/CD with GitHub Actions

Right now every deployment is a manual docker build and gcloud command. Let us automate it. The principle: CI checks every change; CD deploys what was merged. And thanks to Workload Identity Federation, GitHub proves its identity to Google Cloud without any stored key: only the provider name and a service-account e-mail are saved in GitHub.

Figure 9. The CI/CD loop. Code is deployed by CD; models are promoted by the pipeline.

We set up variables and secrets before the first push, because a push to main triggers CD, which needs them.

a) Log in and create the private repository (without pushing yet)

. .\env.ps1
Check-Env GITHUB_REPO,PROJECT_ID,REGION,WIF_PROVIDER,CICD_SA
gh auth login
git init
git add .
git commit -m "Initial MLOps demo"
git branch -M main
gh repo create $GITHUB_REPO --private --source=. --remote=origin

If the repository already exists, use git remote add origin "https://github.com/$GITHUB_REPO.git" instead of gh repo create.

b) Add the variables and secrets

gh variable set GCP_PROJECT_ID --repo $GITHUB_REPO --body $PROJECT_ID
gh variable set GCP_REGION --repo $GITHUB_REPO --body $REGION
gh secret set WIF_PROVIDER --repo $GITHUB_REPO --body $WIF_PROVIDER
gh secret set WIF_SERVICE_ACCOUNT --repo $GITHUB_REPO --body $CICD_SA

gh variable list --repo $GITHUB_REPO
gh secret list --repo $GITHUB_REPO

Prefer the web UI? Same values under Settings → Secrets and variables → Actions: Variables GCP_PROJECT_ID, GCP_REGION; Secrets WIF_PROVIDER, WIF_SERVICE_ACCOUNT.

c) Push, which triggers the first CD run

git push -u origin main
gh run list --repo $GITHUB_REPO --limit 5
gh run watch --repo $GITHUB_REPO

CD works because Steps 6 to 10 already created the services it updates.

d) Try the full loop (optional)

git checkout -b test-ci
Add-Content README.md "# test"
git commit -am "Test CI"
git push -u origin test-ci
gh pr create --repo $GITHUB_REPO --fill
gh pr checks --repo $GITHUB_REPO --watch

When the checks are green, merge with gh pr merge --squash --repo $GITHUB_REPO and watch CD deploy.

Step 13. Observability with Cloud Logging

A platform you cannot see into is a platform you cannot trust. Every component of ours prints one JSON object per event, and Cloud Logging indexes the field jsonPayload.event. That makes questions like "which step failed last night?" a one-line query. In Logs Explorer (Google Cloud console → Logging) try:

-- failed pipeline steps
jsonPayload.event="pipeline_failure"
-- all errors
severity>=ERROR
-- training runs and their metrics
jsonPayload.event="model_training_completed"
-- predictions served by the API
resource.type="cloud_run_revision" jsonPayload.event="model_prediction"
-- cloud run job logs only
resource.type="cloud_run_job" resource.labels.job_name="mlops-pipeline-job"
-- Airflow task logs
resource.type="cloud_composer_environment"

From PowerShell, use the : operator (meaning "contains"), which avoids inner double quotes that PowerShell mangles when calling gcloud:

gcloud logging read "jsonPayload.event:pipeline_failure" --limit 10 --freshness 1d
gcloud run services logs read mlflow --region $REGION --limit 50
gcloud run jobs executions list --job $JOB --region $REGION

The events emitted are: pipeline_step_started/completed/failure, data_extracted, data_validated, data_preprocessed, data_loaded_bigquery, model_training_started/completed, model_evaluation_passed/failed, model_registered, model_loaded, model_prediction, api_connection_failure, dashboard_error, pipeline_report, api_reloaded, and the admin actions admin_job_triggered, admin_dag_triggered, admin_dag_paused (each carrying the admin's e-mail in user). model_prediction also carries submitted_by.

Optionally, turn failures into an alert: create a log-based metric, then an alert policy in Monitoring → Alerting.

gcloud logging metrics create pipeline_failures --description "Failed pipeline steps" --log-filter "jsonPayload.event:pipeline_failure"

Step 14. The final end-to-end check

Here is the definition of success. If you can tick all nine items, the platform works as designed:

Push a change: CI runs the tests.
Merge: CD deploys.
Trigger the DAG in Airflow.
The job trains 3 XGBoost models and MLflow records parameters, metrics and artifacts.
The best model is registered and promoted to champion.
reload_api makes the API serve it (GET /model-info).
The dashboard shows experiments and answers predictions.
Logs Explorer shows the whole run.
A normal user sees only Predictions; an admin sees all tabs and can start a job step.

Step 15. Cleanup (do not skip it)

Composer and Cloud SQL bill continuously. A forgotten Composer environment costs more than a coffee a day. Clean up as soon as you have finished experimenting.

. .\env.ps1
gcloud composer environments delete $COMPOSER_ENV --location $REGION --quiet
foreach ($s in "dashboard","api","mlflow") { gcloud run services delete $s --region $REGION --quiet }
gcloud run jobs delete $JOB --region $REGION --quiet
gcloud secrets delete dashboard-auth-secrets --quiet
gcloud logging metrics delete pipeline_failures --quiet
cd infrastructure\terraform
terraform destroy
cd ..\..

Composer also creates a bucket named like europe-west1-mlops-demo-*: delete it if it remains (gcloud storage rm -r gs://BUCKET_NAME). Also delete the OAuth client in the console (APIs & Services → Credentials). To delete everything at once: gcloud projects delete $PROJECT_ID.

Just pausing? Delete Composer (recreate it in about 25 minutes) and stop Cloud SQL with gcloud sql instances patch mlflow-pg --activation-policy=NEVER (bring it back with --activation-policy=ALWAYS).

Troubleshooting

Things go wrong. Here are the symptoms you are most likely to meet, with the cause and the fix.

Symptom	Likely cause / fix
`unable to get local issuer certificate` / `CERTIFICATE_VERIFY_FAILED` (pip, gcloud, terraform, git, docker build)	Antivirus HTTPS scanning, proxy or VPN re-signs traffic. Test on a phone hotspot, or turn off HTTPS scanning. For `gcloud`: export the proxy's root certificate to `C:\certs\corp-ca.pem` and run `gcloud config set core/custom_ca_certs_file C:\certs\corp-ca.pem`. For other tools set `$env:SSL_CERT_FILE`, `git config --global http.sslCAInfo C:\certs\corp-ca.pem`, `pip config set global.cert C:\certs\corp-ca.pem`.
`pip` fails building `numpy`	You are not on Python 3.11. Recreate the venv with `py -3.11 -m venv .venv` (Step 1).
`--region: expected one argument`, empty values, `Cannot bind argument`	Variables are not loaded (new window). Run `. .\env.ps1`, then `Check-Env PROJECT_ID,REGION,BUCKET,REGISTRY`.
`MLFLOW_URL` / `API_URL` / `WIF_PROVIDER` empty	The step that sets them did not succeed or was not run. Redo Step 7, 10 or 4 and check `env.ps1` contains the line.
Terraform or `gcloud` says a bucket / repo is not found	Names differ from `env.ps1`. `terraform output -raw bucket` is the source of truth for the bucket.
`gcloud billing projects link` quota error	The billing account has too many linked projects. List them with `gcloud billing projects list --billing-account=$BILLING` and unlink unused ones with `gcloud billing projects unlink PROJECT_ID` (unlinked projects lose paid services).
`docker`: `failed to connect to the docker API`	Docker Desktop is not running. Start it and wait until `docker info` works. If it will not start: `wsl --install`, `wsl --update`, enable virtualization in BIOS.
MLflow revision fails with `exec /entrypoint.sh: no such file or directory`	Windows line endings in `mlflow_server\entrypoint.sh`. Convert with the command in Step 1, rebuild and push the image, redeploy.
`mlflow ui` says `unable to open database file`	Wrong SQLite path. `cd` into the smoke-test folder and use `sqlite:///mlflow.db` (Step 1).
`411 Length Required` on `POST /reload`	A POST without a body has no `Content-Length`. Add `-d "{}"` to the `curl.exe` command.
`403 Forbidden` calling MLflow/API	Missing `roles/run.invoker` for the caller (Steps 7, 8, 10), or an expired ID token (refresh `$TOKEN`).
MLflow revision fails to start	`gcloud run services logs read mlflow --region $REGION`: check the Cloud SQL instance is `RUNNABLE`, the secret exists, and `--add-cloudsql-instances` is set.
`evaluate` fails with `ThresholdNotMet`	Model below `MIN_ROC_AUC`. Lower it: `gcloud run jobs update $JOB --region $REGION --update-env-vars MIN_ROC_AUC=0.7`.
API returns `503 No model loaded`	No champion yet: run the pipeline, then `POST /reload`.
DAG not shown in Airflow	Wait 2-3 min; check DAG import errors in the UI; ensure the Composer env variables exist.
`CloudRunExecuteJobOperator` 403	`sa-airflow` needs `roles/run.developer` and `serviceAccountUser` on `sa-training` (Terraform grants both).
CD cannot authenticate	Check the secrets (`gh secret list`) and that `github_repo` in `terraform.tfvars` equals `$GITHUB_REPO` exactly; if not, fix it and run `terraform apply` again.
Dashboard crashes with `ModuleNotFoundError: No module named 'sklearn'`	`src/features/preprocessing.py` must import scikit-learn only inside `add_split` (the dashboard image does not install it).
`redirect_uri_mismatch` when signing in	The redirect URI in the OAuth client must be exactly `<dashboard URL>/oauth2callback` (Step 10b, part 2) and match `redirect_uri` in the secret.
Google says the app is blocked or only test users can sign in	OAuth consent screen is still in Testing: set it to In production.
Dashboard shows an auth configuration error after deploy	The secret file is not mounted: check `--set-secrets "/app/.streamlit/secrets.toml=dashboard-auth-secrets:latest"` and that `sa-dashboard` has `secretAccessor` on the secret.
Sign-in worked before, broken after a push to `main`	CD replaced the dashboard environment variables. Apply Step 10b part 7 (`--update-env-vars`), then redeploy as in part 6.
Prediction works but `submitted_by` is empty / insert fails	The BigQuery column is missing (Step 10b part 1) or the API still runs the old `main.py`: rebuild and redeploy the API.
Pipeline admin: `PermissionDenied` when starting a job	`sa-dashboard` lacks the job permissions or `serviceAccountUser` on `sa-training` (Step 10b part 5).
Pipeline admin: Airflow call fails (401/403/404)	`AIRFLOW_URL` / `AIRFLOW_API` wrong (Airflow 2 vs 3), `roles/composer.user` missing, or Composer was deleted.
BigQuery `Access Denied`	Wait about 1 minute after `terraform apply`; IAM changes propagate with a delay.

Conclusion: what you really built

Step back and look at what exists now. Data flows from a bucket into BigQuery; three models compete on every run; MLflow remembers everything; a strict rule decides whether a newcomer deserves to replace the champion; an API serves the winner; a dashboard lets colleagues use it safely; GitHub Actions ships code without a single stored key; and Cloud Logging tells the story of every run. The XGBoost model was only the excuse. The habits are what transfer to your real projects: orchestrate instead of compute inside the orchestrator, separate shipping code from promoting models, guard promotion with measurable rules, keep services private, keep secrets out of code, and make everything observable.

And now, the most important command of the whole tutorial: go to Step 15 and switch off what bills you.

Full source code and the repository layout: 8-chems/gcp-mlops-demo on GitHub.