Agentic AI on Google Cloud, Part 5: CI/CD with GitHub Actions and Terraform
Parts 1 to 4 built and hardened the assistant. This post makes shipping it routine: every change, to code or to infrastructure, travels through GitHub Actions and Terraform with tests, reviews and approvals, and nothing is ever deployed by typing commands from your laptop.
The series. Part 1: the GCP map and first agent. Part 2: knowledge preparation and ingestion. Part 3: verification and citations. Part 4: testing, security and monitoring. Part 5 (this post): CI/CD with GitHub Actions and Terraform, plus a downloadable project.
1. Why a pipeline, and what goes wrong without one
Picture the first month in production. Someone fixes a bug on Friday by running gcloud run deploy from their laptop. It works, and nobody writes it down. In week three a colleague applies a Terraform change from a different laptop with older code, and the Friday fix disappears. In week four, production behaves differently from staging and nobody can say why, because there is no record of who changed what, and no test ran before any of it.
A pipeline fixes this with one rule: Git is the source of truth, and only the pipeline is allowed to change the cloud. You change a file, open a pull request, and automatic checks run. A person reviews. After merging, the same pipeline applies the change, first to development, then, after a human approval, to production. Every step is logged, repeatable and reversible.
In Part 4 we sketched this with Cloud Build. Here we build the same ideas on GitHub Actions, which is a good fit when your code already lives on GitHub. Pick one tool per project; the structure below works with either.
Two pipelines, because there are two kinds of change
The split matters. The infrastructure pipeline runs Terraform: it creates and updates service accounts, Cloud Run services, Pub/Sub topics, the BigQuery table and so on. The application pipeline only builds container images and tells Cloud Run to run them. To keep them from fighting, Terraform creates the Cloud Run services once and is then told to ignore the image and the traffic split, which belong to the application pipeline.
The vocabulary
| Term | Meaning |
|---|---|
| CI (continuous integration) | Checks that run automatically on every pull request: tests, linting, a Terraform plan |
| CD (continuous delivery) | The automatic path from a merged change to running software, with approvals where you want them |
| Infrastructure as code | Cloud resources described in files and created by a tool, so they can be reviewed and rebuilt |
| Terraform state | Terraform's record of what it created. It lives in a Cloud Storage bucket so the whole team and the pipeline share one truth |
| Plan and apply | Plan shows what would change without changing anything. Apply makes the change. Reading the plan is the review |
| GitHub environment | A named target (dev, prod) with its own variables, secrets and optional required reviewers |
| Drift | When the real cloud no longer matches the code, because someone changed it by hand |
2. How GitHub is allowed to act on GCP (no keys)
The pipeline needs permission to create and deploy things. The old way was to create a service account, download a JSON key and paste it into GitHub as a secret. That key never expires, can be copied, and if it leaks, whoever has it owns your project.
The modern way is Workload Identity Federation. GitHub gives each job a short-lived signed token saying which repository and branch it comes from. You tell GCP once, "trust tokens from this repository". At run time the job trades its token for short-lived access as a service account. There is nothing to leak and nothing to rotate.
We create two such service accounts per environment, on purpose. tf-sa is powerful, because Terraform must create service accounts, topics and databases. deploy-sa is narrow: it can push images and update Cloud Run revisions, nothing more. Application deploys, which happen many times a day, therefore never hold the power to rewrite your infrastructure.
3. Hands-on: build the pipeline
Eight steps. First the one-time foundation (Steps 1 to 2), then the checks that run on every pull request (Steps 3 to 4), then the two delivery pipelines (Steps 5 to 6), and finally keeping them honest over time (Steps 7 to 8). Every file shown is in the downloadable project at the end of the post.
Step 1. Bootstrap each project once
The next job: prepare the ground that the pipeline itself needs. A pipeline cannot create the bucket that stores its own state, nor the trust relationship that lets it log in, so these few things are done once, by a person, with a script. Run it once for the development project and once for production. It creates the state bucket (with versioning, so a bad state file can be recovered), the Workload Identity pool and provider limited to your repository, the two service accounts, and their roles.
# 2. Workload Identity Federation: GitHub Actions proves who it is with a short-lived OIDC token
gcloud iam workload-identity-pools create github --project "$PROJECT" --location global \
--display-name "GitHub Actions" || true
gcloud iam workload-identity-pools providers create-oidc github-provider --project "$PROJECT" \
--location global --workload-identity-pool github \
--issuer-uri "https://token.actions.githubusercontent.com" \
--attribute-mapping "google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.ref=assertion.ref" \
--attribute-condition "assertion.repository=='$GITHUB_REPO'" || trueThe --attribute-condition line is the most important one: it makes GCP refuse tokens from any repository other than yours. The script ends by printing the values you need as GitHub variables.
infra/ folder, and consider tightening it to the main branch (the token carries the branch in attribute.ref) once the pipeline works.Step 2. Describe the platform in Terraform
The next job: turn every manual command from Parts 1 and 2 into a resource. The project keeps the resources in one reusable module (modules/platform) and gives each environment a thin folder (envs/dev, envs/prod) that calls it with its own values. Dev and prod therefore use identical code and differ only in settings such as deletion protection and whether the API is public.
infra/terraform/
├── bootstrap (script, not Terraform) -> ../bootstrap/bootstrap.sh
├── modules/platform/ service accounts, Cloud Run services and job, Pub/Sub, BigQuery,
│ Firestore, Artifact Registry, secrets, log metric
└── envs/{dev,prod}/ main.tf (calls the module) + terraform.tfvars (project id, model names)Two details deserve attention. The environment leaves the state location empty (backend "gcs" {}), and the pipeline supplies the bucket and prefix at run time, which keeps the code identical across projects. And the Cloud Run service tells Terraform to leave the image alone after the first creation:
resource "google_cloud_run_v2_service" "api" {
project = var.project_id
name = "agent-api"
location = var.region
template {
service_account = google_service_account.api.email
scaling {
min_instance_count = 0
max_instance_count = var.max_instances
}
containers {
image = var.api_image
dynamic "env" {
for_each = local.common_env
content {
name = env.key
value = env.value
}
}
env {
name = "AGENT_MODEL_NAME"
value = var.model_name
}
env {
name = "AGENT_VERIFIER_MODEL"
value = var.verifier_model
}
env {
name = "AGENT_CLAIMS_API_TOKEN"
value_source {
secret_key_ref {
secret = google_secret_manager_secret.claims_token.secret_id
version = "latest"
}
}
}
}
}
lifecycle {
ignore_changes = [template[0].containers[0].image, template[0].revision, traffic, client, client_version]
}
depends_on = [google_secret_manager_secret_version.placeholder, google_project_iam_member.api]
}
...
lifecycle {
ignore_changes = [template[0].containers[0].image, template[0].revision, traffic, client, client_version]
}
}Without that lifecycle block, the next Terraform apply would roll production back to whatever image Terraform last knew, undoing the application pipeline. Try it locally first:
cd infra/terraform/envs/dev
terraform init -backend-config="bucket=YOUR-PROJECT-tfstate" -backend-config="prefix=agent/dev"
terraform fmt -recursive ../..
terraform validate
terraform planResource arguments change between provider versions, so treat terraform validate and the provider documentation as the final word. Note also that the claims token is a placeholder secret version; add the real value as a new version outside Terraform, so the token never enters the state file.
Step 3. Check every pull request
The next job: make mistakes cheap. This workflow runs on every pull request, in three parts: Python tests and linting (seconds, no cloud access), static Terraform checks for both environments (formatting and validation, no cloud access), and finally a plan against dev, posted back to the pull request as a comment. The reviewer reads the plan the way they would read a code diff: "this change will create one topic and modify one service account".
name: ci
on:
pull_request:
permissions:
contents: read
id-token: write
pull-requests: write
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- run: pip install -r requirements.txt -r requirements-dev.txt
- run: ruff check src tests
- run: pytest tests/unit -q
terraform-static:
runs-on: ubuntu-latest
strategy:
matrix:
env: [dev, prod]
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- run: terraform fmt -check -recursive infra/terraform
- run: terraform -chdir=infra/terraform/envs/${{ matrix.env }} init -backend=false
- run: terraform -chdir=infra/terraform/envs/${{ matrix.env }} validate
plan-dev:
needs: [test, terraform-static]
runs-on: ubuntu-latest
environment: dev
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.WIF_PROVIDER }}
service_account: ${{ vars.TF_SA }}
- uses: hashicorp/setup-terraform@v3
- name: Plan
run: |
cd infra/terraform/envs/dev
terraform init -input=false -backend-config="bucket=${{ vars.TFSTATE_BUCKET }}" -backend-config="prefix=agent/dev"
terraform plan -input=false -no-color > "$GITHUB_WORKSPACE/plan.txt"
- name: Comment the plan on the pull request
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const plan = fs.readFileSync('plan.txt', 'utf8').slice(0, 60000);
await github.rest.issues.createComment({
issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo,
body: "Terraform plan for dev:\n```\n" + plan + "\n```"
});The permissions block lists id-token: write; without it the job cannot request the token that Workload Identity Federation needs. The plan job runs in the dev environment, which is how it receives that environment's variables.
Step 4. Set up the GitHub environments and protections
The next job: decide who may approve what. Environments are how GitHub turns "this needs a human" into a rule that cannot be skipped.
| Environment | Reviewers | Used by |
|---|---|---|
dev | none: automatic | PR plan, infra apply to dev, app deploy to dev |
prod | required | infra apply to prod, canary deploy to prod |
prod-promote | required | moving production from 10% to 100% |
For dev and prod set the variables printed by the bootstrap script (PROJECT_ID, REGION, TFSTATE_BUCKET, WIF_PROVIDER, TF_SA, DEPLOY_SA) and a secret EVAL_TOKENS holding test-customer tokens for the eval gate. Then protect the main branch: require the ci checks to pass, require a review, and add a CODEOWNERS file so changes under infra/ need the platform team.
Step 5. Deliver infrastructure with Terraform
The next job: apply merged infrastructure changes safely. When a change under infra/ is merged, the pipeline applies it to dev automatically and then pauses at the prod environment until a reviewer approves. The concurrency setting stops two applies from running at once, which would fight over the state lock.
jobs:
dev:
runs-on: ubuntu-latest
environment: dev
concurrency: terraform-dev
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.WIF_PROVIDER }}
service_account: ${{ vars.TF_SA }}
- uses: hashicorp/setup-terraform@v3
- run: |
cd infra/terraform/envs/dev
terraform init -input=false -backend-config="bucket=${{ vars.TFSTATE_BUCKET }}" -backend-config="prefix=agent/dev"
terraform plan -input=false -out=tfplan
terraform apply -input=false tfplanThe same job is repeated for prod with prefix=agent/prod, after the approval. For stricter reviews, save the plan as an artifact in one job and apply exactly that file in the next, so what was approved is what runs.
Step 6. Deliver the application: canary, evaluate, promote
The next job: ship code the way Part 4 described, but automatically. One reusable workflow does the work for any environment: it builds both images tagged with the commit, deploys the API as a new Cloud Run revision with no traffic and the tag canary, updates the ingestion service and batch job, runs the golden-set eval against the canary's own URL, and only then shifts traffic.
# Reusable workflow: build both images, deploy a canary revision, run the eval gate, shift traffic.
name: _deploy
on:
workflow_call:
inputs:
environment:
type: string
required: true
canary_percent:
type: number
default: 10
permissions:
contents: read
id-token: write
jobs:
deploy:
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
env:
REG: ${{ vars.REGION }}-docker.pkg.dev/${{ vars.PROJECT_ID }}/agent
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.WIF_PROVIDER }}
service_account: ${{ vars.DEPLOY_SA }}
- uses: google-github-actions/setup-gcloud@v2
- run: gcloud auth configure-docker ${{ vars.REGION }}-docker.pkg.dev --quiet
- name: Build and push images (tagged with the commit)
run: |
docker build -f Dockerfile -t $REG/api:${{ github.sha }} .
docker build -f Dockerfile.ingest -t $REG/ingest:${{ github.sha }} .
docker push $REG/api:${{ github.sha }}
docker push $REG/ingest:${{ github.sha }}
- name: Deploy API as a canary revision (no traffic)
run: |
gcloud run deploy agent-api --image $REG/api:${{ github.sha }} --region ${{ vars.REGION }} \
--no-traffic --tag canary --quiet
- name: Update ingestion service and batch job
run: |
gcloud run deploy rag-ingestion --image $REG/ingest:${{ github.sha }} --region ${{ vars.REGION }} --quiet
gcloud run jobs update rag-batch-ingestion --image $REG/ingest:${{ github.sha }} --region ${{ vars.REGION }} --quiet
- name: Eval gate against the canary URL
env:
EVAL_TOKENS: ${{ secrets.EVAL_TOKENS }}
run: |
URL=$(gcloud run services describe agent-api --region ${{ vars.REGION }} --format=json \
| jq -r '.status.traffic[] | select(.tag=="canary") | .url')
export GCP_ID_TOKEN=$(gcloud auth print-identity-token --audiences="$URL" 2>/dev/null || true)
echo "$EVAL_TOKENS" > evals/test_tokens.json
pip install -q httpx pyyaml
python evals/run_eval.py "$URL"
- name: Shift traffic to the canary
run: |
gcloud run services update-traffic agent-api --region ${{ vars.REGION }} \
--to-tags canary=${{ inputs.canary_percent }}The caller decides how far to go. Dev takes 100% straight away; production takes 10% and then waits for a second approval before the full switch:
dev:
needs: test
uses: ./.github/workflows/_deploy.yml
with:
environment: dev
canary_percent: 100
secrets: inherit
prod:
needs: dev
uses: ./.github/workflows/_deploy.yml # the "prod" environment requires a reviewer to approve
with:
environment: prod
canary_percent: 10
secrets: inherit
promote-prod:
needs: prod
runs-on: ubuntu-latest
environment: prod-promote # second approval, after watching the 10% canary
steps:
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.WIF_PROVIDER }}
service_account: ${{ vars.DEPLOY_SA }}
- uses: google-github-actions/setup-gcloud@v2
- run: gcloud run services update-traffic agent-api --region ${{ vars.REGION }} --to-latestRead the approvals as two different questions. The prod approval asks "should this version meet real customers at all?" and the prod-promote approval asks "after watching 10% of traffic (Part 4's dashboards and alerts), should everyone get it?". If the answer to the second is no, roll back, which is one command and does not rebuild anything:
gcloud run services update-traffic agent-api --region $REGION --to-revisions PREVIOUS_REVISION=100X-Serverless-Authorization and the customer token in Authorization. A private Cloud Run service reads the first for IAM and your app reads the second for the customer, so one header cannot carry both. The images are rebuilt in each environment's own registry from the same commit; if you need byte-identical artifacts across environments, use a shared registry and grant both runtime identities read access.Step 7. Watch for drift
The next job: catch the hand edit. Someone will eventually change a setting in the console during an incident. A weekly job runs terraform plan against each environment and fails if the plan is not empty, which turns silent drift into a visible failed run.
- name: Detect drift
run: |
cd infra/terraform/envs/${{ matrix.env }}
terraform init -input=false -backend-config="bucket=${{ vars.TFSTATE_BUCKET }}" -backend-config="prefix=agent/${{ matrix.env }}"
code=0
terraform plan -input=false -detailed-exitcode -no-color || code=$?
if [ "$code" -eq 2 ]; then echo "::error::Drift detected in ${{ matrix.env }}"; exit 1; fi
exit "$code"The key is -detailed-exitcode: exit code 0 means no changes, 2 means differences exist, 1 means an error. When drift is found, decide whether the code or the console is right, then update one to match the other.
Step 8. Everyday operations
The next job: know the routine moves, so none of them require improvising.
| Situation | What to do |
|---|---|
| Ship a code change | Pull request, review, merge. The application pipeline does the rest |
| Change infrastructure | Edit Terraform, read the plan comment, merge, approve the prod apply |
| Bad release in production | Move traffic back to the previous revision; fix forward through a normal pull request |
| Urgent fix | Same pipeline, same gates; use workflow_dispatch to start it manually. Do not bypass the checks |
| Rotate the claims token | Add a new Secret Manager version; new instances pick up latest |
| Change model name or limits | Edit terraform.tfvars in a pull request; it follows the infrastructure pipeline |
| Add an environment | New project, run the bootstrap script, copy an envs/ folder, add a GitHub environment |
4. Pitfalls we hit on paper
| Symptom | Likely cause |
|---|---|
| Authentication step fails with a permission error | Missing id-token: write, a wrong provider name, or the attribute condition does not match your repository name (case matters) |
| Deploy fails naming a service account | deploy-sa lacks roles/iam.serviceAccountUser on api-sa or ingest-sa. Grant it after the first Terraform apply |
| Terraform apply rolls production back | The ignore_changes block for image and traffic is missing |
| "Error acquiring the state lock" | Two applies overlapped, or a run was cancelled. Use concurrency; unlock only when sure nothing is running |
| First deploy fails reading the secret | The secret has no version. The placeholder version in the module avoids this |
| Eval gate returns 401 or 403 | The identity token or test tokens are missing, or the audience does not match the canary URL |
| Plan comment is cut off | Comments have a size limit; the workflow truncates it. Open the run log for the full plan |
5. Download the project
Everything in the series is assembled into one repository you can unzip, read and adapt: insurance-agent.zip. It contains the application code (API, agent, tools, verification, retrieval, ingestion), tests and evaluation files, Dockerfiles, the Terraform platform module with dev and prod environments, the bootstrap script and the four GitHub workflows from this post. Its README lists the layout and a local quick start.
insurance-agent/
├── src/{api,agent,verify,rag,ingestion,common}
├── tests/{unit,security} + fake_claims.py
├── evals/ golden set, runner, thresholds
├── infra/bootstrap/ one-time project setup
├── infra/terraform/ modules/platform + envs/{dev,prod}
├── .github/workflows/ ci, _deploy, deploy-app, deploy-infra, drift
└── Dockerfile, Dockerfile.ingest, requirements*.txt, Makefile, README.mdtests/unit pass in a plain Python environment. The workflows parse as valid YAML. Nothing has been run against real GCP or GitHub, and the Terraform has not been through terraform validate. Treat the project as a carefully assembled scaffold: validate it in a sandbox, and confirm ADK, Gen AI SDK, provider and workflow-action details against current documentation.6. Checklist and where to go from here
| Area | Ready when |
|---|---|
| Authentication | Workload Identity Federation with a repository condition; no JSON keys anywhere |
| Separation | Terraform identity and deploy identity are different; image and traffic are owned by the app pipeline |
| Reviews | Plan posted on every pull request; protected main branch; CODEOWNERS on infra |
| Approvals | Production apply, canary and promotion each need a named reviewer |
| Safety nets | Eval gate blocks bad canaries; rollback rehearsed; state bucket versioned |
| Hygiene | Weekly drift check; provider and action versions pinned and reviewed |
This closes the series: an architecture (Part 1), knowledge and ingestion (Part 2), verified answers (Part 3), tested and monitored operation (Part 4) and an automated, reviewable delivery process (Part 5). Natural next steps are write operations with approval workflows on Cloud Workflows, the knowledge graph, a React interface with clickable citations, and a nightly reindex scheduled with Cloud Scheduler.
References: google-github-actions/auth, Workload Identity Federation, Terraform GCS backend, GitHub environments, Cloud Run traffic management.
Comments
Post a Comment