Vertex DevOps

Tip

Add code provided in this page is available in the repo example.

💻 Dev: Compile and run to fasten your dev cycle

When developing a vertex pipeline locally, you may want to iterate quickly. The process is often the following:

1. write new code and test its integration in the pipeline workflow code in a notebook
2. script this new code in vertex/lib/
3. modify associated component(s) in vertex/components and pipelines in vertex/pipelines
4. run the pipeline with dev settings to test the new code on Vertex

The latest may include:

• rebuilding the base image
• compiling the pipeline
• running the pipeline

Build base image

You can use this generic Dockerfile:

FROM python:3.10-slim-buster

ARG PROJECT_ID
ENV PROJECT_ID=${PROJECT_ID}

COPY requirements.txt .
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install -r requirements.txt

COPY vertex .
ENV PYTHONPATH "${PYTHONPATH}:."

Then build it with docker or Cloud Build. For the latest, here is a sample cloudbuild.yaml:

# This config file is meant to be used from a local dev machine to submit a vertex base image build to Cloud Build.
# This generic image will then be used in all the Vertex components of your pipeline.

steps:
  # Build base image
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t', '${_GAR_IMAGE_PATH}',
      '-f', 'vertex/deployment/Dockerfile',
      '--build-arg', 'PROJECT_ID=${PROJECT_ID}',
      '--build-arg', 'GCP_REGION=${_GCP_REGION}',
      '--build-arg', 'GAR_LOCATION=${_GAR_LOCATION}',
      '--build-arg', 'GAR_PIPELINES_REPO_ID=${_GAR_PIPELINES_REPO_ID}',
      '--build-arg', 'VERTEX_STAGING_BUCKET_NAME=${_VERTEX_STAGING_BUCKET_NAME}',
      '--build-arg', 'VERTEX_SERVICE_ACCOUNT=${_VERTEX_SERVICE_ACCOUNT}',
      '.',
    ]
    id: build-base-image

substitutions:
  _GAR_IMAGE_PATH: '${_GAR_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_GAR_DOCKER_REPO_ID}/${_GAR_VERTEX_BASE_IMAGE_NAME}:${_TAG}'
  _GCP_REGION: '${GCP_REGION}'
  _GAR_PIPELINES_REPO_ID: '${GAR_PIPELINES_REPO_ID}'
  _VERTEX_STAGING_BUCKET_NAME: '${VERTEX_STAGING_BUCKET_NAME}'
  _VERTEX_SERVICE_ACCOUNT: '${VERTEX_SERVICE_ACCOUNT}'

options:
  logging: CLOUD_LOGGING_ONLY
  dynamic_substitutions: true

images:
- '${_GAR_IMAGE_PATH}'

tags:
  - vertex-${_GAR_DOCKER_REPO_ID}-base-image-local-${_TAG}

Then you can trigger the build manually using this make command:

export $(cat .env | xargs)
make build-base-image

This command includes the following:

.PHONY: build-base-image
build-base-image:
    @gcloud builds submit --config ./vertex/deployment/cloudbuild_local.yaml \
    --substitutions=_GCP_REGION=${GCP_REGION},_GAR_LOCATION=${GAR_LOCATION},_GAR_DOCKER_REPO_ID=${GAR_DOCKER_REPO_ID},_GAR_VERTEX_BASE_IMAGE_NAME=${GAR_VERTEX_BASE_IMAGE_NAME},_TAG=${TAG},_GAR_PIPELINES_REPO_ID=${GAR_PIPELINES_REPO_ID},_VERTEX_STAGING_BUCKET_NAME=${VERTEX_STAGING_BUCKET_NAME},_VERTEX_SERVICE_ACCOUNT=${VERTEX_SERVICE_ACCOUNT}

Compile and run

Now that you have a base image, you can compile your pipeline and trigger a run that will use the latest version of your docker base image

vertex-deployer deploy --compile --run --env-file .env --config-name my_config.json --skip-validation

✅ CI: Check your pipelines and config integrity

Check your pipelines locally

You can check pipelines integrity and config integrity using the following command:

vertex-deployer check --all

To check a specific pipeline:

vertex-deployer check my_pipeline

Add to CI

You can add a github workflow checking your pipelines integrity using the following file:

name: Check pipelines

on:
  pull_request:
    branches:
      - '*'
  workflow_call:

env:
  PYTHON_VERSION: "3.10"

jobs:
  check-pipelines:
    name: Check Vertex Pipelines
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Set up Python ${{ env.PYTHON_VERSION }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: 'pip'

      - name: Install requirements
        run: |
          python3 -m pip install -r requirements.txt

      - name: Check pipelines
        run: |
          export PYTHONPATH=.
          vertex-deployer check --all

Add to pre-commit hooks

You can add a pre-commit hook checking your pipelines integrity using a local hook:

repos:
  - repo: local
    hooks:
    - id: vertex-deployer-check
      name: check pipelines
      entry: vertex-deployer check -a
      pass_filenames: false
      language: system

🚀 CD: Deploy your pipelines in a standardized manner

Once you have a valid pipeline, you want to deploy it on Vertex. To automate deployment when merging to develop or main, you have multiple options.

Quick start with init

The fastest way to set up CI/CD is with the init command:

vertex-deployer init

When prompted, select github or gitlab as your CI/CD platform. This generates a ready-to-use workflow with multi-environment deployment stages (dev → stg → prd).

• GitHub Actions: creates .github/workflows/cd.yml with Workload Identity Federation authentication
• GitLab CI: creates .gitlab-ci.yml with Docker-in-Docker image builds and manual production deployment gate

The generated templates include: base image build, pipeline validation, and deployment to three environments. You'll need to configure the required CI/CD variables in your platform (see the environment variables section in the generated file).

For GitHub Actions, you'll need to set up Workload Identity Federation and configure the following GitHub repository variables: GCP_PROJECT_ID, GCP_REGION, GAR_LOCATION, GAR_DOCKER_REPO_ID, GAR_PIPELINES_REPO_ID, GAR_VERTEX_BASE_IMAGE_NAME, VERTEX_STAGING_BUCKET_NAME, VERTEX_SERVICE_ACCOUNT, WIF_PROVIDER, WIF_SERVICE_ACCOUNT.

Manual setup options

If you prefer to set up CI/CD manually, you have multiple options:

• use CloudBuild and CloudBuild triggers
• use Github Action to trigger CloudBuild job
• 🚧 use Github Action only

Note

To use cloudbuild for CD, please update you Dockerfile with all these arguments. This will allow you to use vertex-deployer from your base image in CloudBuild.

FROM python:3.10-slim-buster

ARG PROJECT_ID
ARG GCP_REGION
ARG GAR_LOCATION
ARG GAR_PIPELINES_REPO_ID
ARG VERTEX_STAGING_BUCKET_NAME
ARG VERTEX_SERVICE_ACCOUNT

ENV PROJECT_ID=${PROJECT_ID}
ENV GCP_REGION=${GCP_REGION}
ENV GAR_LOCATION=${GAR_LOCATION}
ENV GAR_PIPELINES_REPO_ID=${GAR_PIPELINES_REPO_ID}
ENV VERTEX_STAGING_BUCKET_NAME=${VERTEX_STAGING_BUCKET_NAME}
ENV VERTEX_SERVICE_ACCOUNT=${VERTEX_SERVICE_ACCOUNT}

COPY requirements.txt .
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install -r requirements.txt

COPY vertex .
ENV PYTHONPATH "${PYTHONPATH}:."

CloudBuild

You can use the following cloudbuild.yaml to trigger a deployment on Vertex when merging to develop or main:

steps:
  # Build base image
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t', '${_GAR_IMAGE_PATH}',
      '-f', 'vertex/deployment/Dockerfile',
      '--build-arg', 'PROJECT_ID=${PROJECT_ID}',
      '--build-arg', 'GCP_REGION=${_GCP_REGION}',
      '--build-arg', 'GAR_LOCATION=${_GAR_LOCATION}',
      '--build-arg', 'GAR_PIPELINES_REPO_ID=${_GAR_PIPELINES_REPO_ID}',
      '--build-arg', 'VERTEX_STAGING_BUCKET_NAME=${_VERTEX_STAGING_BUCKET_NAME}',
      '--build-arg', 'VERTEX_SERVICE_ACCOUNT=${_VERTEX_SERVICE_ACCOUNT}',
      '.',
    ]
    id: build-base-image

  # schedule pipeline: compile, upload, schedule
  - name: '${_GAR_IMAGE_PATH}'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        vertex-deployer -log DEBUG deploy dummy_pipeline \
          --compile \
          --upload \
          --run \
          --enable-caching \
          --config-name config_test.json \
          --tags ${_TAG} \
          --schedule --delete-last-schedule --cron '*-*-19-*-*'

    dir: '.'
    id: schedule-dummy-pipeline
    waitFor: ['build-base-image']

substitutions:
  _GAR_IMAGE_PATH: '${_GAR_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_GAR_DOCKER_REPO_ID}/${_GAR_VERTEX_BASE_IMAGE_NAME}:${_TAG}'

options:
  logging: CLOUD_LOGGING_ONLY
  dynamic_substitutions: true

images:
- '${_GAR_IMAGE_PATH}'

tags:
  - vertex-${_GAR_DOCKER_REPO_ID}-deployment-example-${_TAG}

Use CloudBuild trigger [PREFERRED OPTION]

Then, you'll need to link your repo to CloudBuild and create a trigger for each branch you want to deploy on Vertex. The documentation to link your repo is available here.

Then, you can create create a trigger using this make command:

export $(cat .env | xargs)
make create-trigger-cd

This command includes the following:

.PHONY: create-trigger-cd
create-trigger-cd:
    @gcloud builds triggers create github \
    --repo-owner="artefactory" \
    --repo-name="test-vertex-deployer" \
    --name="test-vertex-deployer-trigger" \
    --branch-pattern="main" \
    --build-config=./vertex/deployment/cloudbuild_cd.yaml \
    --project=${PROJECT_ID} \
    --substitutions=_GCP_REGION=${GCP_REGION},_GAR_LOCATION=${GAR_LOCATION},_GAR_DOCKER_REPO_ID=${GAR_DOCKER_REPO_ID},_GAR_VERTEX_BASE_IMAGE_NAME=${GAR_VERTEX_BASE_IMAGE_NAME},_TAG=${TAG},_GAR_PIPELINES_REPO_ID=${GAR_PIPELINES_REPO_ID},_VERTEX_STAGING_BUCKET_NAME=${VERTEX_STAGING_BUCKET_NAME},_VERTEX_SERVICE_ACCOUNT=${VERTEX_SERVICE_ACCOUNT}

Github Action + CloudBuild

You can also use Github Action to trigger CloudBuild job. You'll need to setup GCp authentication from your repo using Workload Identity Federation.

# This workflow deploys a pipeline to Vertex AI Pipelines.
#
# Workflow Steps:
#
# 1. Check pipelines and config integrity
# 2. Authenticate to Google Cloud using Workload Identity Federation (WIF)
# 3. Submit cloud build job to build docker image and deploy pipeline
#
# For more details on setting up Workload Identity Federation for GitHub, visit https://github.com/google-github-actions/auth#setting-up-workload-identity-federation
#
# Your WIF service account must have the following IAM roles:
# - roles/artifactregistry.writer
# - roles/storage.admin
# - roles/cloudbuild.builds.builder

name: Deploy pipelines

on:
  push:
    branches:
      - main

env:
  PYTHON_VERSION: "3.10"
  PROJECT_ID: "my-project"
  GCP_REGION: "europe-west1"
  TAG: "latest"
  # Google Artifact Registry
  GAR_LOCATION: "europe-west1"
  GAR_DOCKER_REPO_ID: "demo-docker"
  GAR_PIPELINES_REPO_ID: "test-pipelines"
  GAR_VERTEX_BASE_IMAGE_NAME: "base-image"
  # Vertex AI
  VERTEX_STAGING_BUCKET_NAME: "my-project-vertex-staging"
  VERTEX_SERVICE_ACCOUNT: "my-service-account@my-project.iam.gserviceaccount.com"

jobs:
  check-pipelines:
      name: Check Pipelines
      uses: ./.github/workflows/check_pipelines.yaml

  deploy-pipelines:
    name: Deploy pipelines
    needs: check-pipelines
    runs-on: ubuntu-latest
    concurrency: deploy-pipelines
    permissions:
      id-token: write
      contents: write

    steps:
    - uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Set up Python ${{ env.PYTHON_VERSION }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
        cache: 'pip'

    - name: Install requirements
      run: |
        python3 -m pip install --upgrade pip
        python3 -m pip install -r requirements.txt

    - name: 'Authenticate to Google Cloud'
      uses: 'google-github-actions/auth@v1'
      with:
        token_format: 'access_token'
        workload_identity_provider: '${{ secrets.WIF_PROVIDER }}' # e.g. - projects/123456789/locations/global/workloadIdentityPools/my-pool/providers/my-provider
        service_account: '${{ secrets.WIF_SERVICE_ACCOUNT }}' # e.g. - my-service-account@my-project.iam.gserviceaccount.com

    - name: Trigger Cloud Build
      run: |
        export PROJECT_ID=vertex-deployer-sandbox-3a8a
        make deploy-pipeline

🚧 Github Action only

Warning

This is a work in progress. Please use CloudBuild for CD. Docker build and push to GCR example is not yet implemented.