Vertex DevOps

Tip

Add code provided in this page is available in the repo example.

💻 Dev: Compile and run to fasten your dev cycle

When developing a vertex pipeline locally, you may want to iterate quickly. The process is often the following:

1. write new code and test its integration in the pipeline workflow code in a notebook
2. script this new code in vertex/lib/
3. modify associated component(s) in vertex/components and pipelines in vertex/pipelines
4. run the pipeline with dev settings to test the new code on Vertex

The latest may include:

• rebuilding the base image
• compiling the pipeline
• running the pipeline

Build base image

You can use this generic Dockerfile:

FROM python:3.10-slim-buster

ARG PROJECT_ID
ENV PROJECT_ID=${PROJECT_ID}

COPY requirements.txt .
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install -r requirements.txt

COPY vertex .
ENV PYTHONPATH "${PYTHONPATH}:."

Then build it with docker or Cloud Build. For the latest, here is a sample cloudbuild.yaml:

# This config file is meant to be used from a local dev machine to submit a vertex base image build to Cloud Build.
# This generic image will then be used in all the Vertex components of your pipeline.

steps:
  # Build base image
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t', '${_GAR_IMAGE_PATH}',
      '-f', 'vertex/deployment/Dockerfile',
      '--build-arg', 'PROJECT_ID=${PROJECT_ID}',
      '--build-arg', 'GCP_REGION=${_GCP_REGION}',
      '--build-arg', 'GAR_LOCATION=${_GAR_LOCATION}',
      '--build-arg', 'GAR_PIPELINES_REPO_ID=${_GAR_PIPELINES_REPO_ID}',
      '--build-arg', 'VERTEX_STAGING_BUCKET_NAME=${_VERTEX_STAGING_BUCKET_NAME}',
      '--build-arg', 'VERTEX_SERVICE_ACCOUNT=${_VERTEX_SERVICE_ACCOUNT}',
      '.',
    ]
    id: build-base-image

substitutions:
  _GAR_IMAGE_PATH: '${_GAR_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_GAR_DOCKER_REPO_ID}/${_GAR_VERTEX_BASE_IMAGE_NAME}:${_TAG}'
  _GCP_REGION: '${GCP_REGION}'
  _GAR_PIPELINES_REPO_ID: '${GAR_PIPELINES_REPO_ID}'
  _VERTEX_STAGING_BUCKET_NAME: '${VERTEX_STAGING_BUCKET_NAME}'
  _VERTEX_SERVICE_ACCOUNT: '${VERTEX_SERVICE_ACCOUNT}'

options:
  logging: CLOUD_LOGGING_ONLY
  dynamic_substitutions: true

images:
- '${_GAR_IMAGE_PATH}'

tags:
  - vertex-${_GAR_DOCKER_REPO_ID}-base-image-local-${_TAG}

Then you can trigger the build manually using this make command:

export $(cat .env | xargs)
make build-base-image

This command includes the following:

.PHONY: build-base-image
build-base-image:
    @gcloud builds submit --config ./vertex/deployment/cloudbuild_local.yaml \
    --substitutions=_GCP_REGION=${GCP_REGION},_GAR_LOCATION=${GAR_LOCATION},_GAR_DOCKER_REPO_ID=${GAR_DOCKER_REPO_ID},_GAR_VERTEX_BASE_IMAGE_NAME=${GAR_VERTEX_BASE_IMAGE_NAME},_TAG=${TAG},_GAR_PIPELINES_REPO_ID=${GAR_PIPELINES_REPO_ID},_VERTEX_STAGING_BUCKET_NAME=${VERTEX_STAGING_BUCKET_NAME},_VERTEX_SERVICE_ACCOUNT=${VERTEX_SERVICE_ACCOUNT}

Compile and run

Now that you have a base image, you can compile your pipeline and trigger a run that will use the latest version of your docker base image

vertex-deployer deploy --compile --run --env-file .env --config-name my_config.json --skip-validation

✅ CI: Check your pipelines and config integrity

Check your pipelines locally

You can check pipelines integrity and config integrity using the following command:

vertex-deployer check --all

To check a specific pipeline:

vertex-deployer check my_pipeline

Add to CI

You can add a github workflow checking your pipelines integrity using the following file:

name: Check pipelines

on:
  pull_request:
    branches:
      - '*'
  workflow_call:

env:
  PYTHON_VERSION: "3.10"

jobs:
  check-pipelines:
    name: Check Vertex Pipelines
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Set up Python ${{ env.PYTHON_VERSION }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: 'pip'

      - name: Install requirements
        run: |
          python3 -m pip install -r requirements.txt

      - name: Check pipelines
        run: |
          export PYTHONPATH=.
          vertex-deployer check --all

Add to pre-commit hooks

You can add a pre-commit hook checking your pipelines integrity using a local hook:

repos:
  - repo: local
    hooks:
    - id: vertex-deployer-check
      name: check pipelines
      entry: vertex-deployer check -a
      pass_filenames: false
      language: system

🚀 CD: Deploy your pipelines in a standardized manner

Once you have a valid pipeline, you want to deploy it on Vertex. To automate deployment when merging to develop or main, you have multiple options.

• use CloudBuild and CloudBuild triggers
• use Github Action to trigger CloudBuild job
• 🚧 use Github Action only

Note

To use cloudbuild for CD, please update you Dockerfile with all these arguments. This will allow you to use vertex-deployer from your base image in CloudBuild.

FROM python:3.10-slim-buster

ARG PROJECT_ID
ARG GCP_REGION
ARG GAR_LOCATION
ARG GAR_PIPELINES_REPO_ID
ARG VERTEX_STAGING_BUCKET_NAME
ARG VERTEX_SERVICE_ACCOUNT

ENV PROJECT_ID=${PROJECT_ID}
ENV GCP_REGION=${GCP_REGION}
ENV GAR_LOCATION=${GAR_LOCATION}
ENV GAR_PIPELINES_REPO_ID=${GAR_PIPELINES_REPO_ID}
ENV VERTEX_STAGING_BUCKET_NAME=${VERTEX_STAGING_BUCKET_NAME}
ENV VERTEX_SERVICE_ACCOUNT=${VERTEX_SERVICE_ACCOUNT}

COPY requirements.txt .
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install -r requirements.txt

COPY vertex .
ENV PYTHONPATH "${PYTHONPATH}:."

CloudBuild

You can use the following cloudbuild.yaml to trigger a deployment on Vertex when merging to develop or main:

steps:
  # Build base image
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t', '${_GAR_IMAGE_PATH}',
      '-f', 'vertex/deployment/Dockerfile',
      '--build-arg', 'PROJECT_ID=${PROJECT_ID}',
      '--build-arg', 'GCP_REGION=${_GCP_REGION}',
      '--build-arg', 'GAR_LOCATION=${_GAR_LOCATION}',
      '--build-arg', 'GAR_PIPELINES_REPO_ID=${_GAR_PIPELINES_REPO_ID}',
      '--build-arg', 'VERTEX_STAGING_BUCKET_NAME=${_VERTEX_STAGING_BUCKET_NAME}',
      '--build-arg', 'VERTEX_SERVICE_ACCOUNT=${_VERTEX_SERVICE_ACCOUNT}',
      '.',
    ]
    id: build-base-image

  # schedule pipeline: compile, upload, schedule
  - name: '${_GAR_IMAGE_PATH}'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        vertex-deployer -log DEBUG deploy dummy_pipeline \
          --compile \
          --upload \
          --run \
          --enable-caching \
          --config-name config_test.json \
          --tags ${_TAG} \
          --schedule --delete-last-schedule --cron '*-*-19-*-*'

    dir: '.'
    id: schedule-dummy-pipeline
    waitFor: ['build-base-image']

substitutions:
  _GAR_IMAGE_PATH: '${_GAR_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_GAR_DOCKER_REPO_ID}/${_GAR_VERTEX_BASE_IMAGE_NAME}:${_TAG}'

options:
  logging: CLOUD_LOGGING_ONLY
  dynamic_substitutions: true

images:
- '${_GAR_IMAGE_PATH}'

tags:
  - vertex-${_GAR_DOCKER_REPO_ID}-deployment-example-${_TAG}

Use CloudBuild trigger [PREFERRED OPTION]

Then, you'll need to link your repo to CloudBuild and create a trigger for each branch you want to deploy on Vertex. The documentation to link your repo is available here.

Then, you can create create a trigger using this make command:

export $(cat .env | xargs)
make create-trigger-cd

This command includes the following:

.PHONY: create-trigger-cd
create-trigger-cd:
    @gcloud builds triggers create github \
    --repo-owner="artefactory" \
    --repo-name="test-vertex-deployer" \
    --name="test-vertex-deployer-trigger" \
    --branch-pattern="main" \
    --build-config=./vertex/deployment/cloudbuild_cd.yaml \
    --project=${PROJECT_ID} \
    --substitutions=_GCP_REGION=${GCP_REGION},_GAR_LOCATION=${GAR_LOCATION},_GAR_DOCKER_REPO_ID=${GAR_DOCKER_REPO_ID},_GAR_VERTEX_BASE_IMAGE_NAME=${GAR_VERTEX_BASE_IMAGE_NAME},_TAG=${TAG},_GAR_PIPELINES_REPO_ID=${GAR_PIPELINES_REPO_ID},_VERTEX_STAGING_BUCKET_NAME=${VERTEX_STAGING_BUCKET_NAME},_VERTEX_SERVICE_ACCOUNT=${VERTEX_SERVICE_ACCOUNT}

Github Action + CloudBuild

You can also use Github Action to trigger CloudBuild job. You'll need to setup GCp authentication from your repo using Workload Identity Federation.

# This workflow deploys a pipeline to Vertex AI Pipelines.
#
# Workflow Steps:
#
# 1. Check pipelines and config integrity
# 2. Authenticate to Google Cloud using Workload Identity Federation (WIF)
# 3. Submit cloud build job to build docker image and deploy pipeline
#
# For more details on setting up Workload Identity Federation for GitHub, visit https://github.com/google-github-actions/auth#setting-up-workload-identity-federation
#
# Your WIF service account must have the following IAM roles:
# - roles/artifactregistry.writer
# - roles/storage.admin
# - roles/cloudbuild.builds.builder

name: Deploy pipelines

on:
  push:
    branches:
      - main

env:
  PYTHON_VERSION: "3.10"
  PROJECT_ID: "my-project"
  GCP_REGION: "europe-west1"
  TAG: "latest"
  # Google Artifact Registry
  GAR_LOCATION: "europe-west1"
  GAR_DOCKER_REPO_ID: "demo-docker"
  GAR_PIPELINES_REPO_ID: "test-pipelines"
  GAR_VERTEX_BASE_IMAGE_NAME: "base-image"
  # Vertex AI
  VERTEX_STAGING_BUCKET_NAME: "my-project-vertex-staging"
  VERTEX_SERVICE_ACCOUNT: "my-service-account@my-project.iam.gserviceaccount.com"

jobs:
  check-pipelines:
      name: Check Pipelines
      uses: ./.github/workflows/check_pipelines.yaml

  deploy-pipelines:
    name: Deploy pipelines
    needs: check-pipelines
    runs-on: ubuntu-latest
    concurrency: deploy-pipelines
    permissions:
      id-token: write
      contents: write

    steps:
    - uses: actions/checkout@v3
      with:
        fetch-depth: 0

    - name: Set up Python ${{ env.PYTHON_VERSION }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ env.PYTHON_VERSION }}
        cache: 'pip'

    - name: Install requirements
      run: |
        python3 -m pip install --upgrade pip
        python3 -m pip install -r requirements.txt

    - name: 'Authenticate to Google Cloud'
      uses: 'google-github-actions/auth@v1'
      with:
        token_format: 'access_token'
        workload_identity_provider: '${{ secrets.WIF_PROVIDER }}' # e.g. - projects/123456789/locations/global/workloadIdentityPools/my-pool/providers/my-provider
        service_account: '${{ secrets.WIF_SERVICE_ACCOUNT }}' # e.g. - my-service-account@my-project.iam.gserviceaccount.com

    - name: Trigger Cloud Build
      run: |
        export PROJECT_ID=vertex-deployer-sandbox-3a8a
        make deploy-pipeline

🚧 Github Action only

Warning

This is a work in progress. Please use CloudBuild for CD. Docker build and push to GCR example is not yet implemented.