Prerequisites & Setup

TL;DR

You need a GCP project ready to use Vertex Pipelines. You'll need to activate the following services:

• CloudBuild
• Artifact Registry
• Storage
• Vertex Pipelines

And create some stuff in it.

📋 Prerequisites

• Unix-like environment (Linux, macOS, WSL, etc.)
• Python 3.8 to 3.10
• Google Cloud SDK
• A GCP project with Vertex Pipelines enabled

🛠️ Setup

1. Setup your GCP environment:

   export PROJECT_ID=<gcp_project_id>
   gcloud config set project $PROJECT_ID
   gcloud auth login
   gcloud auth application-default login
   

2. You need the following APIs to be enabled:

3. Cloud Build API
4. Artifact Registry API
5. Cloud Storage API

6. Vertex AI API

   gcloud services enable \
       cloudbuild.googleapis.com \
       artifactregistry.googleapis.com \
       storage.googleapis.com \
       aiplatform.googleapis.com
   

7. Create an artifact registry repository for your base images (Docker format):

   export GAR_DOCKER_REPO_ID=<your_gar_repo_id_for_images>
   export GAR_LOCATION=<your_gar_location>
   gcloud artifacts repositories create ${GAR_DOCKER_REPO_ID} \
       --location=${GAR_LOCATION} \
       --repository-format=docker
   

8. Build and upload your base images to the repository. To do so, please follow Google Cloud Build documentation.

9. Create an artifact registry repository for your pipelines (KFP format):

   export GAR_PIPELINES_REPO_ID=<your_gar_repo_id_for_pipelines>
   gcloud artifacts repositories create ${GAR_PIPELINES_REPO_ID} \
       --location=${GAR_LOCATION} \
       --repository-format=kfp
   

10. Create a GCS bucket for Vertex Pipelines staging:

    export GCP_REGION=<your_gcp_region>
    export VERTEX_STAGING_BUCKET_NAME=<your_bucket_name>
    gcloud storage buckets create gs://${VERTEX_STAGING_BUCKET_NAME} --location=${GCP_REGION}
    

11. Create a service account for Vertex Pipelines:

    export VERTEX_SERVICE_ACCOUNT_NAME=foobar
    export VERTEX_SERVICE_ACCOUNT="${VERTEX_SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
    
    gcloud iam service-accounts create ${VERTEX_SERVICE_ACCOUNT_NAME}
    
    gcloud projects add-iam-policy-binding ${PROJECT_ID} \
        --member="serviceAccount:${VERTEX_SERVICE_ACCOUNT}" \
        --role="roles/aiplatform.user"
    
    gcloud storage buckets add-iam-policy-binding gs://${VERTEX_STAGING_BUCKET_NAME} \
        --member="serviceAccount:${VERTEX_SERVICE_ACCOUNT}" \
        --role="roles/storage.objectUser"
    
    gcloud artifacts repositories add-iam-policy-binding ${GAR_PIPELINES_REPO_ID} \
       --location=${GAR_LOCATION} \
       --member="serviceAccount:${VERTEX_SERVICE_ACCOUNT}" \
       --role="roles/artifactregistry.admin"