Basic Usage

TL;DR

Deploy pipeline:

vertex-deployer deploy dummy_pipeline \
    --compile \
    --upload \
    --run \
    --env-file example.env \
    --local-package-path . \
    --tags my-tag \
    --config-filepath vertex/configs/dummy_pipeline/config_test.json \
    --experiment-name my-experiment \
    --enable-caching \
    --skip-validation

Check pipelines:

vertex-deployer check --all

🚀 CLI: Deploying a Pipeline with deploy

Let's say you defined a pipeline in dummy_pipeline.py and a config file named config_test.json. You can deploy your pipeline using the following command:

vertex-deployer deploy dummy_pipeline \
    --compile \
    --upload \
    --run \
    --env-file example.env \
    --tags my-tag \
    --config-filepath vertex/configs/dummy_pipeline/config_test.json \
    --experiment-name my-experiment \
    --enable-caching \
    --skip-validation

✅ CLI: Checking Pipelines are valid with check

To check that your pipelines are valid, you can use the check command. It uses a pydantic model to: - check that your pipeline imports and definition are valid - check that your pipeline can be compiled - check that all configs related to the pipeline are respecting the pipeline definition (using a Pydantic model based on pipeline signature)

To validate one or multiple pipeline(s):

vertex-deployer check dummy_pipeline <other pipeline name>

To validate all pipelines in the vertex/pipelines folder:

vertex-deployer check --all

🛠️ CLI: Other commands

config

You can check your vertex-deployer configuration options using the config command. Fields set in pyproject.toml will overwrite default values and will be displayed differently:

vertex-deployer config --all

create

You can create all files needed for a pipeline using the create command:

vertex-deployer create my_new_pipeline --config-type py

This will create a my_new_pipeline.py file in the vertex/pipelines folder and a vertex/config/my_new_pipeline/ folder with multiple config files in it.

init

To initialize the deployer with default settings and folder structure, use the init command:

vertex-deployer init

$ vertex-deployer init
Welcome to Vertex Deployer!
This command will help you getting fired up.
Do you want to configure the deployer? [y/n]: n
Do you want to build default folder structure [y/n]: n
Do you want to create a pipeline? [y/n]: n
All done ✨

list

You can list all pipelines in the vertex/pipelines folder using the list command:

vertex-deployer list --with-configs

🍭 CLI: Options

vertex-deployer --help

To see package version:

vertex-deployer --version

To adapt log level, use the --log-level option. Default is INFO.

vertex-deployer --log-level DEBUG deploy ...