From Manual to Automated

Manually running your ML pipeline is a good start, but it's not a scalable or reproducible solution. To achieve true MLOps, you need to integrate your pipeline with a CI/CD tool like Cloud Build.

1. Cloud Build

Cloud Build is a fully managed CI/CD platform that lets you build, test, and deploy your code. You can use Cloud Build to automate your ML workflows by creating a cloudbuild.yaml file that defines a series of steps to be executed.

Example: cloudbuild.yaml

steps:
  # Install dependencies
  - name: 'python:3.9'
    entrypoint: 'pip'
    args: ['install', '-r', 'requirements.txt']

  # Run unit tests
  - name: 'python:3.9'
    entrypoint: 'pytest'
    args: ['tests/']

  # Compile the pipeline
  - name: 'python:3.9'
    entrypoint: 'python'
    args: ['-m', 'compiler', '--pipeline', 'my_pipeline.py', '--output', 'pipeline.json']

  # Run the pipeline
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: 'gcloud'
    args:
      - 'ai'
      - 'platform'
      - 'pipelines'
      - 'run'
      - '--region'
      - 'us-central1'
      - '--pipeline'
      - 'pipeline.json'

2. Cloud Build Triggers

You can create triggers in Cloud Build to automatically run your pipeline in response to certain events, such as:

• Pushing to a Git repository: You can set up a trigger to run your pipeline whenever you push a new commit to your Git repository. This is useful for continuous integration.
• Creating a pull request: You can set up a trigger to run your pipeline whenever you create a pull request. This is useful for running tests and validating your code before merging it into the main branch.

Knowledge Check