From Scripts to Components

A well-designed ML pipeline is made up of a series of modular, reusable components. Each component should perform a single, well-defined task. This makes your pipeline easier to build, test, and maintain.

1. Identifying Pipeline Components

When designing a pipeline, you should start by breaking down your ML workflow into a series of steps. Each step should be a candidate for a pipeline component.

Some common pipeline components include:

• Data Ingestion: Reading data from a source (e.g., BigQuery, Cloud Storage).
• Data Validation: Checking the quality of your data using TFDV.
• Data Preprocessing: Transforming your data into a format that can be used for training.
• Model Training: Training your model.
• Model Analysis: Evaluating your model's performance using TFMA.
• Model Deployment: Deploying your model to an endpoint for serving.

graph LR
    classDef storage fill:#f9f,stroke:#333,stroke-width:2px;
    classDef component fill:#00d2ff,stroke:#00a8cc,color:#fff,stroke-width:2px;
    classDef model fill:#34d399,stroke:#059669,color:#fff,stroke-width:2px;

    Source[(Data Source)] --> Ingest[Data Ingestion]
    Ingest --> Validate[Data Validation]
    Validate --> Preprocess[Data Preprocessing]
    Preprocess --> Train[Model Training]
    Train --> ModelNode([Trained Model])
    ModelNode --> Analyze[Model Analysis]
    Analyze --> Deploy[Model Deployment]
    Deploy --> Endpoint([Serving Endpoint])

    class Source storage;
    class Ingest,Validate,Preprocess,Train,Analyze,Deploy component;
    class ModelNode,Endpoint model;

2. Pipeline Triggers

Once you have a pipeline, you need to decide how to trigger it to run. There are two main types of triggers:

• Manual Triggers: You can manually trigger a pipeline to run from the Google Cloud Console or using the gcloud command-line tool.
• Automated Triggers: You can set up automated triggers to run your pipeline in response to certain events.

Common Automated Triggers

• Scheduled Triggers: Run your pipeline on a regular schedule (e.g., every day, every week).
• Event-based Triggers: Run your pipeline in response to an event, such as:
  • New data: A new file is uploaded to a Cloud Storage bucket.
  • New code: A new commit is pushed to a Git repository.
  • New model: A new model is registered in the Vertex AI Model Registry.

You can use Cloud Functions or Cloud Pub/Sub to create event-based triggers.

Knowledge Check