statmike/gcp-mlops

Google Cloud Platform Vertex AI end-to-end workflows for machine learning operations

📚 What is gcp‑mlops?

A massive, community‑maintained collection of interactive Jupyter notebooks that demonstrate end‑to‑end machine‑learning (ML) and generative‑AI workflows on Google Cloud. The repo tracks the evolution of Google’s AI platform—from Cloud ML Engine to Vertex AI and the newest Gemini Enterprise Agent Platform—showing how to stitch together services such as BigQuery, Dataflow, Dataproc, Managed Airflow, Cloud Run, GKE, and a host of Google databases into production‑grade MLOps pipelines.

🎯 Core purpose

  • Learn: step‑by‑step, hands‑on notebooks covering everything from data ingestion to model serving.
  • Adapt: reusable patterns you can copy into your own projects.
  • Showcase: how Google Cloud services interoperate for training, serving, monitoring, feature‑stores, RAG, agents, and more.

📂 Repository layout (high‑level)

Section Notebooks Highlights
MLOps 74 Vertex AI training, serving (online, batch, SQL), feature‑store implementations, KFP pipelines, model evaluation/monitoring, experiment tracking
data+ai 40 In‑database AI with BigQuery ML functions, streaming inference via Dataflow, Spark inference on Dataproc, Airflow orchestration, SQL‑based inference on AlloyDB/Spanner
Applied GenAI 53 Gemini SDK, multimodal generation, vector search across 11 GCP databases, RAG engine, grounding checks, ranking, evaluation
Applied Forecasting 20 Time‑series with BQML ARIMA+, Prophet, TensorFlow/Seq2Seq, foundation models (TimesFM), KFP pipelines
Framework Workflows 34 End‑to‑end pipelines for PyTorch, Keras/JAX, CatBoost, Vertex AI AutoML, Flax, R – all sharing a common data source and model registry
Applied ML 26 AI agents built with ADK/MCP, solution prototypes (document processing, product classification), optimization with Vertex AI Vizier
Extras dozens TensorFlow, scikit‑learn, R, AutoML deep‑dives, tips, Document AI guides

🚀 Typical use‑cases

  • Building a production MLOps pipeline on Vertex AI with KFP, feature‑store, and automated monitoring.
  • Deploying LLMs (e.g., Gemini, vLLM, Triton) for online or batch inference, including vector‑search‑backed RAG.
  • Running inference where the data lives – using BigQuery ML functions, Dataflow RunInference, or Spark on Dataproc.
  • Creating AI agents with the Gemini Enterprise Agent Platform and deploying them via Agent Engine.
  • Time‑series forecasting using both classic statistical models (ARIMA+, Prophet) and foundation models (TimesFM).
  • Experimenting with feature‑store designs – managed Vertex Feature Store vs. self‑managed Bigtable.

🛠️ Getting started (quick steps)

  1. Clone the repo
    git clone https://github.com/statmike/gcp-mlops.git
cd gcp-mlops
  2. Open a notebook – each sub‑folder contains a README.md with a short description and a link to the .ipynb file.
  3. Set up a GCP project with the required APIs (Vertex AI, BigQuery, Dataflow, etc.). The notebooks include a helper cell that runs gcloud services enable … for you.
  4. Authenticate in the notebook:
    from google.colab import auth
auth.authenticate_user()
    (or use gcloud auth login locally).
  5. Run the cells – most notebooks spin up only the necessary managed services, so you can experiment on a modest VM.
  6. Adapt – copy the notebook sections you need into your own repo, replace the sample data with your own, and modify the pipeline definitions.

📜 License & contribution

  • License: Apache 2.0 – free for commercial and non‑commercial use.
  • Contributions: welcomed via pull requests; see CONTRIBUTING.md and the code of conduct for guidelines.
  • Disclaimer: Not an official Google product; built on Google Cloud services but not Google‑endorsed.

🔗 Where to go next?

  • Explore the MLOps → Pipelines notebooks to see a full KFP workflow.
  • Dive into Applied GenAI → Retrieval to compare vector‑search latency across BigQuery, Vertex AI Vector Search, and other stores.
  • Try an AI Agent project under Applied ML → AI Agents to see the new Gemini Enterprise Agent Platform in action.

This summary is based solely on the repository’s README; the notebooks themselves contain the detailed code and explanations.

Related

  • Project
    luweiagi/machine-learning-notes

    luweiagi/machine-learning-notes is a Chinese-language, notebook-style project that organizes a large set of machine-learning study notes into a nested table of contents covering math foundations, programming basics, classical ML, and deep learning. It's designed to be read as a rendered website (GitHub Pages) because the source view doesn't display math formulas.

  • Project
    FilippoAiraldi/mpc-reinforcement-learning

    A library for training model-based Reinforcement Learning agents using Model Predictive Control as a function approximation to learn optimal control policies.

  • Project
    brainpy/BrainPy

    BrainPy is a Python framework for computational neuroscience that lets researchers build, simulate, train, and analyze brain dynamics models efficiently using just-in-time compilation. It provides an integrated, extensible platform that runs on CPUs, GPUs, and TPUs.

  • Project
    NVIDIA/torch-harmonics

    torch‑harmonics is a PyTorch library for differentiable spherical harmonic transforms, spherical convolutions, and attention mechanisms, enabling AI models (e.g., Spherical Fourier Neural Operators) to work directly with data on the sphere. It provides GPU‑accelerated modules, multiple grid types, distributed support, and easy installation via pre‑built wheels or source builds.