Overview
An open-source data pipeline module that fetches Looker platform monitoring data via the Looker API and exports it to Google BigQuery. Built to give teams visibility into how their BI platform is actually being used — dashboard views, query history, user activity, and content adoption.
Architecture
- Extraction — Pulls usage and monitoring data from Looker's REST API
- Validation — Pydantic models enforce strict schema validation on extracted data before transformation
- Transformation — Cleans, normalizes, and reshapes raw API responses into analytics-ready tables
- Loading — Inserts validated data into BigQuery with proper partitioning and clustering
Technical Stack
- Python — Core pipeline logic and orchestration
- Pydantic — Data validation and schema enforcement
- SQL — BigQuery transformations and data modeling
- GCP BigQuery — Destination warehouse
- Docker — Containerized deployment for consistency across environments
Data Model
The pipeline materializes tables covering dashboard usage patterns, query performance metrics, user engagement trends, and content adoption rates — turning raw API data into actionable BI-on-BI insights.
Open Source
Listed as an open-source contribution. The module is designed to be reusable across any Looker + BigQuery setup.