End-to-end lineage on DataHub, from Airflow to the dashboard

During my year on Geotab’s data enablement team, the question I heard most often was some version of where does this number come from? A Superset dashboard shows a metric, the metric comes from a BigQuery table, the table is written by an Airflow task, and that task reads three other tables. All of that was knowable, but only by someone who already knew it. I built the pipeline that made it browsable.

The result is a single Airflow DAG that runs every morning and populates DataHub with four kinds of metadata plus the edges between them. Open a table in DataHub and you can walk left to the pipeline that wrote it and right to every chart and dashboard that reads it, with owners and data stewards attached at every hop.

The picture above is what that looks like for one table: upstream tables and the Airflow tasks that write them on the left, the Superset datasets that read it on the right, 135 downstream assets in total.

Source: github.com/zhuhongd/datahub_ingestion

What gets ingested

BigQuery tables and columns. Table-level metadata (description, labels, size, jobs in the last 90 days, functional area, owner, steward) comes from a daily metadata inventory table rather than from the BigQuery API, so one query covers thousands of tables across projects and regions. Column schemas are attached as DataHub schema aspects.

Table-to-table lineage. BigQuery’s job history already knows which tables a query read and which it wrote. A lineage table distils that into (referenced table, destination table) pairs per day; the DAG turns each pair into an upstream-lineage aspect in DataHub.

Airflow pipelines and tasks. Every DAG becomes a DataFlow entity and every task a DataJob, tagged with its Composer environment and region, with the owner and stewards from the DAG’s metadata and a link straight to the task in the Airflow grid. A second SQL pass maps tasks to the tables they produce, which is the edge that connects orchestration to data.

Superset datasets, charts and dashboards. Datasets point at their BigQuery tables, charts point at their datasets, dashboards list their charts. Ownership, publication status and last-visit timestamps ride along as tags and custom properties.

Put together, the graph reads: Airflow task → BigQuery table → BigQuery table → Superset dataset → chart → dashboard.

How it’s built

• DataHub’s Python SDK, not the UI connectors. Each entity is assembled from aspects (DataFlowInfo, Ownership, UpstreamLineage, DashboardInfo, and so on), wrapped in a MetadataChangeProposalWrapper, and emitted with DatahubRestEmitter. URNs are built with the SDK’s URN classes so environment and platform instance are consistent everywhere.
• Sensors before work. Each ingestion task waits on a BigQueryTablePartitionExistenceSensor for that day’s partition of its source inventory table, in reschedule mode so it doesn’t hold a worker slot while it waits.
• Environment mapping. Project names ending in -prod, -raw, -test or -uat map to DataHub’s PROD / TEST / UAT fabrics, so the same table in two environments becomes two entities instead of one confusing merge.
• Secrets and auth. DataHub sits behind Google’s Identity-Aware Proxy, so every call carries an OIDC identity token as a proxy authorization header plus a DataHub personal token; both are fetched at run time from Vault through an internal secrets service, never stored in the repo.
• Deployment. GitLab CI ships the DAG folder to Cloud Composer; the DAG is created paused so a deploy never fires an unreviewed backfill.

Things I’d do differently

• Filtering out date-suffixed table copies (_2020…) with NOT LIKE patterns in SQL was the quickest fix and the ugliest one. A proper allowlist keyed on the inventory’s table type would be cleaner.
• Lineage is table-to-table. Column-level lineage from BigQuery’s job statistics is possible and would answer the which column version of the question.
• The four ingestion scripts share a lot of boilerplate (platform instance creation, environment mapping, owner parsing). One small shared module would have cut the repo roughly in half.

Why it mattered

Lineage sounds like plumbing until you need it. Once it was in DataHub, deprecating a table meant looking at its downstream dashboards instead of asking around, onboarding meant browsing a graph instead of reading tribal knowledge, and “who owns this?” had an answer on the page. It was also my first contact with Google ADK, which was being trialled on the same team, and the first time I had built something the whole data organisation used every day. The agent platform I run now started from that habit: make the system explain itself.