Capture BigQuery object definitions in the profiler extract - #2603
Open
dgomez04 wants to merge 20 commits into
Open
Capture BigQuery object definitions in the profiler extract#2603dgomez04 wants to merge 20 commits into
dgomez04 wants to merge 20 commits into
Conversation
…er extract. Make object DDL and resolved schema metadata first-class DuckDB outputs so profiling can converge with analysis without a separate definition export.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2603 +/- ##
==========================================
+ Coverage 70.97% 71.02% +0.05%
==========================================
Files 110 112 +2
Lines 10012 10026 +14
Branches 1110 1110
==========================================
+ Hits 7106 7121 +15
- Misses 2692 2693 +1
+ Partials 214 212 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
✅ 181/181 passed, 8 flaky, 2 skipped, 1h0m33s total Flaky tests:
Running from acceptance #5395 |
gueniai
reviewed
Aug 6, 2026
gueniai
left a comment
gueniai
left a comment
Collaborator
There was a problem hiding this comment.
I think this should be an optional extract, and it should be on by default on the desktop UI
Contributor
We can use the existing |
Split inventory and DDL into separate pipeline steps so users can choose inventory_and_ddl (recommended) or inventory-only per run without baking the choice into credentials.
Contributor
Author
|
Tested against our internal BigQuery instance - produced the DDL extracts. |
m-abulazm
requested changes
Aug 18, 2026
m-abulazm
left a comment
m-abulazm
left a comment
Contributor
There was a problem hiding this comment.
I like the refactoring and reuse of the pre-existing execute(). I have commented on some minor things that I would like to change
| return {row[0] for row in rows} | ||
|
|
||
|
|
||
| def test_full_extract_produces_12_tables(monkeypatch, tmp_path, fake_credentials, capsys): |
Contributor
There was a problem hiding this comment.
we still need this test and a new one against the two scripts/pipelines
Contributor
Author
There was a problem hiding this comment.
I intentionally dropped this test because it's redundant. With _run_sql_for_iteration mocked, the test mostly asserts that whatever keys in SQL_FILE_TO_ANALYSIS_FLE show up (and it always will). It doesn't validate the SQL, schema fidelity, or that the definitions actually extract.
End-to-end table creation is better as an integration tests once we have the instances.
Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes ### What does this PR do? This PR upgrades the version of `uv` required by this project: 0.12 is now the current version. (This makes local/dev workflows much more convenient: an old version of `uv` is no longer required.) ### Caveats/things to watch out for when reviewing: uv 0.12.1 is the most recent release that satisfies the cool off policy. ### Tests - existing unit tests - existing integration tests
## Changes This PR updates our dependencies in a few ways: - We use `~=` specifies against the full version, so that pip can only ever install newer patch releases. - Eliminate unbounded upper versions. This prevents newer releases of dependencies introducing a conflict or breaking things. (These haven't been tested, which is necessary.) - Update some range-bounds so that the lower bound matches the current pin: this ensures that installations will have _at least_ the version we're testing against. ### Linked issues Relates: #2608 ### Tests - existing unit tests - existing integration tests
Bump version
### Summary - Fixes `chart_concurrent_users_by_hour` and `rs_avg_concurrent_users` so distinct users are counted within a single calendar day before being reduced to an hour-of-day (`0–23`) profile. - Chart reports the busiest day for each hour (`max`); the KPI averages those per-day/hour counts. ### Root Cause Analysis `query_view` exposes the full retained query history (~7 days from `sys_query_history` / previously `stl_query`) with **no `start_time` filter**. Both metrics then grouped only by `date_part(hour, start_time)` and ran `count(distinct user_id)`. That collapses every day into 24 hour-of-day buckets. Distinct users who ran at the same clock hour on different days are unioned into one count — for example, 5 users at 10:00 on day 1 plus 5 different users at 10:00 on day 2 reported **10**, not a peak of **5**. The chart name and KPI imply a diurnal concurrency profile (users active in that hour on a typical/busy day), not the union of all users ever seen at that clock hour across the retention window. True session-overlap concurrency is out of scope; this fix preserves the existing “distinct users in an hour” semantics and corrects the cross-day aggregation. --------- Co-authored-by: Abhishek Dey <abhishek.dey@databricks.com> Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com> Co-authored-by: Abhishek Dey <89523840+dey-abhishek@users.noreply.github.com> Co-authored-by: Andrew Snare <asnare@users.noreply.github.com>
…2625) The Teradata configurator prompted for the password before host, port, user and database, because the password was read into a local variable ahead of the credential dict literal. This made it look like host/user were never asked when the flow paused at the password prompt. Prompt for the connection fields first (host, port, database, user, password), matching the Oracle and Redshift configurators. The password is still read last since the env vault stores an env-var name. <!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? ### Relevant implementation details ### Caveats/things to watch out for when reviewing: ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #.. ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [ ] added unit tests - [ ] added integration tests
## Changes This PR updates the logging during the streaming of Snowflake profiler results, to improve the availability of progress information and also better indicate the code paths being taken. ### Relevant implementation details ### Linked issues Replaces #2611. ### Functionality - modified existing command: `databricks labs lakebridge execute-database-profiler` ### Tests - [ ] manually tested
## Changes This PR refreshes the locked Yarn/JS dependencies that we use for the documentation, without changing any of the package constraints. ### Caveats/things to watch out for when reviewing: This supersedes a lot of the currently open PRs that have been created by dependabot. ### Linked issues Supersedes: - #2615 - #2614 - #2612 - #2606 - #2604 - #2587 - #2586 - #2583 - #2573 - #2528 - #2519 - #2517 - #2506 - #2478 ### Functionality - affects the user documentation build ### Tests - manually tested: verified the documentation looks okay
…on source systems (#2624) Remove unnecessary ORDER BY queries from SQL that we run on source systems when profiling. These ORDER BY clauses generate extra work on the sources and are not necessary. Consumers of the DuckDB data that rely on order need to specify that explicitly. (In general the relational model does not guarantee order is preserved across insertions and updates: queries need to specify this if they require it.) <!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes This PR removes `ORDER BY` clauses from SQL that we run against source systems when profiling: these are just additional work on the sources and are not necessary. Downstream consumers of the data that rely on the order need to specify that when querying the data. (In general the relation model does not guarantee order is preserved across insertion/update: such queries already need to specify this for correctness.) ### Relevant implementation details Only outer `ORDER BY` clauses have been removed that weren't part of a top-N construct. (In addition, an inner `ORDER BY` has been removed a CTE where the ordering was discarded anyway.) ### Functionality - modified existing command: `databricks labs lakebridge execute-database-profiler`
## Changes ### What does this PR do? This PR updates the checks we do for the source directory when analysing, it only needs to be readable instead of writable. At one point the output was written into the source directory, which meant it needed to be writable. However since the analyzer was updated to allow the output location to be specified this has not been necessary. ### Functionality - modified existing command: `databricks labs lakebridge analyze` ### Tests - manually tested - existing unit tests (needed updating) - existing integration tests
Bumps [mypy](https://github.com/python/mypy) from 1.20.2 to 2.3.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/python/mypy/blob/master/CHANGELOG.md">mypy's changelog</a>.</em></p> <blockquote> <h1>Mypy Release Notes</h1> <h2>Next Release</h2> <h3>Packaging changes</h3> <ul> <li>No longer provide mypyc-accelerated wheels for macOS x86_64 [mypyc-wheels <a href="https://redirect.github.com/python/mypy/issues/119">#119</a>](<a href="https://redirect.github.com/mypyc/mypy_mypyc-wheels/pull/119">mypyc/mypy_mypyc-wheels#119</a>)</li> </ul> <h2>Mypy 2.3</h2> <p>We've just uploaded mypy 2.3.0 to the Python Package Index (<a href="https://pypi.org/project/mypy/">PyPI</a>). Mypy is a static type checker for Python. This release includes new features, performance improvements and bug fixes. You can install it as follows:</p> <pre><code>python3 -m pip install -U mypy </code></pre> <p>You can read the full documentation for this release on <a href="http://mypy.readthedocs.io">Read the Docs</a>.</p> <h3>The Upcoming Switch to the New Native Parser</h3> <p>We are planning to enable the new native parser (<code>--native-parser</code>) by default soon. We recommend that you test the native parser in your projects and report any issues in the <a href="https://github.com/python/mypy/issues">mypy issue tracker</a>.</p> <h3>Mypyc Free-threading Memory Safety</h3> <p>Free-threaded Python builds that don't have the GIL require additional synchronization primitives or lock-free algorithms to ensure memory safety when there are race conditions (for example, when a thread reads a list item while another thread writes the same list item concurrently). This release greatly improves memory safety of free threading.</p> <p>List operations are now memory-safe on free threaded Python builds, even in the presence of race conditions. This has some performance cost. For list-heavy workloads, using <code>librt.vecs.vec</code> instead of list is often significantly faster, but note that <code>vec</code> is not (and likely won't be) fully memory safe, and the user is expected to avoid race conditions. The newly introduced <code>librt.threading.Lock</code> helps with this. Using variable-length tuples can also be more efficient than lists, since tuples are immutable and don't require expensive synchronization to ensure memory safety.</p> <p>Instance attribute access is also (mostly) memory safe now on free-threaded builds in the presence of race conditions. We are planning to fix the remaining unsafe cases in a future release.</p> <p>Full list of changes:</p> <ul> <li>Make attribute access memory safe on free-threaded builds (Jukka Lehtosalo, PR <a href="https://redirect.github.com/python/mypy/pull/21705">21705</a>)</li> <li>Fix unsafe borrowing of instance attributes with free-threading (Jukka Lehtosalo, PR <a href="https://redirect.github.com/python/mypy/pull/21688">21688</a>)</li> <li>Make list get/set item more memory safe on free-threaded builds (Jukka Lehtosalo, PR <a href="https://redirect.github.com/python/mypy/pull/21683">21683</a>)</li> <li>Don't borrow list items on free-threaded builds (Jukka Lehtosalo, PR <a href="https://redirect.github.com/python/mypy/pull/21679">21679</a>)</li> <li>Make multiple assignment from list memory-safe on free-threaded builds (Jukka Lehtosalo, PR <a href="https://redirect.github.com/python/mypy/pull/21684">21684</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python/mypy/commit/8aabf8435357eaffceca7237f371e293b8168e54"><code>8aabf84</code></a> Drop +dev from version</li> <li><a href="https://github.com/python/mypy/commit/4d8ad2ab5e86c99581b73775f2c00b9b8265b589"><code>4d8ad2a</code></a> Update changelog for 2.3 release (<a href="https://redirect.github.com/python/mypy/issues/21728">#21728</a>)</li> <li><a href="https://github.com/python/mypy/commit/2c2154672040c52e481f423854d104e6cf172585"><code>2c21546</code></a> [mypyc] Update documentation of race conditions under free threading (<a href="https://redirect.github.com/python/mypy/issues/21726">#21726</a>)</li> <li><a href="https://github.com/python/mypy/commit/a9f62a3cf98a58a7a2607b7c81695802b39f5edc"><code>a9f62a3</code></a> [mypyc] Make attribute access memory safe on free-threaded builds (<a href="https://redirect.github.com/python/mypy/issues/21705">#21705</a>)</li> <li><a href="https://github.com/python/mypy/commit/0faa413ebf7c924a864ef5dabd70303d898e7766"><code>0faa413</code></a> Use <code>PYODIDE</code> environment variable for Emscripten cross-compilation detection...</li> <li><a href="https://github.com/python/mypy/commit/3d75cdb09f0928fa8b83e5ef03572ed878ac8d09"><code>3d75cdb</code></a> [mypyc] Borrow final attributes more aggressively (<a href="https://redirect.github.com/python/mypy/issues/21702">#21702</a>)</li> <li><a href="https://github.com/python/mypy/commit/24c237d85b48f618e655ffff1dc0f19089d9b599"><code>24c237d</code></a> [mypyc] Improve documentation of Final (<a href="https://redirect.github.com/python/mypy/issues/21713">#21713</a>)</li> <li><a href="https://github.com/python/mypy/commit/b5be217392b9b2771d1764066b9d600bf93ce7a8"><code>b5be217</code></a> [mypyc] Update free threading Python compatibility docs (<a href="https://redirect.github.com/python/mypy/issues/21711">#21711</a>)</li> <li><a href="https://github.com/python/mypy/commit/cbcb51add3094ec91b29cdd4c624943bf251b63f"><code>cbcb51a</code></a> Narrow for frozendict membership check (<a href="https://redirect.github.com/python/mypy/issues/21709">#21709</a>)</li> <li><a href="https://github.com/python/mypy/commit/af2bc0f3cc7f2f129f0c11294158d0c292692c3d"><code>af2bc0f</code></a> Sync typeshed (<a href="https://redirect.github.com/python/mypy/issues/21707">#21707</a>)</li> <li>Additional commits viewable in <a href="https://github.com/python/mypy/compare/v1.20.2...v2.3.0">compare view</a></li> </ul> </details> <br /> > **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Snare <asnare@users.noreply.github.com> Co-authored-by: Takero Ibuki <takero.ibuki@databricks.com> Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com> Co-authored-by: Bishwajit <bishwajit.dey@databricks.com> Co-authored-by: Diego Gómez Moreno <diego.gomez@databricks.com> Co-authored-by: Abhishek Dey <89523840+dey-abhishek@users.noreply.github.com> Co-authored-by: Abhishek Dey <abhishek.dey@databricks.com> Co-authored-by: Andrew Snare <andrew.snare@databricks.com>
Bumps [typescript](https://github.com/microsoft/TypeScript) from 5.9.3 to 7.0.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/microsoft/TypeScript/releases">typescript's releases</a>.</em></p> <blockquote> <h2>TypeScript 6.0.3</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.2%22">fixed issues query for TypeScript 6.0.2 (Stable)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.3%22">fixed issues query for TypeScript 6.0.3 (Stable)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.2%22">fixed issues query for TypeScript 6.0.2 (Stable)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0.1 RC</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0-rc/">release announcement blog post</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22">fixed issues query for TypeScript 6.0.0 (Beta)</a>.</li> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.1%22">fixed issues query for TypeScript 6.0.1 (RC)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> <h2>TypeScript 6.0 Beta</h2> <p>For release notes, check out the <a href="https://devblogs.microsoft.com/typescript/announcing-typescript-6-0-beta/">release announcement</a>.</p> <ul> <li><a href="https://github.com/Microsoft/TypeScript/issues?utf8=%E2%9C%93&q=milestone%3A%22TypeScript+6.0.0%22+is%3Aclosed+">fixed issues query for Typescript 6.0.0 (Beta)</a>.</li> </ul> <p>Downloads are available on:</p> <ul> <li><a href="https://www.npmjs.com/package/typescript">npm</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/microsoft/TypeScript/commits">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~microsoft1es">microsoft1es</a>, a new releaser for typescript since your current version.</p> </details> <br /> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Snare <andrew.snare@databricks.com>
## Changes This is a preliminary change to the repo to ensure that ruff's import-checking linters pass. These checks are enabled in the next version: turning them on here means that the changes needed for ruff can be implemented piecemeal. These supersede the equivalent checks from pylint, which we want to eliminate. ### Relevant implementation details Only imports are affected by this PR; there are no behavioural changes. ### Tests - existing unit tests - existing integration tests
Bumps [postcss-nesting](https://github.com/csstools/postcss-plugins/tree/HEAD/plugins/postcss-nesting) from 13.0.2 to 14.0.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/csstools/postcss-plugins/blob/main/plugins/postcss-nesting/CHANGELOG.md">postcss-nesting's changelog</a>.</em></p> <blockquote> <h3>14.0.1</h3> <p><em>July 23, 2026</em></p> <ul> <li>Updated <a href="https://github.com/csstools/postcss-plugins/tree/main/packages/selector-resolve-nested"><code>@csstools/selector-resolve-nested</code></a> to <a href="https://github.com/csstools/postcss-plugins/tree/main/packages/selector-resolve-nested/CHANGELOG.md#401"><code>4.0.1</code></a> (patch)</li> </ul> <h3>14.0.0</h3> <p><em>January 14, 2026</em></p> <ul> <li>Updated: Support for Node <code>20.19.0</code> or later (major).</li> <li>Removed: <code>commonjs</code> API. In supported Node versions <code>require(esm)</code> will work without needing to make code changes.</li> <li>Updated <a href="https://github.com/csstools/postcss-plugins/tree/main/packages/selector-resolve-nested"><code>@csstools/selector-resolve-nested</code></a> to <a href="https://github.com/csstools/postcss-plugins/tree/main/packages/selector-resolve-nested/CHANGELOG.md#400"><code>4.0.0</code></a> (major)</li> <li>Updated <a href="https://github.com/csstools/postcss-plugins/tree/main/packages/selector-specificity"><code>@csstools/selector-specificity</code></a> to <a href="https://github.com/csstools/postcss-plugins/tree/main/packages/selector-specificity/CHANGELOG.md#600"><code>6.0.0</code></a> (major)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/csstools/postcss-plugins/commits/HEAD/plugins/postcss-nesting">compare view</a></li> </ul> </details> <br /> > **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Snare <andrew.snare@databricks.com>
## Changes This PR upgrades docusaurus, used to build our documentation: 3.9.2 → 3.10.2 ### Relevant implementation details Further to the version bumps, the dependabot configuration has been adjusted so that it will group docusaurus updates instead of creating individual PRs. ### Linked issues Rolls up: - #2469 - #2470 - #2471 ### Tests - manually tested: built (via `make docs-serve`) and then eye-balled the documentation, nothing seems amiss. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Drop preview managed_table_type, remove unused metadata __main__, expose execute() on inventory/definitions entrypoints, and tidy constants/test helpers.
|
|
…-profiling Keep the Snowflake key-pair auth work from main (cryptography dependency, snowflake_auth imports, load_snowflake_private_key) alongside the BigQuery object-definition extracts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Customers today run the BigQuery profiler for inventory/usage metrics, then separately export object definitions for analysis. This pull request starts converging those flows by making object definitions first-class outputs of the profiler DuckDB extract.
table_definitionsOne row per table-like object from
INFORMATION_SCHEMA.TABLES:table_catalog/table_schema/table_name)table_type,managed_table_type,creation_timeddl(CREATE TABLE / VIEW / …)default_collation_namebase_table_*,snapshot_time_msreplica_source_*Omits transient replication health fields (
replication_status/replication_error).base_table_*andreplica_source_*are source structural provenance, not Unity Catalog runtime lineage.column_definitionsOne row per top-level column from
INFORMATION_SCHEMA.COLUMNS:ordinal_positiondata_type, nullability, defaults, collation, roundingpolicy_tags(serialized to JSON for DuckDB)Nested STRUCT/ARRAY type expressions remain in
data_type;COLUMN_FIELD_PATHSis deferred as too granular for current consumers.routine_definitionsOne row per UDF / table function / aggregate function / procedure from
INFORMATION_SCHEMA.ROUTINES:routine_type, returndata_typeroutine_body/routine_definition/external_languageconnectionddlTest plan
tests/unit/assessment/test_bq_metadata_extract.py(expects 15 analysis tables)execute-database-profiler --source-tech bigqueryagainst a project/region and inspecttable_definitions,column_definitions,routine_definitionsin the DuckDB extractbigquery.tables.listandbigquery.routines.getwhere least-privilege roles are used