Repository navigation
Conversation
…base SnowflakeEngineAdapter.get_table_last_modified_ts queried the unqualified INFORMATION_SCHEMA.TABLES view, which only contains objects from the connection's current database. Tables in any other database were never found, so the freshness signal raised MissingSourceError for models that read external tables from another database. Group the tables by database and query each database's own INFORMATION_SCHEMA.TABLES. The query is now built with sqlglot instead of string formatting, so table and schema names are escaped as literals. Signed-off-by: Dhruv Gupta <todhruvg@gmail.com>
Collaborator
|
Hey @DG47 — thanks for fixing this. Querying each database's own INFORMATION_SCHEMA is the right fix, and the new test covers the RAW vs ANALYTICS case from the issue. One edge case: the tables are grouped by Could you key the groups on the catalog identifier itself, so quoted and unquoted names stay separate? A small test that passes both Thanks again for taking the time to put this up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #6131.
SnowflakeEngineAdapter.get_table_last_modified_tsreadLAST_ALTEREDfrom the unqualifiedINFORMATION_SCHEMA.TABLESview. In Snowflake that view only lists tables in the session's current database, so tables in any other database were never returned. Thefreshness()signal then raisedMissingSourceErrorfor models that read external tables from a different database than the gateway's (for exampleRAWvsANALYTICS).This change groups the requested tables by database and queries each database's own
INFORMATION_SCHEMA.TABLES("RAW".INFORMATION_SCHEMA.TABLES, and so on). TheTABLE_CATALOGpredicate is no longer needed because each view is already scoped to one database. The BigQuery implementation groups per dataset in the same way. The query is now built with sqlglot rather than string formatting, so table and schema names are escaped as string literals. Tables without a catalog still go to the current database's information schema.Test Plan
test_get_table_last_modified_ts_multiple_catalogstotests/core/engine_adapter/test_snowflake.py. It requests three tables across two databases and checks that one query per database is issued and the timestamps are combined. It fails onmain(one unqualified query) and passes with this change.pytest tests/core/engine_adapter -m "not docker and not remote" -n 8: 714 passed, 3 skipped.pytest -n 8 -m "fast and not web and not github and not dbt and not jupyter and not cicdonly and not isolated": 2436 passed, 6 failed. The 6 failures are the pyodbc connection-string tests, which fail locally onmaintoo because unixODBC is not installed.SKIP=prettier,eslint pre-commit run --all-files: ruff, ruff-format, mypy and valid migrations all pass.The Snowflake freshness integration tests are skipped for Snowflake (
_skip_snowflakeintest_freshness.py), and I did not run this against a live Snowflake account.Checklist
make styleand fixed any issuesmake fast-test)git commit -s) per the DCO