Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Incremental strategy ignore does not ignore missing columns, throws error instead #237

Open
2 tasks done
bruno-szdl opened this issue Jun 11, 2024 · 0 comments
Open
2 tasks done
Labels
enhancement New feature or request help_wanted Extra attention is needed

Comments

@bruno-szdl
Copy link

Is this a new bug?

  • I believe this is a new bug
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

In the docs it says

This is the behavior if on_schema_change: ignore, which is set by default, and on older versions of dbt.

If you add a column to your incremental model, and execute a dbt run, this column will not appear in your target table.

Similarly, if you remove a column from your incremental model, and execute a dbt run, this column will not be removed from your target table.

When I add a column it ignores it, but when I remove a column it raises an error. I tested it in 3 different adapters (snowflake, bigquery, duckdb) and it throws the same error. It says it can't find the missing column.

Expected Behavior

I expected it not to throw an error and just ignore it (add nulls maybe).

Steps To Reproduce

Run this model

{{ config(
    materialized='incremental'
    , on_schema_change='ignore'
) }}

select 
    1 as id
    , 100 as value
    , 'created' as status
    , cast('2024-01-01' as date) as date
union all
select 
    2 as id
    , 200 as value
    , 'processed' as status
    , cast('2024-01-02' as date) as date

Then remove a column and try rerunning it. In this case, I am also adding a new column

{{ config(
    materialized='incremental'
    , on_schema_change='ignore'
) }}

select 
    3 as id
    , 300 as value
    , True as is_processed
    , cast('2024-01-03' as date) as date

Relevant log output

Snowflake:
SQL compilation error: error line 3 at position 34
invalid identifier 'STATUS'

BigQuery:
Unrecognized name: status at [38:25]

DuckDB:
Binder Error: Referenced column "status" not found in FROM clause!

Environment

- OS: MacOS
- Python: 3.9.6
- dbt-adapters: dbt-snowflake, dbt-bigquery, dbt-duckdb

Additional Context

version 1.8

I also tested on 1.3 to test an older version, and same problem

@bruno-szdl bruno-szdl added bug Something isn't working triage labels Jun 11, 2024
@amychen1776 amychen1776 added enhancement New feature or request and removed bug Something isn't working triage labels Aug 28, 2024
@amychen1776 amychen1776 changed the title [Bug] Incremental strategy ignore does not ignore missing columns, throws error instead [Feature] Incremental strategy ignore does not ignore missing columns, throws error instead Aug 28, 2024
@amychen1776 amychen1776 added the help_wanted Extra attention is needed label Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help_wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants