I need to add delta data to table x. So, I need to create a cycle. I read table x, then I find the delta between the source and table x. After that, I insert the delta to table x. How to do this in dbt? Is it possible to break the cycle?
3 Answers
I change it from ref to source. so instead of using ref, I use source. I create another schema for the source. just to break the cycle in dbt. so basically, 1 table has ref and source at the same time.
what you need is an incremental model, have a look at: incremental models in DBT
6If the full scan approach is fine, then you can just create a query
select col1,col2, col2... from source_Table minus select col1,col2, col2... from x this will give you all the records that are there in source_table but not in table x.
If the full scan is not fine, then you can need to make sure that you have updated_timestamp in your source table, and based on that you can write an incremental model.
2
