Tutorial: Dependencies Management¶

dmp-af brings a new way to manage dependencies between models. This is done by using the dependencies section in model's config.

There are two features that dmp-af brings to the table:

• skipping upstream dependencies
• wait not only last upstream model run, but all runs between the last and the current model runs

Skipping upstream dependencies¶

Sometimes it's required to run the model as soon as possible, without waiting for the upstream models to be finished. This can be achieved by setting skip to true in the dependencies section of the model's config.

config:
  dependencies:
    upstream.model.name:
      skip: true

To try this feature, go to dmn_jaffle_analytics/ods/dmn_jaffle_analytics.ods.customers.sql (in the examples) and add the following code to the config section:

dependencies:
  dmn_jaffle_shop.stg.customers:
    skip: true

and run the build_manifest.sh script to build the manifest file.

This will affect dmn_jaffle_analytics__hourly DAG: the dmn_jaffle_analytics.ods.customers model won't have dependency on dmn_jaffle_shop.stg.customers.

With dependency	Without dependency

Wait policy for upstream dependencies¶

If your model with rarer scheduling depends on the model with more frequent scheduling, you may want to wait for all runs of the upstream model between the last and the current model runs.

For example, if the model has @daily scheduling and depends on @hourly model, it's possible to wait for all 24 last runs (default behavior is to wait for only the last run).

This can be achieved by setting wait_policy to all in the dependencies section of the model's config:

config:
  dependencies:
    upstream.model.name:
      wait_policy: all

List of Examples¶

1. Basic Project: a single domain, small tests, and a single target.
2. Advanced Project: several domains, medium and large tests, and different targets.
3. Manual scheduling: domains with manual scheduling.
4. Maintenance and source freshness: how to manage maintenance tasks and source freshness.
5. Kubernetes tasks: how to run dbt models in Kubernetes.
6. Integration with other tools: how to integrate dmp-af with other tools.
7. [Preview] Extras and scripts: available extras and scripts.