To run pipeline DAGs for datasources workflow, plan is to evaluate Airflow.

Plan is to:

• run containers for Airflow in our Docker PaaS
• deploy through Jenkins DAG content like in D2754 to /srv/airflow/nasqueron/dags
• monitor errors through Sentry

Fallback plan if evaluation isn't successful

In June, evaluate if we keep or not Airflow.

If not, as a fallback, alternative plans are:

• fantoir-datasource workflow deploy.yaml: a YAML document tell what commands to run with what options
• Jenkins pipeline to run each fantoir-datasource command
• Both: Jenkins is responsible to run fantoir-datasource workflow deploy.yaml and fantoir-datasource workflow wikidata.yaml

This plan needs to be put in action before decom Airflow.