HomeDevCentral

Run fantoir-datasource as Airflow pipeline
17a993716152Unpublished

Unpublished Commit · Learn More

Not On Permanent Ref: This commit is not an ancestor of any permanent ref.
This commit has been deleted in the repository: it is no longer reachable from any branch, tag, or ref.

Description

Run fantoir-datasource as Airflow pipeline

Summary:
Provide a _pipelines component for Apache Airflow
to run workflows as DAGs.

The fantoir_fetch pipeline will run fantoir-datasource fetch.

When a new version is available, the fetch pipeline will run the fantoir_import
pipeline, to completethe import with import, wikidata and promote.

Pipelines are stored in the dags/ directory,
Python helper code in the dags/nasqueron_datasources/pipelines directory.

Ref T1750

Test Plan: Run workflows

Reviewers: dereckson

Maniphest Tasks: T1750

Differential Revision: https://devcentral.nasqueron.org/D2754

Details

Provenance
derecksonAuthored on Jan 24 2023, 01:14
derecksonPushed on Jan 14 2024, 21:25
Parents
rDSfc13adda27ce: Use fantoir_YYYMM format to suggest FANTOIR table name
Branches
Unknown
Tags
Unknown

Event Timeline