Page MenuHomeDevCentral

Run fantoir-datasource as Airflow pipeline
Needs ReviewPublic

Authored by dereckson on Tue, Jan 24, 01:09.

Details

Reviewers
dereckson
Maniphest Tasks
T1750: Import FANTOIR database
Summary

Provide a _pipelines component for Apache Airflow
to run workflows as DAGs.

The fantoir_fetch pipeline will run fantoir-datasource fetch.

When a new version is available, the fetch pipeline will run the fantoir_import
pipeline, to completethe import with import, wikidata and promote.

Pipelines are stored in the dags/ directory,
Python helper code in the dags/nasqueron_datasources/pipelines directory.

Ref T1750

Test Plan

Run workflows

Diff Detail

Repository
rDS Nasqueron Datasources
Lint
No Lint Coverage
Unit
No Test Coverage
Branch
pipelines
Build Status
Buildable 4358
Build 4625: arc lint + arc unit

Event Timeline

dereckson created this revision.