Page MenuHomeDevCentral

Deploy Airflow to Dwellers
ClosedPublic

Authored by dereckson on Mar 29 2023, 19:58.
Tags
None
Referenced Files
F2964123: D2943.id7493.diff
Sat, May 18, 22:06
Unknown Object (File)
Mon, May 13, 23:30
Unknown Object (File)
Mon, May 13, 17:32
Unknown Object (File)
Mon, May 13, 17:32
Unknown Object (File)
Mon, May 13, 17:32
Unknown Object (File)
Mon, May 13, 17:32
Unknown Object (File)
Sun, May 12, 12:39
Unknown Object (File)
Sun, May 12, 06:35
Subscribers
None

Details

Summary

D2754 offers to run fantoir-datasource as a Airflow pipeline.

To do so, we need an Airflow instance, which means:

  • PostgreSQL database on db-A cluster
  • Redis and application containers on the Docker PaaS

Like Sentry, Airflow service is deployed by realm, a realm being
a network and a set of containers for that service, sharing resources
like credentials.

Plan is to deploy it to Dwellers for evaluation purpose,
then move it to docker-002 if we adopt it in production.

Ref T1750

Test Plan

Deploy on Dwellers

Diff Detail

Repository
rOPS Nasqueron Operations
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

dereckson created this revision.
dereckson retitled this revision from Deploy Airflow to Deploy Airflow to Dwellers.Mar 29 2023, 21:12
dereckson edited the summary of this revision. (Show Details)

Airflow runs well.

Two changes needed:

  • switch IP to 127.0.0.1
  • nginx (and a DNS record so)

Bind app ports to 127.0.0.1

This revision is now accepted and ready to land.Mar 30 2023, 01:37
This revision was automatically updated to reflect the committed changes.