Page MenuHomeDevCentral

Upgrade to AirFlow 2.8.0 and Python 3.11
ClosedPublic

Authored by dereckson on Jan 12 2024, 18:43.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Nov 24, 22:55
Unknown Object (File)
Sun, Nov 24, 22:55
Unknown Object (File)
Sun, Nov 24, 22:32
Unknown Object (File)
Sun, Nov 24, 21:39
Unknown Object (File)
Fri, Nov 22, 10:01
Unknown Object (File)
Fri, Nov 22, 08:31
Unknown Object (File)
Fri, Nov 22, 03:03
Unknown Object (File)
Mon, Nov 18, 03:50
Subscribers
None

Details

Summary

By default, upstream images are still built with Python 3.8.
More recent Python builds

Test Plan

Build the image. The build shows the Sentry SDK is correctly available
in version 1.39.2 for Python 3.11.

Diff Detail

Repository
rDAF Docker image for Airflow
Lint
Lint Passed
Unit
No Test Coverage
Branch
main
Build Status
Buildable 5155
Build 5436: arc lint + arc unit

Event Timeline

dereckson created this revision.
This revision is now accepted and ready to land.Jan 12 2024, 18:43
This revision was automatically updated to reflect the committed changes.

Deployment

Dwellers
$ deploy-containers-service airflow
[…]
----------
          ID: airflow_web
    Function: docker_container.running
      Result: True
     Comment: Replaced container 'airflow_web'. State changed from 'stopped' to 'running'. Container has a new image.
     Started: 18:45:58.009764
    Duration: 6838.744 ms
     Changes:
              ----------
              container_id:
                  ----------
                  added:
                      225f26be8d11efdfe1ca5f6fdaa0292b469411c3fd6f17448cea1ce6f2302748
                  removed:
                      - 7db02e9132e4034668ea6436f6abca659995bd7c3fb9d8e27e09ee2d1aeb7f76
              image:
                  ----------
                  new:
                      sha256:51f5e1b11188b4b10efd6a44528a13a559a937cc18cc9c6f0b1166c8dc8090b7
                  old:
                      sha256:12037a55b9dd7fb6d59308e92b619f8bff2a016ea04889b2bd464b8d8db0f3bd
              state:
                  ----------
                  new:
                      running
                  old:
                      stopped
[…]

$ docker ps -a | grep airflow
a47d45ac75fa   nasqueron/airflow                "/usr/bin/dumb-init …"   37 seconds ago   Up 32 seconds               8080/tcp, 127.0.0.1:46555->5555/tcp                              airflow_flower
3e6ebd708fdd   nasqueron/airflow                "/usr/bin/dumb-init …"   41 seconds ago   Exited (1) 23 seconds ago                                                                    airflow_triggerer
2a598edd0da0   nasqueron/airflow                "/usr/bin/dumb-init …"   46 seconds ago   Up 41 seconds               8080/tcp                                                         airflow_worker
a6f733917e85   nasqueron/airflow                "/usr/bin/dumb-init …"   51 seconds ago   Up 46 seconds               8080/tcp                                                         airflow_scheduler
225f26be8d11   nasqueron/airflow                "/usr/bin/dumb-init …"   57 seconds ago   Exited (1) 36 seconds ago                                                                    airflow_web
c4921b264ee8   nasqueron/airflow                "/usr/bin/dumb-init …"   3 minutes ago    Up 3 minutes                8080/tcp                                                         priceless_nightingale
b41d7ea35c5b   redis                            "docker-entrypoint.s…"   9 months ago     Exited (0) 9 months ago                                                                      airflow_redis

$ airflow nasqueron upgrade
/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:49 MovedIn20Warning: [31mDeprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. [32mTo prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". [36mSet environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message.[0m (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
DB: postgresql+psycopg2://airflow:***@172.27.27.8/airflow
Performing upgrade with database postgresql+psycopg2://airflow:***@172.27.27.8/airflow
[2024-01-12 18:48:05,187] {migration.py:205} INFO - Context impl PostgresqlImpl.
[2024-01-12 18:48:05,187] {migration.py:212} INFO - Will assume transactional DDL.
[2024-01-12 18:48:05,203] {db.py:1572} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Upgrades done
2.5.2

One of the issue with the current procedure is the airflow command is still under 2.5.2:

$ docker logs airflow_web
ERROR: You need to upgrade the database. Please run `airflow db upgrade`. Make sure the command is run using Airflow version 2.8.0.