AirflowJWTManager recovery

Description

Airflow DAGs need a valid JWT and X.509 certificates to communicate with Hopsworks and Hops. JWTs should be renewed frequently otherwise tasks will fail to execute. AirflowJWTManager in Hopsworks is responsible to issue JWTs for new DAGs, renew tokens for non paused DAGs and invalidate non active tokens.

In case Hopsworks restarts, it will lose all information of which tokens should be renewed but Airflow tasks will keep running, and eventually will fail. We need a recovery mechanism for AirflowJWTManager to rebuild its internal in-memory state during startup.

Assignee

Antonios Kouzoupis

Reporter

Antonios Kouzoupis

Labels

None

Fix versions

Priority

Medium
Configure