Skip to content

fix serialize_dag schema migration data column type#13269

Closed
houqp wants to merge 1 commit intoapache:masterfrom
houqp:qp_migration
Closed

fix serialize_dag schema migration data column type#13269
houqp wants to merge 1 commit intoapache:masterfrom
houqp:qp_migration

Conversation

@houqp
Copy link
Member

@houqp houqp commented Dec 23, 2020

text column in mysql is too small for large DAGs, resulting in invalid json blob being stored.

@houqp houqp requested a review from kaxil December 23, 2020 04:52
@potiuk
Copy link
Member

potiuk commented Dec 23, 2020

Should not we add another migration for that? I think the gennie is out of the bottle now, and in order to fix it for someone who already run the migration, we have to add a new migration.

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

@kaxil
Copy link
Member

kaxil commented Dec 23, 2020

Yup we will need to add another migration for this.

Should we make the Column Type consistent with f66a46d#diff-fd7ce54d2146064dd83398f84ab648a0ffad06e941a76c53c82901257204f779 ?

@kaxil kaxil added this to the Airflow 2.0.2 milestone Feb 4, 2021
conn.execute("SELECT JSON_VALID(1)").fetchone()
except (sa.exc.OperationalError, sa.exc.ProgrammingError):
json_type = sa.Text
json_type = sa.LargeBinary
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not LargeText? (I.e. and this col be binary or text?)

@houqp
Copy link
Member Author

houqp commented Feb 19, 2021

After thinking more about this, we should just drop this change because in Airflow 2.x, all these dags will be stored as json type in MySQL (>5.6) or Postgres. So perhaps what we should do instead is to add a check in airflow-upgrade-check to see if existing column has been converted to json?

@ashb
Copy link
Member

ashb commented Feb 19, 2021

After thinking more about this, we should just drop this change because in Airflow 2.x, all these dags will be stored as json type in MySQL (>5.6) or Postgres. So perhaps what we should do instead is to add a check in airflow-upgrade-check to see if existing column has been converted to json?

What would have converted that type to JSON?

(Users shouldn't ever have to manually edit the metadata db -- that's what migrations are for)

@ashb ashb removed this from the Airflow 2.0.2 milestone Mar 18, 2021
@ashb
Copy link
Member

ashb commented Mar 18, 2021

@houqp Should we close this PR then?

@github-actions
Copy link

github-actions bot commented May 4, 2021

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label May 4, 2021
@github-actions github-actions bot closed this May 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale Stale PRs per the .github/workflows/stale.yml policy file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants