Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upImplement strict projection in more transforms #35
Labels
Comments
This was referenced Jul 13, 2019
|
I think we could close this issue. Shall we? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Strict projection isn't required and wasn't implemented for several of the partitioning transformations. When strict projection isn't implemented (the
projectStrictmethod returnsnull) Iceberg will fall back to a safe implementation. For example, residual evaluation will not remove predicates because they cannot be guaranteed to be true, and deletes can't determine that all values in a file match so the file can't be deleted (deletes usually fall back to min/max metrics evaluation).Implementing strict projection for all transforms where possible will help query efficiency, will make deletes faster, etc.