apache / iceberg Public

32 Pull requests merged by 16 people

• Hive: ORC vectorization fails when split offsets are considered during split generation

  #3748 merged Dec 16, 2021

• Core: Use consistent literal definition in TableProperties

  #3753 merged Dec 15, 2021

• Core: Deprecate default distribution constant in TableProperties

  #3755 merged Dec 15, 2021

• Spark: Remove no longer needed SparkTestTable

  #3754 merged Dec 15, 2021

• [SPARK] Backport cache.expiration-interval-ms config to Spark 3.1

  #3734 merged Dec 15, 2021

• [SPARK] Backport cache.expiration-interval-ms config to Spark 3.0

  #3735 merged Dec 15, 2021

• Spark 2.4: Add back rewrite data files action

  #3685 merged Dec 15, 2021

• build: apply publishing config after project evaluation

  #3743 merged Dec 15, 2021

• Aliyun: Add iceberg-aliyun to engines runtime

  #3725 merged Dec 15, 2021

• Core : Adding more information in CreateSnapshotEvent Summary

  #3696 merged Dec 14, 2021

• [SPARK] Backport SpakTestBaseWithCatalog to Spark 3.0 to simplify run…

  #3736 merged Dec 14, 2021

• Doc: Update Files metadata table

  #3422 merged Dec 14, 2021

• [SPARK] Backport SpakTestBaseWithCatalog to Spark 3.1 to simplify run…

  #3737 merged Dec 14, 2021

• Spark 3.0: Backport SparkReadConf.streamingSkipDeleteSnapshots and read from timestamp changes

  #3729 merged Dec 14, 2021

• Spark 3.1: Backport SparkReadConf.streamingSkipDeleteSnapshots and read from timestamp changes

  #3727 merged Dec 14, 2021

• Docs: update Nessie version to 0.17.0

  #3706 merged Dec 14, 2021

• Add FileIO implementation for Google Cloud Storage

  #3711 merged Dec 14, 2021

• Spark: Support vectorized reads with equality deletes

  #3557 merged Dec 14, 2021

• Add configurable cache expiry to caching catalog

  #3543 merged Dec 14, 2021

• Spec: Add snapshot tagging and branching

  #3425 merged Dec 14, 2021

• Spark: Refactor building write distribution and ordering

  #3720 merged Dec 13, 2021

• Core: Commit state should be unknown even when new location is not in history

  #3717 merged Dec 12, 2021

• Site Docs - Fix Flink batch job incorrectly labeled as streaming

  #3680 merged Dec 12, 2021

• Doc : Modify the attribute description

  #3694 merged Dec 12, 2021

• Spark-3.0: Add test cases for alter column with complex types

  #3693 merged Dec 12, 2021

• Spark-3.1: Add test cases for alter column with complex types

  #3692 merged Dec 12, 2021

• [Flink] Update flink-runtime jar naming with flink version at the end (like we do with Spark)

  #3713 merged Dec 12, 2021

• Spark: Move WAP conf to SparkWriteConf

  #3721 merged Dec 12, 2021

• Docs: add spark create clause CTAS/RTAS example

  #3712 merged Dec 10, 2021

• Flink: Add tests to check whether should remove meta columns in source reader

  #3477 merged Dec 10, 2021

• AWS: support S3FileIO alternative endpoint and credentials

  #3658 merged Dec 10, 2021

• Core: ensure Kryo serializability of MetricsConfig

  #3702 merged Dec 10, 2021

21 Pull requests opened by 17 people

• Spark: Add register Truncate UDF

  #3708 opened Dec 10, 2021

• Types literals all in one primitives pr

  #3714 opened Dec 10, 2021

• Types literals all in one generics pr

  #3715 opened Dec 10, 2021

• Spark: Support passing specific list of snapshot id for ExpireSnapshots procedure

  #3718 opened Dec 11, 2021

• Spark: support delete_reachable_files procedure

  #3719 opened Dec 11, 2021

• Spark: Use snapshot schema when reading snapshot

  #3722 opened Dec 12, 2021

• Fix Iceberg's parquet reader returning nulls incorrectly for parquet files written by writers that don't use list and element as names.

  #3723 opened Dec 13, 2021

• [Spark][Core]: Support RewriteDataFiles when Files are Completely Eliminated by Deletes

  #3724 opened Dec 13, 2021

• Spark: adds the ability to parse iceberg hexadecimal text

  #3728 opened Dec 13, 2021

• slove the problem that unable to delete table files by hivecatalog

  #3730 opened Dec 13, 2021

• Docs: Add stream-from-timestamp in spark-configuration.md

  #3732 opened Dec 13, 2021

• Core: Don't delete files during transaction commit when commit state is unknown

  #3733 opened Dec 13, 2021

• [Flink] fix data overflow while convert Integer to Long (#3738)

  #3740 opened Dec 14, 2021

• Core: Reading manifetsFiles parallel with ManifestGroup#planFiles

  #3742 opened Dec 14, 2021

• Push down partition filter to Spark when Importing File Based Tables

  #3745 opened Dec 14, 2021

• Spark: We should probably say why we cannot process the snapshot in SparkMicroBatchStream

  #3749 opened Dec 15, 2021

• Build: publish snapshot packages for all known versions

  #3751 opened Dec 15, 2021

• Core: add config for disabling FileIO hadoop configuration serialization

  #3752 opened Dec 15, 2021

• Spark: Fix UnresolvedException for some filters in rewrite_data_files procedure

  #3757 opened Dec 15, 2021

• Docs: add link to how to use iceberg with dremio on side nav bar

  #3758 opened Dec 16, 2021

• NPE in Parquet Writer Metrics when writing un-truncatable strings

  #3760 opened Dec 16, 2021

2 Issues closed by 2 people

• Broken dependency resolution in 0.13.0-SNAPSHOT and Spark 3.2

  #3620 closed Dec 15, 2021

• Flink cannot restart from checkpoint due to shifted index order in DataFile

  #3700 closed Dec 13, 2021

17 Issues opened by 17 people

• Add backwards compatibility read support for Parquet list

  #3759 opened Dec 16, 2021

• In-filter throws exception for rewrite_data_files procedure.

  #3756 opened Dec 15, 2021

• Some tests are leaking temp folders

  #3750 opened Dec 15, 2021

• throw exception : InvalidOperationException(message:The following columns have types incompatible with the existing columns in their respective positions : idd1) when add column

  #3747 opened Dec 15, 2021

• Update iceberg compatibility description

  #3746 opened Dec 15, 2021

• How to reduce rewrite manifest conflict after rewrite action?

  #3744 opened Dec 14, 2021

• [Improvement] ParallelIterable#hasNext submit reading ManifestFile task slowly with DataTableScan#planTasks

  #3741 opened Dec 14, 2021

• Problem of data overflow when convert Integer to Long

  #3738 opened Dec 14, 2021

• Cannot use RewriteDataFiles on Hadoop Tables

  #3731 opened Dec 13, 2021

• Slove the problem that failed to remove the data files when using HiveCatalog.dropTable

  #3726 opened Dec 13, 2021

• flink iceberg source reading array types fail with Cast Exception

  #3716 opened Dec 11, 2021

• CVE-2021-44228 - Log4j Remote Execution

  #3710 opened Dec 10, 2021

• Partition Metadata table breaks with a partition column named "partition"

  #3709 opened Dec 10, 2021

• Spark: I encountered the following error when writing to the partition table using truncate function

  #3707 opened Dec 10, 2021

• Explore spark struct streaming write iceberg and synchronize to hive Metastore

  #3705 opened Dec 10, 2021

• Inserts to the table partitioned by 'fixed' column fail

  #3704 opened Dec 10, 2021

• DeleteOrphanFiles or ExpireSnapshots outofmemory

  #3703 opened Dec 10, 2021

37 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Add FileIO, InputFile, and OutputFile abstract base classes

  #3691 commented on Dec 14, 2021 • 32 new comments

• [CORE] Specification for an HTTP REST catalog

  #3561 commented on Dec 15, 2021 • 30 new comments

• Spark: Implement copy-on-write DELETE

  #3661 commented on Dec 16, 2021 • 29 new comments

• Flink: FLIP-27 Iceberg source split

  #3501 commented on Dec 15, 2021 • 26 new comments

• fix data loss in multi-threading append with HadoopTable

  #3663 commented on Dec 14, 2021 • 15 new comments

• [SPARK] Make drop namespaces call respect CASCADE and IF EXISTS

  #3701 commented on Dec 13, 2021 • 14 new comments

• Aliyun: Add iceberg-aliyun document

  #3686 commented on Dec 15, 2021 • 12 new comments

• Flink: Fix ALTER for tables with primary key fields

  #3614 commented on Dec 13, 2021 • 11 new comments

• Add common view format spec

  #3188 commented on Dec 15, 2021 • 9 new comments

• [CORE] Add in a NOT_STARTS_WITH operator

  #2062 commented on Dec 16, 2021 • 7 new comments

• Core: Allow deleting and adding the same column

  #3632 commented on Dec 14, 2021 • 7 new comments

• Checkstyle:Add checkstyle rule to prevent using raw HashMap, HashSet, ArrayList

  #3689 commented on Dec 14, 2021 • 5 new comments

• AWS: Add unit tests for GlueCatalog's isValidIdentifier method

  #3698 commented on Dec 13, 2021 • 4 new comments

• Spark: We should probably say why we cannot process the snapshot in SparkMicroBatchStream

  #3699 commented on Dec 13, 2021 • 2 new comments

• Create a docker based demo environment for user testing and community onboarding

  #2676 commented on Dec 14, 2021 • 2 new comments

• Add documentation about Iceberg config naming convention

  #3678 commented on Dec 15, 2021 • 2 new comments

• Arrow: FIXED type support

  #3029 commented on Dec 10, 2021 • 2 new comments

• fix startsWith expression NullPointerException error caused by null v…

  #3645 commented on Dec 16, 2021 • 2 new comments

• AWS: Make the error message of GlueCatalog's isValidIdentifier more detailed

  #3668 commented on Dec 15, 2021 • 2 new comments

• HiveTableOperations: update HMS transient_lastDdlTime param on snapshot writes

  #3670 commented on Dec 10, 2021 • 2 new comments

• Support bucket table for Iceberg

  #430 commented on Dec 10, 2021 • 1 new comment

• Report I/O metrics to Spark

  #3062 commented on Dec 10, 2021 • 1 new comment

• Spark: Pushdown Filters / Improve Performance when Importing File Based Tables

  #3532 commented on Dec 14, 2021 • 1 new comment

• iceberg HiveCatalog insert exception of GSS initiate failed

  #3127 commented on Dec 16, 2021 • 1 new comment

• Core: add Jackson serialization util for Trino and Presto

  #3210 commented on Dec 16, 2021 • 1 new comment

• [Python] support iceberg transforms in python

  #3450 commented on Dec 12, 2021 • 1 new comment

• [DRAFT] potential direction for custom registration of procedures

  #3579 commented on Dec 13, 2021 • 1 new comment

• types and literals in one w/ generics

  #3601 commented on Dec 12, 2021 • 1 new comment

• retain flinkCheckpointInfo when rewrite datafiles to ensure restore flink job from checkpoint success

  #3647 commented on Dec 10, 2021 • 1 new comment

• Add watermark in flink table

  #3681 commented on Dec 12, 2021 • 1 new comment

• API: Add Immutables to FileScanTask's subclasses

  #3697 commented on Dec 10, 2021 • 1 new comment

• ParallelIterable may cause memory pressure for presto coordinator

  #3179 commented on Dec 15, 2021 • 0 new comments

• Core: add snapshot reference to table metadata

  #3104 commented on Dec 13, 2021 • 0 new comments

• Spark: Compact Medium Size Files (#460)

  #3292 commented on Dec 10, 2021 • 0 new comments

• spark: supplement spark-queries.md doc

  #3482 commented on Dec 10, 2021 • 0 new comments

• Skip processing snapshots of type Overwrite during readStream

  #3517 commented on Dec 15, 2021 • 0 new comments

• [WIP] Core : Dont consider groups for compaction if they have a single file…

  #3676 commented on Dec 11, 2021 • 0 new comments