Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
-
Updated
Sep 27, 2020 - Python
Describe the bug
A record can have a source record position. If the record is produced by the stream processing then it points to the record that was processed. Otherwise, the source record position is less than 0. This allows the see the causality.
Currently, there are some records that have a source record position but that are not produced by processing the record.
To Reproduce