data-pipeline

Please translate the FAQ into English
the FAQ issue are as follows, apache/incubator-seatunnel#267

It would be nice if doit could use as many processes as there are CPUs instead of only a hardcoded number.

We're using marshmallow to parse whylogs config from YAML

However, Pydantic is much more powerful as it allows users to set config via various mechanims, from YAML, JSON to Environment settings.

We should consider moving to pydantic

Environment

• klio* version(s):
  • klio-cli: v1.0.5

Description of the bug

When

Write unit test coverage for SafeDataset and SafeDataLoader, along with the functions in utils.py.

Support copy into queries

The following packages must be updated in package.json:

• @aws-cdk/aws-dynamodb
• @aws-cdk/aws-events
• @aws-cdk/aws-events-targets
• @aws-cdk/aws-iam
• @aws-cdk/aws-lambda
• @aws-cdk/aws-lambda-event-sources
• @aws-cdk/core

The latest published version on NPM is 1.109.0 - all of these packages should be updated to at least 1.100.0 if possible.

We should add a Github action that runs make generate and succeeds if there is no diff. It should run on each push to a PR.

We could do the same with proto files (generate them and make sure there's no diff).

Don't worry about event types, just have it forward messages over HTTP in the SSE protocol.

Read more: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events