data-pipeline

Hi, everyone, at present we have added many new plugins to the dev branch. We plan to release a version in March. Community contributors are welcome to test it. If you are interested in this, please leave a message in the issue (if you are interested plugin), I will assign it to you, please leave a message in this issue when you are finished, (if you have additional information, you can submit an

It would be nice if doit could use as many processes as there are CPUs instead of only a hardcoded number.

(1) Add docstrings to methods
(2) Covert .format() methods to f strings for readability
(3) Make sure we are using Python 3.8 throughout
(4) zip extract_all() in ingest_flights.py can be simplified with a Path parameter

We're using marshmallow to parse whylogs config from YAML

However, Pydantic is much more powerful as it allows users to set config via various mechanims, from YAML, JSON to Environment settings.

We should consider moving to pydantic

Environment

• klio* version(s):
  • klio-cli: v1.0.5

Description of the bug

When

Support copy into queries

Write unit test coverage for SafeDataset and SafeDataLoader, along with the functions in utils.py.

We should add a Github action that runs make generate and succeeds if there is no diff. It should run on each push to a PR.

We could do the same with proto files (generate them and make sure there's no diff).

The following packages must be updated in package.json:

• @aws-cdk/aws-dynamodb
• @aws-cdk/aws-events
• @aws-cdk/aws-events-targets
• @aws-cdk/aws-iam
• @aws-cdk/aws-lambda
• @aws-cdk/aws-lambda-event-sources
• @aws-cdk/core

The latest published version on NPM is 1.109.0 - all of these packages should be updated to at least 1.100.0 if possible.

Don't worry about event types, just have it forward messages over HTTP in the SSE protocol.

Read more: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

@scottwn commented on Tue Dec 14 2021

• Docstrings style is inconsistent in the codebase. We need a second pass on the docstrings to confirm that they are consistent with the style specified in CONTRIBUTING

• Addition of LocalExecuter