data-pipeline

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

I found out we do not have a search component for our documents. I think we should add this component to our document for better search

Usage Scenario

No response

Related is

It would be nice if doit could use as many processes as there are CPUs instead of only a hardcoded number.

(1) Add docstrings to methods
(2) Covert .format() methods to f strings for readability
(3) Make sure we are using Python 3.8 throughout
(4) zip extract_all() in ingest_flights.py can be simplified with a Path parameter

Summary

There is a typo in the output here: https://github.com/whylabs/whylogs/blob/mainline/src/whylogs/cli/demo_cli.py#L113

Also we can update this to better align with our examples and suggested use, e.g. we can remove prompting for 'project' and 'pipeline' since these aren't required, and streamline the script to simplify the demo experience.

Environment

• klio* version(s):
  • klio-cli: v1.0.5

Description of the bug

When

Support copy into queries

Write unit test coverage for SafeDataset and SafeDataLoader, along with the functions in utils.py.

We should add a Github action that runs make generate and succeeds if there is no diff. It should run on each push to a PR.

We could do the same with proto files (generate them and make sure there's no diff).

Guide here:
https://dev.to/aws-builders/how-to-migrate-cdk-v1-to-cdk-v2-in-10-minuets-6i6

Follow up for #197 and #215

• Fix the ORDER BY issue so that we don't accidentally overwrite extra dispatch IDs https://github.com/AgnostiqHQ/covalent/pull/215/files#diff-d5a1b52ef809a88edbb01367cf4a161ddc0b9ce35c885981d8a8363cb0588949R151
• Figure out why middleware doesn't work https://github.com/AgnostiqHQ/covalent/pull/215/files#diff-d5a1b52ef809a88edbb01367cf4a161ddc0b9ce35c8859

Don't worry about event types, just have it forward messages over HTTP in the SSE protocol.

Read more: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events