etl

Tell us about the problem you're trying to solve

If a source incorrectly declares its schema (e.g; it says the "ID" column is a number when it's really a string) then we only find out about that when the destination fails upon encountering one such record. This has two problems:

1. It incurs unnecessary cost on the destination system e.g: you sync 1 billion records just fine, then the 1-bill

Under the hood, Benthos csv input uses the standard encoding/csv packages's csv.Reader struct.

The current implementation of csv input doesn't allow setting the LazyQuotes field.

We have a use case where we need to set the LazyQuotes field in order to make things work correctly.

I am trying to implement retry logic for Aws Aurora Mysql.

In Linq2db there is retyrpolicy already implemented except there is no way to pass retrypolicy to DataContext.
So I have extended the DataContext as follows.

public class RetryingDataContext : DataContext
{
protected IRetryPolicy _retryPolicy;
public RetryingDataContext(IDataProvider dataProvider, string c

The functions in this file should be factored out to a separate utility lib as they are reused in bitcoin-etl https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/misc_utils.py

if they are not class methods then the method would be invoked for every test and a session would be created for each of those tests.

`class PySparkTest(unittest.TestCase):
@classmethod
def suppress_py4j_logging(cls):
logger = logging.getLogger('py4j')
logger.setLevel(logging.WARN)

@classmethod
def create_testing_pyspark_session(cls):
    return Sp

Sending a rest call to delete a job specification throws 404 where as grpc call works fine. Steps to reproduce

curl -X DELETE "http://localhost:9100/v1/project/my-project/namespace/kush/helloworld" -H  "accept: application/json"

Right now, these aren't caught until we try to gob-encode. Consider failing faster in type-checking to avoid too much confusion/loss when it works with local execution.

If i have an input yaml connecting to jdbc source like:

inputs:
  somedb:
    jdbc:
      connectionUrl: jdbc:jtds:redact
      user: someuser
      password: somepass

the log has

2020-12-08 17:01:28,076 [main] INFO  com.yotpo.metorikku.Job - these are the config inputs: Some(Map(.....somepass

ideally this should be printed as ****** in the log