etl

Tell us about the problem you're trying to solve

As follow up to airbytehq/airbyte#4721, do proper yaml parsing/extract to be used in https://github.com/airbytehq/airbyte/blob/7decb12818ad2cc0c7b3039c3d5e620236250edd/airbyte-workers/src/main/resources/dbt_transformation_entrypoint.sh#L7

Describe the solution you’d like

we should write (a python?) script that uses

Issue from the Dagster Slack

[dagster_shell] defer environ copy to solid run time

This issue was generated from the slack conversation at: https://dagster.slack.com/archives/C01U954MEER/p1623181007055200?thread_ts=1623181007.055200&cid=

During migration of old sync code to async discovered that we don't have async versions of some API.

问题描述：关于工程中某个工作流节点单独执行不报错，保存工作流时提示：NotClassDefFoundError：Cloud not initialize class dispatch.Http$

问题原因：linkis-publish微服务关于netty-3.6.2.Final.jar升级包缺失

问题解决：将升级包上传重启linkis-publish微服务即可。

The functions in this file should be factored out to a separate utility lib as they are reused in bitcoin-etl https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/misc_utils.py

if they are not class methods then the method would be invoked for every test and a session would be created for each of those tests.

`class PySparkTest(unittest.TestCase):
@classmethod
def suppress_py4j_logging(cls):
logger = logging.getLogger('py4j')
logger.setLevel(logging.WARN)

@classmethod
def create_testing_pyspark_session(cls):
    return Sp

Right now, these aren't caught until we try to gob-encode. Consider failing faster in type-checking to avoid too much confusion/loss when it works with local execution.