ShardingSphere storage configuration, e2e test and doc for SkyWalking

[wu-sheng]

@tristaZero

I created this to track the thing you are working one.

To All
We will add ShardingSphere proxy configuration file with documentation to guide users to use ShardingSphere + MySQL to support larger scale system monitoring.

[tristaZero]

@wu-sheng Hi, thanks for this issue which any updates from me would be related to.

[tristaZero]

@wu-sheng Hi, sir. I am considering using ShardingProxy to be the backend storage for sw. If i add a configuration, something like my.cnf for MySQL, do you think i should put it into sw project? If so, where is a better place?

[wu-sheng]

I think the configuration should be placed in install/shardingsphere/proxy/my.cnf. At the same time, the following changes are expected

1. readme should be updated
2. storage doc should be updated.
3. e2e test should be changed. A tip, based on this PR https://github.com/apache/skywalking/pull/4335/files#diff-5c4acdc03cc5bf967f23c6c3564a8cb7, the mysql download path is not right. We are trying to fix in PR Provide profile analyze query #4335, you should notice at yours too.

[tristaZero]

Hi @wu-sheng
Not only my.cnf, a sharding-algorithm.jar is necessary to be put into install/shardingsphere/proxy/ to handle sharding tables and dbs.

If user plan to adopt sharding proxy as backend server, he has to download sharding proxy and copy my.cnf, and sharding-algorithm.jar from install/shardingsphere/proxyto specified dir.

Do you think the above solution is feasible? And then how to specify my workload? Here is a task list from my understanding,

1. Provide my.cnf, a sharding-algorithm.jar and put them into install/shardingsphere/proxy/
   2.Update readme
   3.Update storage doc
   4.Create e2e test referring to test/e2e/mysql

If i came to wrong conclusion, please be clear what i should do next. Thanks in advance.

[wu-sheng]

sharding-algorithm.jar needs to be taken care of differently. It is a binary distribution, so it can't be hosted in any repo.

[wu-sheng]

If this runs into the release process, it seems you need another repo? If that, I would like to ask you to propose this in the mail list including purpose, scope, repo name etc.

[tristaZero]

sharding-algorithm.jar needs to be taken care of differently. It is a binary distribution, so it can't be hosted in any repo.

@wu-sheng Agree with you, and it is also what i am confused and have no idea to deal with it. I can provide a sharding-algorithm.java, but user needs to complie it to generate sharding-algorithm.jar.

[wu-sheng]

There are some options

1. As you said, you could provide the source codes hosted in SkyWalking main repo, then, it will be released as a separate jar in 7.0.0. You could update the document to point to the jar, even it is not available today.
2. Open a new repo to host it and ask for a release process for it. As you haven't been a committer/PMC, you need one to help you with this.

Which one do you want?

[tristaZero]

@wu-sheng Here is another news,
ShardingSphere is planning to do some optimization for sharding algorithm. And the main idea is to provide user some default sharding algorithms, therefore there is no need of sharding-algorithm.jar, and user just chooses one of existing algorithms and configure it in my.cnf.

I am not sure the concrete date for its completion, but don't worry, once it finishes, i will continue to update this issue here.
I would like to hear you suggestion.

[tristaZero]

There are some options

1. As you said, you could provide the source codes hosted in SkyWalking main repo, then, it will be released as a separate jar in 7.0.0. You could update the document to point to the jar, even it is not available today.

2. Open a new repo to host it and ask for a release process for it. As you haven't been a committer/PMC, you need one to help you with this.

Which one do you want?

First one sounds easy to make progress.

[wu-sheng]

Then the expected module will be oap-server/server-storage-plugin/storage-shardingsphere-algorithms. WDYT?

[wu-sheng]

ShardingSphere is planning to do some optimization for sharding algorithm. And the main idea is to provide user some default sharding algorithms, therefore there is no need of sharding-algorithm.jar, and user just chooses one of existing algorithms and configure it in my.cnf.

It is OK for you about when that comes, you could remove the codes and module from SW.