[BEAM-11033] Updates Dataflow Metrics processor for portable job submission

[chamikaramj]

Currently, Dataflow metrics processor expects Dataflow internal step names generated for v1beta3 job description in metrics returned by Dataflow service.

But with portable job submission, Dataflow uses PTransform ID (in proto pipeline) as the internal step name. Hence metrics processor should be updated to handle this.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Dataflow	Flink	Samza	Spark	Twister2
Go		---		---		---
Java
Python				---		---
XLang		---		---		---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[chamikaramj]

R: @pabloem

[chamikaramj]

Refactored code to not to depend on an exception path. PTAL.

[codecov]

Codecov Report

Merging #13024 into master will increase coverage by 0.00%.
The diff coverage is 79.37%.

@@           Coverage Diff           @@
##           master   #13024   +/-   ##
=======================================
  Coverage   82.49%   82.49%           
=======================================
  Files         455      455           
  Lines       54867    54890   +23     
=======================================
+ Hits        45264    45283   +19     
- Misses       9603     9607    +4     

Impacted Files	Coverage Δ
...les/complete/juliaset/juliaset/juliaset_test_it.py	0.00% <0.00%> (ø)
sdks/python/apache_beam/utils/timestamp.py	95.43% <ø> (ø)
sdks/python/apache_beam/utils/windowed_value.py	92.54% <ø> (ø)
...dks/python/apache_beam/metrics/monitoring_infos.py	96.64% <50.00%> (ø)
sdks/python/apache_beam/metrics/cells.py	80.73% <53.84%> (ø)
...n/apache_beam/runners/dataflow/dataflow_metrics.py	75.94% <62.50%> (+1.62%)	⬆️
sdks/python/apache_beam/io/gcp/bigquery.py	79.40% <69.23%> (ø)
sdks/python/apache_beam/metrics/execution.py	87.40% <73.91%> (ø)
...s/python/apache_beam/runners/worker/log_handler.py	87.71% <83.33%> (ø)
...ks/python/apache_beam/runners/worker/sdk_worker.py	89.63% <86.11%> (+0.31%)	⬆️
... and 56 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52ef994...484b616. Read the comment docs.

[pabloem]

LGTM. I just have questions about how this works with the Dataflow API / UI / etc.

Does this mean that the Dataflow API reports metrics with portability-API step names? Is this for all portable jobs? Does this mean that the UI has made this change as well?

[chamikaramj]

With portable job submission, we directly convert Beam proto pipeline to internal Dataflow steps (instead of going through v1beta3 steps). Here we will be using transform IDs (from proto pipeline transform map) as the Dataflow step names. Hence this change to map such step names to user names expected by metrics.

Do you think we need additional changes to map such step names to UI for displaying metrics ?
cc: @ajamato @ananvay @robertwb

[pabloem]

I guess currently, the API works this way:

Dataflow service --(v1beta3 names)--> SDK

And the SDK translates v1b3 names to user names.

Is something like this happening for portable job submission?

Dataflow service --(v1beta3 names)--> jobservice ---(fnapi names) --> SDK 

and SDK translating fnapi names to user names?

[chamikaramj]

So for portable job submission "v1beta3 step names" do not exist. So we'll have:
Dataflow service --(fnapi names) --> jobservice --> SDK

At least looking at a simple wordcount job, seems like expected metrics are available in the UI.
I'll get this in. We can do additional fixes related to metrics if needed.