[Bug] Cannot search http headers on trace page

[shelltea]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

1. Set SW_PLUGIN_HTTP_INCLUDE_HTTP_HEADERS=x-request-id,x-correlation-id in environment.
2. OAP-Server Set SW_SEARCHABLE_TAG_KEYS=http.method,http.headers,status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker in environment.
3. Send a request with header x-request-id and x-correlation-id like this:

curl "http://localhost:8080/a" \
     -H 'X-Request-ID: 64b95619-be46-4b4b-98f9-14fa24bc3452' \
     -H 'X-Correlation-ID: a3370c3d-53d5-4a3b-b6e2-411eebd42dec'

4. On trace page
   
   
5. Search tag like this:
   
   
   Got nothing ,or search like this:
   
   

What you expected to happen

Can search with http headers.

How to reproduce

Just like that.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

We never used a searchable key like this. Do you debug more about why this search can't be applied?

[shelltea]

@wu-sheng Hi, How to debug this problem?

[wu-sheng]

Through OAP source codes, of course. This is your private env. It is hard to tell the different or root cause just from UI screenshots.

[shelltea]

I think I found the problem.
The segment in elasticsearch:

{
	"_index": "sw_segment-20211216",
	"_type": "_doc",
	"_id": "0ce24b8c082e4f1bb38e4fc472889645.175.16396487055200000",
	"_version": 1,
	"_seq_no": 4525794,
	"_primary_term": 1,
	"found": true,
	"_source": {
		"start_time": 1639648705520,
		"trace_id": "0ce24b8c082e4f1bb38e4fc472889645.175.16396487055200001",
		"data_binary": "",
		"service_id": "a3MtdHJhY2UtaWQtd2Vi.1",
		"latency": 270,
		"time_bucket": 20211216175825,
		"endpoint_id": "a3MtdHJhY2UtaWQtd2Vi.1_R0VUOi9h",
		"is_error": 0,
		"service_instance_id": "a3MtdHJhY2UtaWQtd2Vi.1_a3MtdHJhY2UtaWQtd2ViLWxvY2Fs",
		"segment_id": "0ce24b8c082e4f1bb38e4fc472889645.175.16396487055200000",
		"tags": [
			"http.method=GET",
			"http.headers=x-request-id=[256ce597-1e0e-4086-a78a-deaf9ead8b45]\nx-correlation-id=[579b8397-17a4-4cf3-a4d5-477106f8f613]"
		]
	}
}

There is a \n between the two headers. When I search on trace page the condition is:

[
  {
    "key": "http.headers",
    "value": "x-request-id=[256ce597-1e0e-4086-a78a-deaf9ead8b45] x-correlation-id=[579b8397-17a4-4cf3-a4d5-477106f8f613]"
  }
]

So I changed the config toSW_PLUGIN_HTTP_INCLUDE_HTTP_HEADERS=x-request-id only include one header, And now I can search like this:

[wu-sheng]

I see. I think it is better to adjust plugin for header collecting? Which should generate one tag per header?

[wu-sheng]

I see. I think it is better to adjust plugin for header collecting? Which should generate one tag per header?

[llIlll]

I see. I think it is better to adjust plugin for header collecting? Which should generate one tag per header?

Will this have compatibility issues? it will be better to have both?