in our company we need aggregate services statistics
for some reason we decide to use rsyslogd to send applications statistics (json format) to elasticsearch system directly (this tutorial).
but we faced this issue:
when send json statistics to local rsyslogd by logger command , everything was ok.
but
when application (java) send these statistics to rsyslogd (with backlog framework)
mmjsonparse error:
9448.407432204:main Q:Reg/w0 : Called action, logging to mmjsonparse
9448.407443744:main Q:Reg/w0 : Action 1 transitioned to state: itx
9448.407450424:main Q:Reg/w0 : entering actionCalldoAction(), state: itx, actionNbr 1
9448.407465385:main Q:Reg/w0 : mmjsonparse: no JSON cookie: '{"subject": "Report","report_no": 2411,"report_time_from": 1479309445405,"report_time_until": 1479309448406,"report_time_duration": 3001,"upload_lessThan1M_count": 0,"upload_lessThan1M_size": 0,"upload_btw1Mto2M_count": 0,"upload_btw1Mto2M_size": 0,"upload_btw2Mto5M_count": 0,"upload_btw2Mto5M_size": 0,"upload_btw5Mto10M_count": 0,"upload_btw5Mto10M_size": 0,"upload_btw10Mto20M_count": 0,"upload_btw10Mto20M_size": 0,"upload_btw20Mto50M_count": 0,"upload_btw20Mto50M_size": 0,"upload_btw50Mto100M_count": 0,"upload_btw50Mto100M_size": 0,"upload_moreThan100M_count": 0,"upload_moreThan100M_size": 0,"upload_total_size": 0,"upload_total_count": 0,"thumb_count": 0,"thumb_time": 0,"download_lessThan1M_count": 0,"download_lessThan1M_size": 0,"download_btw1Mto2M_count": 0,"download_btw1Mto2M_size": 0,"download_btw2Mto5M_count": 0,"download_btw2Mto5M_size": 0,"download_btw5Mto10M_count": 0,"download_btw5Mto10M_size": 0,"download_btw10Mto20M_count": 0,"download_btw10Mto20M_size": 0,"download_btw20Mto50M_count": 0,"download_btw20Mto50M_size": 0,"download_btw50Mto100M_count": 0,"download_btw50Mto100M_size": 0,"download_moreThan100M_count": 0,"download_moreThan100M_size": 0,"download_total_size": 0,"download_total_count": 0,"cache_served_count": 0,"cache_served_size": 0,"cache_new_count": 0,"cache_new_size": 0}'
9448.407510073:main Q:Reg/w0 : Action 1 transitioned to state: rdy
9448.407517838:main Q:Reg/w0 : PRIFILT 'local2.*'
9448.407527643:main Q:Reg/w0 : pmask: X X X X X X X X X X X X X X X X X X FF X X X X X X X
9448.407598847:main Q:Reg/w0 : PRIFILT condition result is 1
9448.407604631:main Q:Reg/w0 : ACTION 2 [omelasticsearch:action(type="omelasticsearch" ...)]
9448.407617909:main Q:Reg/w0 : executing action 2
9448.407622728:main Q:Reg/w0 : Called action, logging to omelasticsearch
9448.407631995:main Q:Reg/w0 : action 3 queue: qqueueAdd: entry added, size now log 1, phys 1 entries
software details: (rsyslog 8.4.2-1+deb8u2 , rsyslog-elasticsearch )
syslog-config:
#load needed modules
#load needed modules
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
module(load="mmjsonparse") #for parsing CEE-enhanced syslog messages
module(load="omelasticsearch") #for indexing to Elasticsearch
#try to parse structured logs
local2.* action(type="mmjsonparse")
#define a template to print field "foo"
template(name="justFoo" type="list") {
property(name="$!all-json")
}
#and now let's write the contents of field "foo" in a file
#action(type="omfile"
# template="justFoo"
# file="/tmp/foo")
local2.* action(type="omelasticsearch"
server="192.168.218.42"
serverport="9200"
template="justFoo"
searchIndex="stats"
searchType="stats"
bulkmode="on"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
