Report generator for Logstash parse failures

Since quite some I’m using Logstash (actually the whole ELK stack) for collecting, enriching and storing log events from various servers and applications.

While Logstash is great for this job, sometimes it cannot parse some log events because the events have an unknown formatting or my parsing rules don’t match well enough.

I used to manually search for such parse failures in the stored events from time to time. While this worked basically, it required me to remember “ah, maybe I should have a look for parse failures”. This happened not that often, so sometimes I had many parse failures for a long time although they were easy to fix.

Finally, I wrote a simple script to generate a simple report of parse failures happened in the last seven days. This script is executed as cronjob every seven days and will send the report via email to me.
Yay, now it’s really easy: I just need to read that tiny mail with parse failures and then decide whether I want to fix them or rather not.

In case anyone is interested, the script can be downloaded here: report_logstash_parse_failures.py

Additionally, for my convenience, I set up a Saved Search in Kibana:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[
  {
    "_id": "Parse-Failures",
    "_type": "search",
    "_source": {
      "title": "Parse Failures",
      "description": "",
      "hits": 0,
      "columns": [
        "tags",
        "logsource",
        "program",
        "message"
      ],
      "sort": [
        "@timestamp",
        "desc"
      ],
      "version": 1,
      "kibanaSavedObjectMeta": {
        "searchSourceJSON": "{\"index\":\"logstash-*\",\"filter\":[{\"$state\":{\"store\":\"appState\"},\"meta\":{\"alias\":\"Parse Failures\",\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":false,\"value\":\"{\\\"filtered\\\":{\\\"filter\\\":{\\\"terms\\\":{\\\"tags\\\":[\\\"_grokparsefailure\\\",\\\"_jsonparsefailure\\\",\\\"_log_level_normalization_failed\\\"]}}}}\"},\"query\":{\"filtered\":{\"filter\":{\"terms\":{\"tags\":[\"_grokparsefailure\",\"_jsonparsefailure\",\"_log_level_normalization_failed\"]}}}}}],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}}"
      }
    }
  }
]

The Saved Search can be imported directly into Kibana after downloading it from here: kibana_saved_search_parse_failures.json

Happy Logging!