Skip to content

Importing logs via ElasticSearch interface

Jump to bottom
Max Landauer edited this page Apr 6, 2021 · 8 revisions

This wiki describes how to use the aelastic component to read logs from an ElasticSearch database rather than a file. Note that this wiki does not cover sending anomalies to an ElasticSearch database, which can simply be achieved by utilizing the KafkaEventHandler as an output module.

Setup

First, make sure that the aminer is installed. Instructions on how to install the aminer are provided in the Getting Started. Next, follow the installation guidelines of aminer-aelastic to install the interface component that queries your ElasticSearch database and provides the logs to the aminer. Also make sure that you have an ElasticSearch database containing log data up and running. In our case, a simple filebeat is used to write Apache access logs as well as syslogs into an index called filebeat.

Configuration

Open the configuration for the aelastic daemon (available in /etc/aminer/elasticsearch.conf) and set the correct parameters for the IP address of your ElasticSearch database as well as the index name. In the following example, the IP is set to localhost and the name of the index is aminertest:

[DEFAULT]
unixpath = /var/lib/aelastic/aminer.sock
host = http://localhost:9200
searchsize = 100
index = aminertest
savestate = False
output = False
timestamp = @timestamp

Moreover, the logs should be sorted when read from the database. Sorting is based on the timestamp, which is stored in the field @timestamp in ElasticSearch. As you can see in the config, this field is set by default; however, you may have to change it accordingly in case that your logstash and ElasticSearch setup produces different database entries.

Parsing

Speaking about database entries, let's have a look on the result of a sample query of size 2, which are shown in the following.

ubuntu@elk:~$ curl -XGET 'localhost:9200/filebeat-7.12.0-2021.03.24-000001/_search?q=*.*&size=2'
{"took":96,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":1.0,"hits":[{"_index":"filebeat-7.12.0-2021.03.24-000001","_type":"_doc","_id":"O8NSZHgBWTcyXtcRE-x7","_score":1.0,"_source":{"@timestamp":"2021-03-24T12:59:56.723Z","log":{"file":{"path":"/var/log/syslog"},"offset":68941},"message":"Mar 24 12:59:56 mail systemd[1]: Stopping Filebeat sends log files to Logstash or directly to Elasticsearch....","input":{"type":"log"},"agent":{"name":"mail.thermometer.com","type":"filebeat","version":"7.12.0","hostname":"mail.thermometer.com","ephemeral_id":"4402368c-9177-4adb-b42a-b810b56f0dfa","id":"83e6cc50-9746-4410-b200-b54070394c20"},"ecs":{"version":"1.8.0"},"host":{"containerized":false,"name":"mail.thermometer.com","ip":["192.168.66.57","fe80::f816:3eff:fef6:d13c"],"mac":["fa:16:3e:f6:d1:3c"],"hostname":"mail.thermometer.com","architecture":"x86_64","os":{"kernel":"4.9.0-13-amd64","codename":"stretch","type":"linux","platform":"debian","version":"9 (stretch)","family":"debian","name":"Debian GNU/Linux"},"id":"c3b9a0562e6c441c851be51235d46b6a"}}},{"_index":"filebeat-7.12.0-2021.03.24-000001","_type":"_doc","_id":"PMNSZHgBWTcyXtcRE-x7","_score":1.0,"_source":{"@timestamp":"2021-03-24T12:59:59.706Z","log":{"offset":69053,"file":{"path":"/var/log/syslog"}},"message":"Mar 24 12:59:56 mail filebeat[5540]: 2021-03-24T12:59:56.555Z#011INFO#011beater/filebeat.go:515#011Stopping filebeat","input":{"type":"log"},"ecs":{"version":"1.8.0"},"host":{"id":"c3b9a0562e6c441c851be51235d46b6a","containerized":false,"ip":["192.168.66.57","fe80::f816:3eff:fef6:d13c"],"mac":["fa:16:3e:f6:d1:3c"],"name":"mail.thermometer.com","hostname":"mail.thermometer.com","architecture":"x86_64","os":{"platform":"debian","version":"9 (stretch)","family":"debian","name":"Debian GNU/Linux","kernel":"4.9.0-13-amd64","codename":"stretch","type":"linux"}},"agent":{"hostname":"mail.thermometer.com","ephemeral_id":"4402368c-9177-4adb-b42a-b810b56f0dfa","id":"83e6cc50-9746-4410-b200-b54070394c20","name":"mail.thermometer.com","type":"filebeat","version":"7.12.0"}}}]}}

Quite a lot of fields! We need to define a parser for the aminer so that all fields are parsed correctly. Since the data from ElasticSearch is in json format, we will employ the JsonParsingModelElement. Note that we also import the ApacheAccessParsingModel (available in /etc/aminer/conf-available/ait-lds) to parse the message field in case that it is an Apache access log. We could do the same for syslog, however, the SyslogParsingModel does not cover logs from filebeat. You could extend the parsing model, however, for this simple demonstration we will use an AnyByteDataModelElement that just reads in any log as a string in case that it does not match the ApacheAccessParsingModel.

Another important thing to note is that the input log resource is set to the socket created by the aelastic daemon, which we set by the parameter unixpath in the elasticsearch.conf. Other than files, the socket must have the keyword unix in the beginning of the string. Also note that we added a ParserCount in the Analysis section so that we see how many logs were correctly parsed. Finally, a simple StreamPrinterEventHandler is used to print the ParserCount output as well as possibly unparsed logs to the console.

LearnMode: True # optional
AminerUser: 'aminer'  # optional default: aminer
AminerGroup: 'aminer' # optional default: aminer

LogResourceList:
        - 'unix:///var/lib/aelastic/aminer.sock'

Parser:
       - id: _indexField
         type: VariableByteDataModelElement
         name: 'index'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:/'

       - id: _typeField
         type: VariableByteDataModelElement
         name: 'type'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:/'

       - id: _idField
         type: VariableByteDataModelElement
         name: 'id'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:/'

       - id: _scoreField
         type: FixedDataModelElement
         name: 'score'
         args: 'null'

       - id: timestampField
         type: DateTimeModelElement
         name: 'timestamp'
         date_format: '%Y-%m-%dT%H:%M:%S.%fZ'

       - id: offsetField
         type: DecimalIntegerValueModelElement
         name: 'offset'

       - id: pathField
         type: VariableByteDataModelElement
         name: 'path'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:/'

       - id: syslogField
         type: AnyByteDataModelElement # SyslogParsingModel
         name: 'message'

       - id: apacheField
         type: ApacheAccessParsingModel
         name: 'apache'

       - id: messageField
         type: FirstMatchModelElement
         name: 'message'
         args:
           - apacheField
           - syslogField

       - id: inputTypeField
         type: VariableByteDataModelElement
         name: 'type'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:/'

       - id: ecsVersionField
         type: VariableByteDataModelElement
         name: 'ecs'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:/'

       - id: ipv4Field
         type: VariableByteDataModelElement
         name: 'ipv4'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:/'

       - id: ipv6Field
         type: VariableByteDataModelElement
         name: 'ipv6'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:/'

       - id: macField
         type: VariableByteDataModelElement
         name: 'mac'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: hostnameField
         type: VariableByteDataModelElement
         name: 'hostname'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: architectureField
         type: VariableByteDataModelElement
         name: 'architecture'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: platformField
         type: VariableByteDataModelElement
         name: 'platform'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: versionField
         type: VariableByteDataModelElement
         name: 'version'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: familyField
         type: VariableByteDataModelElement
         name: 'family'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: osNameField
         type: VariableByteDataModelElement
         name: 'name'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: kernelField
         type: VariableByteDataModelElement
         name: 'kernel'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: codenameField
         type: VariableByteDataModelElement
         name: 'codename'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: typeField
         type: VariableByteDataModelElement
         name: 'type'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: idField
         type: VariableByteDataModelElement
         name: 'id'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: nameField
         type: VariableByteDataModelElement
         name: 'name'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: containerizedField
         type: VariableByteDataModelElement
         name: 'containerized'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: agentNameField
         type: VariableByteDataModelElement
         name: 'name'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: agentTypeField
         type: VariableByteDataModelElement
         name: 'type'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: agentVersionField
         type: VariableByteDataModelElement
         name: 'version'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: agentHostnameField
         type: VariableByteDataModelElement
         name: 'hostname'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: ephemeralField
         type: VariableByteDataModelElement
         name: 'ephemeral_id'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: agentIdField
         type: VariableByteDataModelElement
         name: 'id'
         args: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789_-.()/:'

       - id: sortField
         type: DecimalIntegerValueModelElement
         name: 'sort'

       - id: parserField
         start: True
         type: FirstMatchModelElement
         name: 'parser'
         args:
           - json
           - emptyField

       - id: emptyField
         type: FixedWordlistDataModelElement
         name: 'empty'
         args:
           - ''

       - id: json
         type: JsonModelElement
         name: 'model'
         key_parser_dict:
            _index: _indexField
            _type: _typeField
            _id: _idField
            _score: _scoreField
            _source:
               "@timestamp": timestampField
               log:
                  offset: offsetField
                  file:
                     path: pathField
               message: messageField
               input:
                  type: inputTypeField
               ecs:
                  version: ecsVersionField
               host:
                  ip:
                    - ipv4Field
                    - ipv6Field
                  mac:
                    - macField
                  hostname: hostnameField
                  architecture: architectureField
                  os:
                     platform: platformField
                     version: versionField
                     family: familyField
                     name: osNameField
                     kernel: kernelField
                     codename: codenameField
                     type: typeField
                  id: idField
                  name: nameField
                  containerized: containerizedField
               agent:
                  name: agentNameField
                  type: agentTypeField
                  version: agentVersionField
                  hostname: agentHostnameField
                  ephemeral_id: ephemeralField
                  id: agentIdField
            sort:
              - sortField

Input:
        verbose: True
        multi_source: False
        timestamp_paths: "/parser/model/accesslog/time"
        json_format: True

Analysis:
        - type: ParserCount
          paths:
            - '/parser/model/_source/message/model/status_code'
            - '/parser/model/_source/message/model/host'
          labels:
            - 'apache'
            - 'syslog'
          report_interval: 5

EventHandlers:
        - id: "stpe"
          json: true # optional default: false
          type: "StreamPrinterEventHandler"

Store the config in /etc/aminer/config.yml.

Running

You are now ready to start the aelastic daemon. There are two possibilities to do that: Either start the script directly using the command

ubuntu@user-0:$ sudo -u aminer /usr/lib/aelastic/aelasticd.py

Alternatively, you can start the daemon as a service as follows.

ubuntu@user-0:$ sudo systemctl start aelasticd

Once the aelastic daemon is running, you can start the aminer as usual with the command

ubuntu@user-0:$ sudo aminer -C -c /etc/aminer/config.yml

After a few seconds, you should see one of two things: Either you receive a bunch of anomalies caused by unparsed logs. In this case, your parser did not fit the logs stored in the ElasticSearch database and you need to adapt the parser. Since verbose output is set to True, you should see at what point the parsing stopped; it should therefore be relatively straightforward to extend the parser accordingly. Obviously you want to get to the second possibility: The logs are correctly processed and parsed. In this case, you should see that the log counter of the ParserCount is continuously increasing. Fantastic! Also, you should get some of these events printed on the console:

2021-04-06 13:54:03 New path(es) detected
NewMatchPathDetector: "DefaultNewMatchPathDetector" (1 lines)
  {
  "AnalysisComponent": {
    "AnalysisComponentIdentifier": 1,
    "AnalysisComponentType": "NewMatchPathDetector",
    "AnalysisComponentName": "DefaultNewMatchPathDetector",
    "Message": "New path(es) detected",
    "PersistenceFileName": "Default",
    "AffectedLogAtomPaths": [
      "/parser/model/_source/message/model",
      "/parser/model/_source/message/model/client_ip/client_ip",
      "/parser/model/_source/message/model/sp1",
      "/parser/model/_source/message/model/client_id",
      "/parser/model/_source/message/model/sp2",
      "/parser/model/_source/message/model/user_id",
      "/parser/model/_source/message/model/sp3",
      "/parser/model/_source/message/model/time",
      "/parser/model/_source/message/model/sp4",
      "/parser/model/_source/message/model/tz",
      "/parser/model/_source/message/model/sp5",
      "/parser/model/_source/message/model/fm/request",
      "/parser/model/_source/message/model/sp8",
      "/parser/model/_source/message/model/status_code",
      "/parser/model/_source/message/model/sp9",
      "/parser/model/_source/message/model/content_size",
      "/parser/model/_source/message/model/combined",
      "/parser/model/_source/message/model/combined/combined",
      "/parser/model/_source/message/model/combined/combined/sp10",
      "/parser/model/_source/message/model/combined/combined/referer",
      "/parser/model/_source/message/model/combined/combined/sp11",
      "/parser/model/_source/message/model/combined/combined/user_agent",
      "/parser/model/_source/message/model/combined/combined/sp12",
      "/parser/model/_source/message/model/fm/request/method",
      "/parser/model/_source/message/model/fm/request/sp6",
      "/parser/model/_source/message/model/fm/request/request",
      "/parser/model/_source/message/model/fm/request/sp7",
      "/parser/model/_source/message/model/fm/request/version"
    ],
    "ParsedLogAtom": {
      "/parser/model": {
        "_index": "filebeat-7.12.0-2021.03.24-000001",
        "_type": "_doc",
        "_id": "hRH-kngBWTcyXtcRq2pH",
        "_score": null,
        "_source": {
          "@timestamp": "2021-04-02T14:31:05.819Z",
          "host": {
            "containerized": false,
            "ip": [
              "192.168.66.57",
              "fe80::f816:3eff:fef6:d13c"
            ],
            "mac": [
              "fa:16:3e:f6:d1:3c"
            ],
            "hostname": "mail.thermometer.com",
            "architecture": "x86_64",
            "os": {
              "kernel": "4.9.0-13-amd64",
              "codename": "stretch",
              "type": "linux",
              "platform": "debian",
              "version": "9 (stretch)",
              "family": "debian",
              "name": "Debian GNU/Linux"
            },
            "id": "c3b9a0562e6c441c851be51235d46b6a",
            "name": "mail.thermometer.com"
          },
          "agent": {
            "type": "filebeat",
            "version": "7.12.0",
            "hostname": "mail.thermometer.com",
            "ephemeral_id": "4402368c-9177-4adb-b42a-b810b56f0dfa",
            "id": "83e6cc50-9746-4410-b200-b54070394c20",
            "name": "mail.thermometer.com"
          },
          "log": {
            "offset": 733793,
            "file": {
              "path": "/var/log/apache2/mail.thermometer.com-access.log"
            }
          },
          "message": "192.168.66.69 - - [02/Apr/2021:14:31:01 +0000] \"GET /services/portal/ HTTP/1.1\" 200 23215 \"http://mail.thermometer.com/kronolith/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/77.0.3865.90 HeadlessChrome/77.0.3865.90 Safari/537.36\"",
          "input": {
            "type": "log"
          },
          "ecs": {
            "version": "1.8.0"
          }
        },
        "sort": [
          1617373865819
        ]
      },
      "/parser/model/index": "filebeat-7.12.0-2021.03.24-000001",
      "/parser/model/type": "_doc",
      "/parser/model/id": "hRH-kngBWTcyXtcRq2pH",
      "/parser/model/score": "null",
      "/parser/model/_source/timestamp": 1617373865.819,
      "/parser/model/_source/host/containerized": "false",
      "/parser/model/_source/host/ipv4": "fe80::f816:3eff:fef6:d13c",
      "/parser/model/_source/host/mac": "fa:16:3e:f6:d1:3c",
      "/parser/model/_source/host/hostname": "mail.thermometer.com",
      "/parser/model/_source/host/architecture": "x86_64",
      "/parser/model/_source/host/os/kernel": "4.9.0-13-amd64",
      "/parser/model/_source/host/os/codename": "stretch",
      "/parser/model/_source/host/os/type": "linux",
      "/parser/model/_source/host/os/platform": "debian",
      "/parser/model/_source/host/os/version": "9 (stretch)",
      "/parser/model/_source/host/os/family": "debian",
      "/parser/model/_source/host/os/name": "Debian GNU/Linux",
      "/parser/model/_source/host/id": "c3b9a0562e6c441c851be51235d46b6a",
      "/parser/model/_source/host/name": "mail.thermometer.com",
      "/parser/model/_source/agent/type": "filebeat",
      "/parser/model/_source/agent/version": "7.12.0",
      "/parser/model/_source/agent/hostname": "mail.thermometer.com",
      "/parser/model/_source/agent/ephemeral_id": "4402368c-9177-4adb-b42a-b810b56f0dfa",
      "/parser/model/_source/agent/id": "83e6cc50-9746-4410-b200-b54070394c20",
      "/parser/model/_source/agent/name": "mail.thermometer.com",
      "/parser/model/_source/log/offset": 733793,
      "/parser/model/_source/log/file/path": "/var/log/apache2/mail.thermometer.com-access.log",
      "/parser/model/_source/message/model": "192.168.66.69 - - [02/Apr/2021:14:31:01 +0000] \"GET /services/portal/ HTTP/1.1\" 200 23215 \"http://mail.thermometer.com/kronolith/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/77.0.3865.90 HeadlessChrome/77.0.3865.90 Safari/537.36\"",
      "/parser/model/_source/input/type": "log",
      "/parser/model/_source/ecs/ecs": "1.8.0",
      "/parser/model/sort": 1617373865819,
      "/parser/model/_source/message/model/client_ip/client_ip": 3232252485,
      "/parser/model/_source/message/model/sp1": " ",
      "/parser/model/_source/message/model/client_id": "-",
      "/parser/model/_source/message/model/sp2": " ",
      "/parser/model/_source/message/model/user_id": "-",
      "/parser/model/_source/message/model/sp3": " [",
      "/parser/model/_source/message/model/time": 1617373861,
      "/parser/model/_source/message/model/sp4": " +",
      "/parser/model/_source/message/model/tz": 0,
      "/parser/model/_source/message/model/sp5": "] \"",
      "/parser/model/_source/message/model/fm/request": "GET /services/portal/ HTTP/1.1",
      "/parser/model/_source/message/model/sp8": "\" ",
      "/parser/model/_source/message/model/status_code": 200,
      "/parser/model/_source/message/model/sp9": " ",
      "/parser/model/_source/message/model/content_size": 23215,
      "/parser/model/_source/message/model/combined": " \"http://mail.thermometer.com/kronolith/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/77.0.3865.90 HeadlessChrome/77.0.3865.90 Safari/537.36\"",
      "/parser/model/_source/message/model/combined/combined": " \"http://mail.thermometer.com/kronolith/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/77.0.3865.90 HeadlessChrome/77.0.3865.90 Safari/537.36\"",
      "/parser/model/_source/message/model/combined/combined/sp10": " \"",
      "/parser/model/_source/message/model/combined/combined/referer": "http://mail.thermometer.com/kronolith/",
      "/parser/model/_source/message/model/combined/combined/sp11": "\" \"",
      "/parser/model/_source/message/model/combined/combined/user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/77.0.3865.90 HeadlessChrome/77.0.3865.90 Safari/537.36",
      "/parser/model/_source/message/model/combined/combined/sp12": "\"",
      "/parser/model/_source/message/model/fm/request/method": 0,
      "/parser/model/_source/message/model/fm/request/sp6": " ",
      "/parser/model/_source/message/model/fm/request/request": "/services/portal/",
      "/parser/model/_source/message/model/fm/request/sp7": " ",
      "/parser/model/_source/message/model/fm/request/version": "HTTP/1.1"
    }
  },
  "LogData": {
    "RawLogData": [
      "\n{\"_index\": \"filebeat-7.12.0-2021.03.24-000001\", \"_type\": \"_doc\", \"_id\": \"hRH-kngBWTcyXtcRq2pH\", \"_score\": null, \"_source\": {\"@timestamp\": \"2021-04-02T14:31:05.819Z\", \"host\": {\"containerized\": false, \"ip\": [\"192.168.66.57\", \"fe80::f816:3eff:fef6:d13c\"], \"mac\": [\"fa:16:3e:f6:d1:3c\"], \"hostname\": \"mail.thermometer.com\", \"architecture\": \"x86_64\", \"os\": {\"kernel\": \"4.9.0-13-amd64\", \"codename\": \"stretch\", \"type\": \"linux\", \"platform\": \"debian\", \"version\": \"9 (stretch)\", \"family\": \"debian\", \"name\": \"Debian GNU/Linux\"}, \"id\": \"c3b9a0562e6c441c851be51235d46b6a\", \"name\": \"mail.thermometer.com\"}, \"agent\": {\"type\": \"filebeat\", \"version\": \"7.12.0\", \"hostname\": \"mail.thermometer.com\", \"ephemeral_id\": \"4402368c-9177-4adb-b42a-b810b56f0dfa\", \"id\": \"83e6cc50-9746-4410-b200-b54070394c20\", \"name\": \"mail.thermometer.com\"}, \"log\": {\"offset\": 733793, \"file\": {\"path\": \"/var/log/apache2/mail.thermometer.com-access.log\"}}, \"message\": \"192.168.66.69 - - [02/Apr/2021:14:31:01 +0000] \\\"GET /services/portal/ HTTP/1.1\\\" 200 23215 \\\"http://mail.thermometer.com/kronolith/\\\" \\\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/77.0.3865.90 HeadlessChrome/77.0.3865.90 Safari/537.36\\\"\", \"input\": {\"type\": \"log\"}, \"ecs\": {\"version\": \"1.8.0\"}}, \"sort\": [1617373865819]}"
    ],
    "Timestamps": [
      1617717243.1
    ],
    "DetectionTimestamp": 1617717243.1,
    "LogLinesCount": 1
  }
}

They just indicate that a new type of log event has been observed. Notice how even the message is dissected by the aminer parser - for example, the request of the access is parsed as "/parser/model/_source/message/model/fm/request/request": "/services/portal/". This is helpful to add more advanced detectors to the pipeline. To get some inspiration on how to add these detectors, check out the AMiner TryItOut that covers some basic detectors for various logs - including Apache access logs!

Clone this wiki locally