Microservices Logging | ELK Stack | Elastic Search | Logstash | Kibana

Published on Oct 28, 2019 by Mukesh Kumar

A major challenge in a distributed system is to understand what is going on and even more importantly – what is going wrong, where and why.Logs management plays a critical role when we encounter an error in the application. If we do not manage logs properly then it will be almost impossible to find error in distributed(microservices) architecture.In distributed architecture it is important to collect and manage the logs in a centralized place which can be managed by IT team easily. Here, the ELK stack comes to the rescue. In this post, I will cover what ELK is and how to collect the logs from different microservices and push them to one centralized location.

What Is ELK?

ELK is an acronym for Elasticsearch, Logstash, and Kibana.The ELK stack is a complete log analytics solution, built on a combination of three open source tools—Elasticsearch, Logstash, and Kibana.ELK utilizes the open source stack of Elasticsearch for deep search and data analytics; Logstash for centralized logging management, which includes shipping and forwarding the logs from multiple servers, log enrichment, and parsing; and finally, Kibana for powerful and beautiful data visualizations. ELK stack is currently maintained and actively supported by the company called Elastic (formerly, Elasticsearch).

Elasticsearch

Elasticsearch is a distributed open source search engine based on Apache Lucene. It released under an Apache 2.0 license which means that it can be downloaded, used, and modified without any cost. Elasticsearch features are available through JSON over a RESTful API. The searching capabilities are backed by a schema-less Apache Lucene Engine, which allows it to dynamically index data without knowing the structure beforehand. Elasticsearch is able to achieve fast search responses because it uses indexing to search over the texts. Elasticsearch is used by many big companies, such as GitHub,linkedin SoundCloud, FourSquare, Netflix, and many others.

Logstash

Logstash is a data pipeline that helps to collect, parse, and analyze a large variety of structured and unstructured data and events generated across various systems in distribute architecture. It provides plugins to connect to various types of input sources and platforms, and is designed to efficiently process logs, events, and unstructured data sources for distribution into a variety of outputs with the use of its output plugins, namely file, stdout (as output on console running Logstash), or Elasticsearch.

Kibana

Kibana is an open data visualization platform that helps in visualizing any kind of structured and unstructured data stored in Elasticsearch indexes. Kibana uses the powerful search and indexing capabilities of Elasticsearch exposed through its RESTful API to display powerful graphics for the end users. From basic business intelligence to real-time debugging, Kibana plays its role through exposing data through beautiful histograms, geomaps, pie charts, graphs, tables, and so on.

Kibana makes it easy to understand large volumes of data. Its simple browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

Some of the key features of Kibana are as follows:

• It provides flexible analytics and a visualization platform for business intelligence.
• It provides real-time analysis, summarization, charting, and debugging capabilities.
• It provides an intuitive and user friendly interface, which is highly customizable through some drag and drop features and alignments as and when needed.

ELK Stack installation

A Java runtime is required to run ELK Stack. The latest version of Java is recommended for the installation. At the time of writing this book, the minimum requirement is Java 7. You can use the official Oracle distribution, or an open source distribution, such as OpenJDK.

You can verify the Java installation by running the following command.

If you have verified the Java installation in your system, we can proceed with the ELK installation.

Install the Elastic Stack products you want to use in the following order:

• Elasticsearch (install instructions)
• Kibana (install)
• Logstash (install)

Installing Elasticsearch

You can download the latest version of Elasticsearch from https://www.elastic.co/
downloads/elasticsearch. Dowmload elasticsearch and extract it in local drive.

Note:-For this tutorial we are using 6.5.4 version of elasticsearch, Kibana and logstash.

Running Elasticsearch

Now navigate to bin folder of installation and run the elasticsearch.bat using the command prompt.

Once elasticsearch sterted you can access it at localhost:9200.

Installing Kibana

1. You can download the latest version of Elasticsearch from here and unzip into any folder.
2. Open config/kibana.yml in an editor and set elasticsearch.url to point at your Elasticsearch instance. In our case as we will use the local instance just uncomment elasticsearch.url: "http://localhost:9200".
3. Now run bin\kibana.bat from command prompt.
4. Once started successfully, Kibana will start on default port 5601 and Kibana UI will be accessible at http://localhost:5601.

In this section we will discuss how to use ELK with default log format for a Spring Boot application.

For this guide, I've setup a demo Spring Boot application with logging enabled and with Logstash configuration that will send log entries to Elasticsearch. Demo application is a simple spring boot CRUD application available in github.

Application will store logs into a log file. Logstash will read and parse the log file and ship log entries to an Elasticsearch instance. Finally, we will use Kibana to search and analyze the logs.

Spring Boot ELK Stack Example

Here I am going to use employee management app for the demo of ELK stack. This is a simple spring boot service which will expose various endpoints for employee management like add employee, get employee, delete employee etc.

This project is available in my github repository. You can clone the project from below github URL.

https://github.com/javawebtutor/elk.git

Final Project Structure:

Configure Spring Boot's Log File :

In order to have Logstash ship log files to Elasticsearch, we need to configure Spring Boot to store log entries into a file. We will establish the following pipeline.

Spring Boot App → Log File → Logstash → Elasticsearch

There are other ways of accomplishing the same thing, such as configuring logback to use TCP appender to send logs to a remote Logstash instance via TCP, and many other configurations.

I prefer the file approach because it's simple and you can easily add it to existing systems and nothing will be broken if for some reason Logstash stops working or if Elasticsearch not reachable.

Let's configure Spring Boot's log file. The simplest way to do this is to configure log file name in application.properties. Open application.properties under resources folder and add below configuration entries.

Logstash Configuration:

We need to create a logstash configuration file so that it listen to the log file and push log messages to elastic search. Here is the logstash configuration used in the example, please change the log path as per your setup.

Typical Logstash config file consists of three main sections: input, filter and output. Each section contains plugins that do relevant part of the processing for example file input plugin that reads log events from a file or elasticsearch output plugin which sends log events to Elasticsearch.

Sample logstash config file

input {
  file {
    type => "java"
    path => "C:/spring//spring-boot-crud/application.log"
    ...
  }
}
 
filter {
    grok {
      match => ["message",...]
      add_tag => ["stacktrace"]
    }
  }
   
  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
  }
  ...
}
 
output {
   
  stdout {
    codec => rubydebug
  }
 
  # Sending properly parsed log events to elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Let us discuss all the section with details for more understanding of this configuration

Input Section:

Here's the sample input section code:

		
input {
  file {
    type => "java"
    path => "/path/to/application.log"
    codec => multiline {
      pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
      negate => "true"
      what => "previous"
    }
  }
}

1. We're using file plugin.
2. type is set to java and it's just an additional piece of metadata in case you will use multiple types of log files in the future.
3. path is the absolute path to the log file. It must be absolute - Logstash relies only on absolute path.
4. We're using multiline codec which means that multiple lines may correspond to a single log event,
5. In order to detect lines that should logically be grouped with a previous line we use a detection pattern as given below.
pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*" → Each new log event needs to start with date.
negate => "true" → if it doesn't start with a date ...
what => "previous" → ... then it should be grouped with a previous line.

File input plugin, as configured, will tail the log file (e.g. only read new entries at the end of the file). Therefore, when testing, in order for Logstash to read something you will need to generate new log entries.

Filter Section:

Filter section contains plugins that perform intermediary processing on an a log event. In our case, event will either be a single log line or multiline log event grouped according to the rules described above. In the filter section we will do several things:

• Tag a log event if it contains a stacktrace. This will be useful when searching for exceptions later on.
• Parse out (or grok, in logstash terminology) timestamp, log level, pid, thread, class name (logger actually) and log message.
• Specified timestamp field and format - Kibana will use that later for time based searches.

Filter section for Spring Boot's log format looks like this:

	
filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }

  #Grokking Spring Boot's default log format
  grok {
    match => [ "message", 
               "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- \[(?[A-Za-z0-9-]+)\] [A-Za-z0-9.]*\.(?[A-Za-z0-9#_]+)\s*:\s+(?.*)",
               "message",
               "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(?.*)"
             ]
  }

  #Parsing out timestamps which are in timestamp field thanks to previous grok section
  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
  }
}		  

• if [message] =~ "\tat" → If message contains tab character followed by at (this is ruby syntax) then...


• ... use the grok plugin to tag stacktraces:
  • match => ["message", "^(\tat)"] → when message matches beginning of the line followed by tab followed by at then...
  • add_tag => ["stacktrace"] → ... tag the event with stacktrace tag.
• Use the grok plugin for regular Spring Boot log message parsing:
  • First pattern extracts timestamp, level, pid, thread, class name (this is actually logger name) and the log message.
  • Unfortunately, some log messages don't have logger name that resembles a class name (for example, Tomcat logs) hence the second pattern that will skip the logger/class field and parse out timestamp, level, pid, thread and the log message.
• Use date plugin to parse and set the event date:
  match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ] → timestamp field (grokked earlier) contains the timestamp in the specified format

Output Section:

Output section contains output plugins that send event data to a particular destination. Outputs are the final stage in the event pipeline. We will be sending our log events to stdout (console output) to Elasticsearch.

Compared to filter section, output section is rather straightforward:

	
output {
  # Print each event to stdout, useful for debugging. Should be commented out in production.
  # Enabling 'rubydebug' codec on the stdout output will make logstash
  # pretty-print the entire event as something similar to a JSON representation.
  stdout {
    codec => rubydebug
  }

  # Sending properly parsed log events to elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"]
	index => "employee-transaction"
  }
}	 

• We are using multiple outputs: stdout and elasticsearch.
• stdout { ... } → stdout plugin prints log events to standard output (console).
• codec => rubydebug → Pretty print events using JSON-like format
• elasticsearch { ... } → elasticsearch plugin sends log events to Elasticsearch server.
• hosts => ["localhost:9200"] → Hostname where Elasticsearch is located - in our case, localhost running on port 9200.
• index => "employee-transaction" -> Index name in Elasticsearch.

Putting it all together

Finally, the three parts - input, filter and output - need to be copy pasted together and saved into logstash.conf config file. Place this logstash.conf file inside bin folder of logstash and run Logstash by using following command.

If everything went well, Logstash is now shipping log events to Elasticsearch.

Run the application and Verify Microservice Generated Logs

Run the application and create some data in DB by executing post request. Sample post request to create employee data is given below.

Now Go to the application root directory and verify that the log file i.e. application.log has been created and execute couple of request by using endpoints and verify that logs are getting added in the log file.

Configure Kibana:

One final thing before viewing the logs in Kibana, we need to configure the Index Patterns. Here we configured index name as employee-transaction. We can always change this index pattern in logstash side and configure in Kibana.

n Kibana the index pattern management page will look like below. With this configuration we are pointing Kibana to Elasticsearch index(s) of your choice. Logstash will create indices with the name employee-transaction. We can configuration index in Kibana console http://localhost:5601/app/kibana and going to Management link in left panel as shown below.

To create index click on Management -> Index Patterns -> Create Index Pattern and provide the index name as employee-transaction and then click on Next Step.

In next screen select "I don't want to use the Time Filter and click on "create index" button.

Verify ELK Stack:

Now when all components are up and running, let’s verify the whole ecosystem.Go to application and test the end points couple of times so that logs got generated and then go to Kibana console and see that logs are properly stacked in the Kibana with lots of extra feature like we can filter, see different graphs etc.

In Kibana dashboard click "Discover" tab. You can see index which we created and in my case it is employee-transaction.Click on the index and you can see application logs with lot of customization option as shown below.

Cool! You're now ready to take control of your logs using ELK stack and start customizing your log management configuration. You can download the sample application used from github: https://github.com/javawebtutor/elk.git. It's already configured to write logs in a file and has the Logstash config as described above although absolute paths need to be changed in logstash.conf.

I hope this article helps you in setting up your logging and monitoring environment effectively.Microservices is the way forward architecture. However, it does require you to do few things correctly to be able to make effective use of this architecture.Drop me your questions and suggestion in comments section.

Whats Next

Previous Next Article