Microservice logging

Distributed Logging

As we have seen in previous articles, one of the key characteristics when building microservice-based applications is its distributed nature. Because services are distributed across many different processes running in separate containers, it is significantly more difficult to troubleshoot applications using the same methods we used with monolithic applications.

One of the most important troubleshooting resources is the application log. In a monolithic world, the log provides a chronological sequence of event that the troubleshooting engineer can step through to review application log events. But what happens when your application has been decomposed into many different services often running replica copies? Where do you begin? With microservices, we must approach logging differently.

Externalize and Centralize

As each microservice runs in a separate container, we need to externalize the storage of each service's log data. We do this for two reasons. First, container file systems are ephemeral and do not survive a restart or container crash. By externalizing the log data outside of the container, we can retain the log information beyond the life of any container. It is a generally accepted best practice to choose a common location outside of the application for persisting log data. We do this to avoid losing data in the event of a large scale application failure. By externalizing the data to a single static location, we have centralized all of the application's logging data which significantly simplifies the processing of the log data.

Logging Data

With microservice architectures, each service should write all its log data as raw text files encoded as a JSON log message. JSON provides a lightweight, universally readable, and writeable format that simplifies the production and consumption of logging events.

Every log event should include its timestamp, its service type, log-level, and a correlation id. The correlation id is populated by the first service in any chain of services or the API Gateway if one is used. It is passed the chain of participating service to provide a simple key with which to trace a request's log events across multiple services.

Once the JSON data has been generated and written to the local filesystem it gets picked up by a Log Collector.

LOG Collectors

The first stage in a distributed logging pipeline is the Log Collector. A log collector is a daemon process running on the same container as the service and is responsible for parsing the services log files and transporting the logging data to a backend system. Two of the most popular log collectors are Logstash and Fluentd. Both are open-source and run on Linux and Windows platforms.

Index the Log data

The second stage of a distributed logging pipeline indexes the incoming data to make it searchable. Searchability is commonly accomplished by inserting and indexing each field of the log event data into a database record. By indexing each field of the log record, we can perform complex queries against the data.

One of the most popular tools for this is Elastisearch. Elastisearch provides an open-source, indexed, queryable NoSQL datastore for storing and querying logging events. It is built on top of the Apache Lucene search engine and exposes a RESTful API to simplify query and integration into third-party applications.

Visualizing the Log Data

The last step in the logging pipeline is to visualize the log data. Our brains are optimized for seeing patterns in images so it is often advantageous to convert our log data into meaningful images that convey complex log data into simple images. To accomplish this task, we employ a visualization tool. Two of the most popular open-source visualization tools used are Kibana and Elastisearch. Kibana is an Elastisearch plugin that provides visualization of Elastisearch data, while Grafana is a general purpose dashboard and graphing tool. Both tools are excellent choices for visualizing log data.

Summary

With microservice-based applications, it is essential to externalize and centralize log data from individual services to make the data searchable. Once the log data has been made searchable, the application logs can be queried and visualized to monitor and troubleshoot an application.

Coming Up

Now that we have unified our logging, we take a look at how to unify the configuration of our services. In the next article, we will take a look at handling Microservice Configuration.