By Matej Elias
Last Validated on June 1 2021 · Originally Published on June 1, 2021 · Viewed 2.2k times

Introduction

Nowadays, even the smallest IT environment can generate a huge amounts of data. These endless logs include unexpected application behaviour, crashes, and reports together with user activities and hardware resources. For this data to be useful they must be processed, analyzed, and displayed in a way that is easy for humans to understand.

Methods by which the data are analyzed very much depend on the use-case, tools, and type of the data itself. However, visualizing collected data is consideredt the best practice, no matter what data you collect. Visualizing this kind of data is very helpful in detecting patterns and taking actions against undesirable behaviours.

Human brains are not very good at seeing patterns across vast amount of data. We are simply not designed to do so. On the other hand computers with the right software are excellent at pattern recognition and data analysis.

In this comparison, we will take a look at the two most popular open-source tools that helps users understand and visualize trends and patterns in vast amounts of log data generated by their IT environments.


Grafana

Grafana is a free open-source software for data visualization. It is a powerful and feature-rich tool for exploring, creating, and sharing dashboards with your colleagues. It allows you to query, visualize, alert on and understand your metrics no matter where they are stored.

Grafana

Grafana provides support for over 30 data sources with tools specially crafted for each one. You can choose from a vast amount of visualization options such as graphs, tables, and heatmaps and combine them in creating dashboards.


Kibana

You may have already heard about Kibana as it is used with the open-source ElasticSearch and it is the K in the world's most popular open-source log analysis platform - ELK Stack.

ELK is an acronym for three open-source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch.

Among the core features are data querying and analysis.

Kibana

With Kibana, you can analyze vast amounts of data coming from Elasticsearch with ease. You can choose from a rich selection of visualization options and combine those option to create dashboards which are accessible from anywhere.


Logs and Metrics

First, let's talk about the difference between metrics and logs. Metrics represent change over time and are used in monitoring hardware resources such as CPU utilization, RAM capacity, or disk usage. Metrics are collected in specific predefined intervals. For example, you can check every second if the CPU is overheating or how much space is allocated on the RAM.

On the other hand, logs are system-generated messages appearing randomly with no specific order as a reaction to an event. Every log comes with a different set of data. For example, when an application crashes a log message is generated and collected to be processed in the future. Content of this message can be the time when the crash occurred and the page/file where the application crashed. Later you can analyze those logs and see if there is any correlation between them.

The main use of Grafana is the analysis and visualization of metrics. Kibana is focused on logs and other log-dependent uses such as forensics and security. Both tools are expanding their scope. You can analyze logs using Grafana but it doesn't provide the same capabilities as Kibana and conversely.


Setup and Configuration

Installation and subsequent configuration are pretty easy and straightforward in both cases. Both applications provide a user-friendly installation setup and are available for most operating systems such as Windows, Linux, Mac, and even platforms like Docker, although Kibana supports more installation options.

Grafana is configured using .ini configuration file which is much easier than Kibanas YAML configuration files that are syntax-sensitive. You can also override Grafanas configuration options using environment variables. The documentation of Grafana data sources is very well-written and specific. Each data source comes with its own configuration manual.

Because Kibana only uses Elasticsearch data sources, an Elasticsearch instance of the same version as your Kibana is required. Dashboards can be configured using syntax-sensitive YAML language in the configuration file.


Data Sources

Grafana supports over 30 data sources such as Graphite, Prometheus, InfluxDB, MySQL, PostgreSQL, and Elasticsearch. Every supported data source has its own configuration tools specially crafted and comes whit extensive documentation. Every data source has its query editor that is customized for the features and capabilities that are included in that data source. Other data sources can be added via plugins.

On the other hand, Kibana is designed to work only with Elasticsearch and does not support any other type of data source. If you want to extrapolate data from other sources you need to ship it into an ELK Stack to apply Kibana to it.


Authentication

Grafana ships with built-in user control and authentication mechanisms. This allows you to control and restrict access to your dashboards including a Lightweight Directory Access Protocol (LDAP) or an external SQL server. Grafa users can create organizations and groups that allow them to team up on projects. Members of an organization can access the dashboards of their organization and every member of an organization has a specific role with a variety of permissions. Users are also able to set up unique API keys.

In contrast, Kibana dashboards are open to the public by default. That means dashboards don't have to exist within an organization. However, this can be changed using either the X-Pack or SearchGuard. Kibana provides many dashboard configuration options as well as default user authentication options. Also, there are many security plugins available to download such as X-Pack or SearchGuard.


Alerts

Grafana has a built-in alert system that allows users to have complete control over alert settings. Users can create custom alerts for any time series metric and can apply rules and conditions to them. This allows users to create very complex alert handlers and systems to handle very special cases such as failed connections or data availability. There are also many alert message destinations to choose from. You can send messages to email, Slack, PagerDuty, and custom webhooks.

Kibana doesn't have any alert system at all out of the box. To add the alert system to Kibana you can either use a hosted ELK Stack or use X-Pack. Another option for enabling alerts is through the Elasticsearch API and through functions called watchers. Watchers are functions that periodically run a query & perform a specific task based on the result. Whit that functionality you can create a query that checks the availability of a specific resource and sends a message if the resource is not available.


Querying

As was already mentioned, Grafana is mainly used for metrics analysis. That means Grafanas's user interface is better optimized for analyzing time-series data, making it best suited for monitoring things that change over time. It is not optimized for displaying logs and other kinds of data. Grafana provides fewer data querying and refining options when compared with Kibana.

However, Grafana uses Query Editor for querying. Every source of data has a different built-in Query Editor. That means the syntax varies according to the data source. Prometheus querying will be different than Graphite querying.

Kibana's most powerful feature is considered Querying and data exploration. Users have many querying options such as Lucene syntax, the Elasticsearch Query DSL, or the experimental Kuery. Lucene and Elasticsearch Query DSL are very powerful querying languages but they are not very intuitive and involve a certain learning curve. The learning curve is very steep, but when those languages are fully mastered, querying using them is very powerful and efficient.

The user interface comes with a search box for Elasticsearch queries and supports charts, graphs, and other visualizations generated by querying logs based on HTTP requests. Provided results can be saved and used later. Searched data can be widened using search parameters.


Dashboards and Visualizations

In the case of visualization capabilities, there is no clear winner as both tools are known for their powerful visualizations.

Grafana dashboards are very feature-rich. Every visualization in Grafana is called a panel. Using multiple panels you can create dashboards where each panel represents a different set of data. As mentioned earlier, you can assign an organization to those dashboards and restrict access to users outside of the organization.

Grafana supports a vast amount of visualization types such as gauge, heat map histograms, tables, charts, and many more. Moreover, new users can be pleased to hear that Grafana comes with numerous ready to use dashboards that were crafted for specific data sources. That means that starting with Grafana cannot be easier. Also, Grafana offers a great amount of flexibility in data formatting.

In the case of visualization capabilities, there is no clear winner as both tools are known for their powerful visualizations.

Grafana dashboards are very feature-rich. Every visualization in Grafana is called a panel. Using multiple panels you can create dashboards where each panel represents a different set of data. As mentioned earlier, you can assign an organization to those dashboards and restrict access to users outside of the organization.

Grafana supports a vast amount of visualization types such as gauge, heat map histograms, tables, charts, and many more. Moreover, new users can be pleased to hear that Grafana comes with numerous ready to use dashboards that were crafted for specific data sources. That means that starting with Grafana cannot be easier. Also, Grafana offers a great amount of flexibility in data formatting.

Kibana also offers a rich variety of visualization types that can be combined to create dashboards. Kibana dashboards are very versatile as data can be filtered on a whim, and dashboards can be edited and opened in full-page format. You can also use premade dashboards that are ready for specific data sets out of the box.

Both dashboards provide a great number of customization options for data visualization bot overall, Grafana has a wider variety of customization capabilities when compared to Kibana.


Community

Both Grafana and Kibana are very much alive and have a big community contributing to the project. When looking at both projects on GitHub we can see that Grafana has more than 14,000 code commits while Kibana has more than 17,000. Both projects are highly active, but when taking a closer look at the frequency of commits we can see that Kibana is in a slight lead.

Grafana commits over time (as of 5/28/2020)
Kibana commits over time (as of 5/28/2020)

In terms of popularity among the users, both tools are highly popular, although Grafana is in the slight lead, this can change over time.

Both tools are getting regular updates and developers are still working on bringing the best experience to the users.


Pros and Cons

Grafana

Pros:

  • Grafana supports a vast amount of community templates and plugins that are ready to use and can improve your overall experience.
  • Grafana provides support for over 30 data sources. other data sources can be added via plugins.
  • Grafana offers highly customizable dashboards with custom alerts and notifications.
  • Grafana has a rich built-in user authentication system and offers the creation of organizations.
  • Grafana provides a rich set of additional features such as data annotations or snapshots.

Cons:

  • Grafana has poor support for log analysis as its main focus is on metrics analysis.
  • Data collection and storage must be set up separately.

Kibana

Pros:

  • Kibana doesn't require additional coding or infrastructure.
  • Kibana offers the Ability to create custom ways to visualize data.
  • Kibana offers an ability to explore relations among data.
  • Kibana has a built-in anomaly detection system.
  • Kibana has a well-designed user interface.
  • Kibana provides data sharing and exporting abilities.

Cons:

  • Kibana has poor support for metrics analysis as its main focus is on log analysis and analysis of other log-dependent data.
  • Kibana affected by ElasticStack's limitations

Conclusion

In the core, both tools are very powerful and feature-rich. However, both are used for different data. Grafana is a great tool for metrics analysis such as hardware resource monitoring. Grafana offers a built-in user authentication system with the ability to create group dashboards and supports over 30 data sources. Kibana is excellent at log analysis. Kibana users can query and apply relevant filters to collected logs to get visualizations and display them in a custom way.

Grafana and Kibana are great choices for organizations of any size and scale. As both tools are used for different data, in tandem can create a powerful analysis platform.