Understanding Splunk: A Powerful Tool for Data Analysis and Security

7 Oct 2024

In this blog, we'll provide a clear introduction to Splunk: What it is, how it works, its most common use cases, and why it's so popular. By keeping things straightforward, our aim is to offer value to both technical and non-technical readers.

What is Splunk?

This question has been asked many times and it’s impressive how difficult it is to get a decent answer. You can spend hours on Splunk’s website going through their blogs and articles and not find a satisfying answer. Unfortunately, that is the case with most technology vendors and their websites. They tend to flood you with vague marketing phrases that don’t offer any specifics.

Feel free to go through the Splunk article below “What does Splunk do?“ and let me know if this gives you any useful information as to what Splunk is and what it can do:
https://www.splunk.com/en_us/blog/learn/what-splunk-does.html

It is often referred to as a SIEM tool (Security Information and Event Management) or nowadays as an Observability tool, however this definition as accurate as it is, does not do it justice.

This blog is going to be my own take on answering those questions. I will avoid being vague but will also use more common day-to-day vocabulary to convey the message. Once you are done with this article, let me know how did I do?

Here we go:

The first thing to understand about Splunk is not an application that fulfils a single purpose. Think of Splunk not as a single application with a limited use case, but as a framework—a collection of tools that can be applied in various ways to meet different needs. Much like Excel, Splunk gives you a flexible platform to process and analyse data, but the way you use it is entirely up to you.

Let's use two popular Microsoft applications as an example: PowerPoint and Excel.

PowerPoint has a designated purpose. It lets you create presentations and there isn’t much else that it’s good for. Excel on the other hand has a far more open-ended purpose. It provides you with a spreadsheet that you can populate with raw data and tools/features like adding formulas, pivoting your data, visualizing it, and many more. What you use it for and how you use it is up to you. You can analyse your financial data, keep a task list, manage project burndowns, keep a skills matrix of all your project resources, etc.

Splunk is more like Excel and in fact in many ways it could be considered like the next step. It’s a platform combined of a collection of tools and features that enables you to analyse your data and draw valuable conclusions.

How does Splunk work?

Splunk ingests massive amounts of machine data (logs, metrics, and traces), indexes it, and allows users to search and visualize the information. These three capabilities—data ingestion, indexing, and analysis—form the backbone of the platform.

This is an over-simplification and it might sounds very basic but trust me, these are all you need in your toolbox to gather valuable information and insight of your business. Data generated across your systems reflects the behaviour and performance of the business. Visualising complex data into comprehensive diagrams will lead to drawing accurate conclusions and those will lead to successful decision making.

Imagine you are given access to new prototype version of Excel. This version has a new feature that magically populates your spreadsheet with any real world value that you want.

If you are responsible for managing a manufacturing plant, imagine your cells are populated with your production line data: output per machine, current output compared to weekly average, power consumption, buffer queue size, time spent waiting for components, etc.

…or if you are IT systems admin, imaging having a spreadsheet with all your resource utilization data, free disk capacity, number of critical services not running, number of hosts not responding…

…or if you are a high level manager, imagine having a spreadsheet with todays income, costs, number of positive/negative tweets about your company/product, how much time your app takes to load on clients phones/computers, how many projects or tasks are overdue…

Now for all these cases, imagine that this spreadsheet updates itself with the latest data in real-time, along with all your diagrams, pivots, statistics.

How much more valuable information would that provide you?
How much more visibility would you have over the business performance?
How much more transparent would your operations become?
How much easier would it become for you to make the correct decisions and do your job well?

This is what Splunk enables.

What is Splunk used for?

Now that we have a basic understanding of what Splunk does from the functional perspective we can focus on its common use-cases.

As mentioned, the applications of Splunk can be vast, however there are a few most common practices. Based on the number of projects that my team and myself have completed we can see a clear majority of implementations focus on one of these two topics:

Infrastructure Monitoring (IM)

Cybersecurity

Infrastructure Monitoring with Splunk

IM use-cases (often referred to as Observability) focus on utilising Splunk as an analytics tool to: monitor and report on the state and health of the IT environment, identify issues, and take action to reduce downtime. Its goal is to answer questions such as:

How is our environment doing today?

Are there any signs that there might be a problem?

Are all critical systems up and responsive?

What is our performance looking like right now?

Having a good degree of visibility across your entire system enables quick identification of problems and swift response. A mature enough setup will even allow you to pre-empt issues by identifying problems before they take effect. This will result in shorter outages and the reduction of outage occurrences.

Splunk as a Cybersecurity tool

For Cybersecurity implementations Splunk can use it’s data analytics capabilities to find security-related incidents in the data. This can be as simple as looking for individual events but also combinations of events which (when put together) build a security use-case classified as a possible Indicator of Compromise (IoC). Some examples of those use cases would be:

Brute force attempt – multiple failed logon attempts executed in a short period

Network Exposure – changes in firewall configurations, opening the internal network to external access

Dispersed remote connections – remote connections to the corporate network (VPN) by the same users from 2 or more geographically dispersed locations

Locking down your systems and your network is a critical practice that must be followed to reduce vulnerability however, you won’t be certain if your environment is secure until you also monitor it in real time and quickly respond to IoCs.

There are many more use cases. Some of them will be less common and some will related to the two use cases mentioned above, either as a part of them or derived from them as a separate practice. Here are a few examples of them:

PCI compliance monitoring

Business Intelligence

Business Process Mining

Production site monitoring

Business Operations monitoring

IoT monitoring

Application Performance monitoring

Real User monitoring (RUM)

Synthetic monitoring

Financial monitoring and analysis

To summarise this chapter the possibilities are quite vast, however in all cases they can be boiled down to delivering visibility over complex systems.

Is this something that you have available in your company?
Do you have good visibility over your systems?
Are you aware of how well your digital product is performing today?

Is Splunk popular? Why is Splunk so popular?

Obviously the chapter’s title already answers its own question however I will still emphasize on it.

Splunk Inc. has been around for over 20 years and since its first release, the software has been continuously improved with search performance boosts, new features and quality of life improvements. This has ultimately put Splunk on the leaderboard of the SIEM solutions currently available on the market.

The SIEM market is a relatively new one (new-ish) and although mature, it still has a long way to go. There are multiple alternative solutions out there, each one with its pros and cons. No one solution is perfect or even close to perfect. However, the commonly agreed conclusion is that Splunk is one of the best solutions out there. It clearly stands out when it comes to factors such as: ease of deployment, ease of maintenance, usability and Time to Value. The global research and advisory group Gartner keeps a close eye on the SIEM market and for many consecutive years they have been placing Splunk as a visionary market leader on the magic quadrants.

Y23Q2 Magic Quadrant for Security Information and Event Management

You might think “Sounds great! But what’s the catch?”, and you would be right to ask that question as there is one major catch. Splunk is a costly investment. To re-iterate: all SIEM solutions are costly however Splunk the top-shelf product. I will not go into the details of that with price examples and comparisons. Just be warned, if you are considering implementing Splunk you will need to set a decent budget aside to cover the following:

License costs
The original pricing model was based on the volume of data ingested however recently an alternative was introduced where your bill is based on the allocation of resources like vCPUs. This can offer flexibility that some clients could take advantage of however don’t expect that to significantly reduce the licensing costs without impacting the performance of your solution. Splunk is designed to scale and it’s not uncommon for clients to ingest TBs of data every day. Searching through such volumes will require sufficient resources. Sure you can save money by getting a car equipped with a small moped engine, but how useful is that car going to be?
Hardware costs
Regardless of whether you are planning to deploy Splunk on-prem or in the cloud, you must provide it with sufficient resources. Depending on their role in the cluster, Splunk requires hosts to have 32-64 vCPUs, 12-64 GB of RAM and a performant storage solution. The number of such hosts will depend on the ingestion volume, number of users and how heavily the platform is utilised. Be prepared to have at least 6 of such hosts. Smaller deployments can be seen but they are not production-worthy. On the other hand it is not uncommon to see larger deployments having 50+ hosts.
Skilled resources
Splunk is a vast and complex solution. You can’t expect your current resources to “play around with it over the weekend and figure out how things work”. The successful deployment, implementation and maintenance will depend on trained and experienced consultants who can guide the company to success. Here you have the flexibility of either hiring contractors, professional service consultants from Splunk and partner companies, or using Splunk Education Programs to train your internal resources. Regardless of your choice, there are costs involved.

Alternative solutions might entice with lower license cost, hardware requirements and cheaper or even free training, however all factors need to be taken into account.

Will they provide equal or better performance and functionality?

Are they as easy to deploy and maintain?

Do they scale equally well?

…And one of the most important questions that no one will ever provide you with an answer: What are the limitations?

There are also alternatives that reach outside of the list of Splunk’s direct competitors. Some companies chose the direction of building their own solution, not entirely from scratch but by putting smaller applications together into a stack. This can be composed of either open-source apps as well as a mix of open-source and paid apps, where each one is only responsible for a particular part of the overall functionality. One app would serve as a data storage solution with the possibility to fetch and search that data while another could be responsible for the visualisation of the aggregated results and yet another might be responsible for transforming the data. This approach of using a stack in oppose to using a single (monolithic) application will also have its advantages and disadvantages. On the upside, it can offer significant savings by utilising open-source solutions. It also prevents you from being locked into a single vendor and inheriting all the issues that come with it, such as compatibility issues, pricing increases or product discontinuations. On the flip side, consider that open-source based stacks are generally less stable, rely on community contributions for product stability and improvements and might not be considered enterprise-level products. In addition you will require a greater number of engineers, each responsible for a segment of the stack, who will troubleshoot and maintain it. Sure, you gain more control over the direction of development however you require more resources and now product maintenance becomes your responsibility. With monolithic solution you pay a premium so you don’t have to worry about any of that.

There is no wrong chose here and the right solution depends on the requirements and priorities.
Based on my personal experience I can see a clear pattern where companies who prioritize cost saving and have a significant amount of technically capable resources will lean towards the option of using a stack solution. This would apply to medium-sized tech development companies, startups and enterprises in the area of technology (either hardware or software). The other group would be the medium to large-scale production and service delivery enterprises. Those large players in the banking, pharmaceutical and manufacturing industries tend to have strong demands for a functional, unified product and have the financial capabilities to justify such demands. They tend to have less flexibility to invest in a long-term project of developing a stack solution and require a stable and performant solution that can be delivered in a short time. They are less likely to be discouraged by the price tag as long as the quality of the product can justify it.

Conclusion

In conclusion, Splunk is much more than a simple tool—it’s a powerful framework that can be adapted to a wide range of use cases. Whether it's infrastructure monitoring, cybersecurity, or business process optimisation, Splunk offers the visibility and insights needed to make data-driven decisions quickly. With its ability to scale and integrate with complex environments, Splunk remains a popular choice for enterprises looking for a reliable and versatile platform. While the costs may be significant, the benefits in terms of real-time analysis, automation, and operational transparency often outweigh the investment.

Interested in learning how Splunk can optimize your business operations or improve your cybersecurity? Our team of Splunk Consultants has extensive experience in deploying and managing Splunk implementations across a variety of industries. Contact us to see how we can help you leverage the full potential of this powerful platform.