How I developed a full-fledged Security Operations Centre using ELK Stack?
A packet matching the signature of malicious ransomware has been detected on the network. A red flag has been raised in real time on the dashboard of the security operation centre. The security team has located which user has received that packet and contacted them straight away and have advised them regarding the steps they need to take.
Do you realise what just happened in the above scenario, a problem that could have caused havoc has been brought under control due to continuous logging and advanced monitoring that was taking place in the security operation centre? So this gives us an idea about how important SOC is in a real-life scenario.
Now let us get into the nitty-gritty of how you can create your own security operations centre and help your organisation be ahead and secure.
I have grown up watching various sci-fi movies most of them based on cyber-sec where a criminal master mind tries to hack the organisation and good hackers try to stop them. They work from the security operation centres and detect the hack and then carry out a counter attack, where they locate the criminal hacker and that leads to his arrest.
To develop something even close to it is my childhood dream but it seemed like an impossible task till now. I got started working on this when my work colleague asked me to develop this and he guided me to look into ELK stack and OSquery
So, let me first explain it to you how the Security Operation Centre operates. The first step is to gather as many logs as possible, try to gather diverse logs i.e. networking logs, system logs, command line logs etc. The more logs you collect the better it is, remember you do need to parse them to make any sense out of them. The second step is to use those logs to extract data out of it and use it for your benefit by specifically monitoring certain parameters of those logs and then displaying it on a dashboard in real time.
- Logging ( OSquery )
So, the first step is to generate logs and store them properly so that you can use them to extract meaningful data out of them. To carry out this process we use OSquery, this tool has been made by devs at Facebook. So the reason we are using OSquery is that we can write our own configuration file and with the help of the OSquery daemon we begin to log data and store them in the system. These logs can contain a humongous amount of information, you can check that out here. You can obtain a hell lot number of logs depending on your requirements and then have it stored for future use.
You can download osquery and play around with it to understand what information it gives out and what is important to you. The default osquery configuration that I have been using is mentioned below you can change them according to your own needs. You can copy the configuration for yourself from here.
You can go ahead and carry out the whole installation process and have in-depth knowledge about OSquery here. Now that we are generating a huge amount of logs let’s find out a way to properly document them and display it on a dashboard so that we can get feedbacks and understand what is happening.
- Aggregating & Analysing Log ( ELK Stack )
We have obtained a huge number of logs but it is all junk if we can figure out what that means and for that, we need to search the logs for the details that we consider as red flags and display it to the security team. These red flags as soon as they are displayed to the security team they can then work on it and try to counter it.
The tools that I used to get our logs that are saved on the local machine to the dashboard which we can then monitor in real-time are following.
This tool is an extremely optimised and fast search engine which uses Apache Lucene in the background. It is the tool of choice for our project as the number of logs that we are generating is humongous and if we are not able to extract data out of it quickly and with extreme ease then the whole point of having these fancy logs become pointless. ElasticSearch needs data logs in the json format to carry out the efficient search and lucky for us in this case that OSquery saves their logs in the json format.
You can go through the details of setting up ElasticSearch here and read through the detailed documentation to have a better idea about it.
These are monitoring options that are provided by the ELK stack by default to enhance your data logging capabilities. Beats can be considered as a stack of tools that consists of Filebeat, Auditbeat, Packetbeat, Heartbeat etc.
Filebeat - It is a shipper for forwarding & centralising log data.Auditbeat - A shipper that is installed on servers to audit the activities of users and processes on your systems.Packetbeat - Real-time network packet analyser that can be used with ElasticSearch to provide an application monitoring and performance analytics system.Heartbeat - Daemon that is installed on a remote server to periodically check the status of your services and determine whether they are available
There are other tools that are shipped by elastic stack under beats which you can find here and depending on your needs can implement them as well.
Kibana is the last tool that we need from this suite of tools provided by ELK stack. It is used to help the sysadmins and security professionals visualise the data that they have been logging by running various processes. We can have a look at various other elastic stack features that we can use in sync with Kibana. The dashboard is really amazing and can be changed according to our liking by passing in the queries and the output that we need.
You can go through the documentation to properly configure Kibana and then you can boast about your very own security operation centre.
Security Operation Centre is a must for an organisation that keeps security as a priority. From this article we can understand that setting up these dashboards aren’t that hard all you need to do is make changes in the configuration files according to your need and you can setup your own SOC within an hour. The amount of control such SOC provides is truly amazing and you can make your organisation extremely secure by keeping tabs on the stuff that is happening on your network, which users are carrying out CPU/GPU intensive work etc.
P.S. Feel free to contact me if you need any help with setting this up for your own home or your organisation, let’s collaborate!
If you enjoyed it please do clap & let’s collaborate. Get, Set, Hack!
Telegram : https://t.me/aditya12anand
Twitter : twitter.com/aditya12anand
LinkedIn : linkedin.com/in/aditya12anand/
E-mail : [email protected]