Build a SIEM (Security Information and Event Management) system using ELK Stack with threat detection, correlation rules, and SOC dashboard.
Build a SIEM (Security Information and Event Management) system using ELK Stack with threat detection, correlation rules, and SOC dashboard. This comprehensive guide covers everything from design through implementation, testing, and deployment.
Collect from: Linux servers (Filebeat: /var/log/syslog, auth.log, nginx/apache access logs), Windows (Winlogbeat: Security, System, Application event logs), Network devices (syslog from Cisco/Palo Alto to Logstash), Cloud (AWS CloudTrail → S3 → Logstash), Applications (structured JSON logs). Centralize all to Logstash → parse → Elasticsearch. Index naming: logstash-{source}-{YYYY.MM.DD} for daily indices with lifecycle policy (hot→warm→cold→delete).
10 components required for this project.
| # | Component | Purpose | Qty |
|---|---|---|---|
| 1 | Elasticsearch 8.x | Log storage and full-text search | x1 |
| 2 | Logstash | Log ingestion, parsing, and enrichment | x1 |
| 3 | Kibana | Visualization and SIEM dashboard | x1 |
| 4 | Filebeat | Log shipping from servers | x1 |
| 5 | Winlogbeat | Windows Event Log collection | x1 |
| 6 | Elastic SIEM (free tier) | Built-in threat detection rules | x1 |
| 7 | MaxMind GeoIP database | IP geolocation enrichment | x1 |
| 8 | Sigma rules | Vendor-agnostic detection rules | x1 |
| 9 | MITRE ATT&CK Navigator | Coverage visualization | x1 |
| 10 | Wazuh (alternative) | Open-source SIEM with agent | x1 |
Follow these 3 steps carefully.
Collect from: Linux servers (Filebeat: /var/log/syslog, auth.log, nginx/apache access logs), Windows (Winlogbeat: Security, System, Application event logs), Network devices (syslog from Cisco/Palo Alto to Logstash), Cloud (AWS CloudTrail → S3 → Logstash), Applications (structured JSON logs). Centralize all to Logstash → parse → Elasticsearch. Index naming: logstash-{source}-{YYYY.MM.DD} for daily indices with lifecycle policy (hot→warm→cold→delete).
Logstash pipeline: input → filter → output. grok plugin: parse unstructured text using regex patterns. Grok pattern for nginx: %{IPORHOST:client_ip} - %{USER:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:version}" %{NUMBER:status} %{NUMBER:bytes}. mutate: add/rename/remove fields. geoip: add lat/lon for client IP. useragent: parse browser/OS from User-Agent. date: parse timestamp to @timestamp. Output: elasticsearch://es-host:9200, index: logstash-nginx-%{+YYYY.MM.DD}.
Convert Sigma rules to Elasticsearch queries for threat detection. Detection examples: brute force (>10 failed auth in 5 minutes from same IP → alert), privilege escalation (sudo or su command executed → alert), lateral movement (new SMB connection between internal hosts → alert), data exfiltration (unusual outbound data volume → alert), malware indicators (known bad IP/domain in DNS logs → critical alert). Use Elastic SIEM's built-in ML jobs for anomaly detection: unusual login time, unusual process for user.
Core code for logstash.conf:
# Logstash Pipeline Configuration input { beats { port => 5044 } syslog { port => 514; type => "syslog" } } filter { if [type] == "nginx_access" { grok { match => { "message" => '%{IPORHOST:client_ip} - %{USER:ident} \\[%{HTTPDATE:timestamp}\\] "%{WORD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}" %{NUMBER:status:integer} %{NUMBER:bytes:integer}' } } geoip { source => "client_ip"; target => "geoip" } useragent { source => "user_agent"; target => "ua" } mutate { add_field => { "environment" => "production" } convert => { "status" => "integer"; "bytes" => "integer" } } # Flag suspicious: 4xx/5xx status if [status] >= 400 { mutate { add_tag => ["error_response"] } } } if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGBASE}" } } # Detect brute force if [program] == "sshd" and "Failed password" in [message] { mutate { add_tag => ["ssh_brute_force_attempt"] } } } } output { elasticsearch { hosts => ["elasticsearch:9200"] index => "catb-logs-%{type}-%{+YYYY.MM.dd}" } if "ssh_brute_force_attempt" in [tags] { http { url => "http://alertmanager:9093/api/v1/alerts" http_method => "post" format => "json" } } }Test Log Management and SIEM System by verifying each subsystem individually before full integration.
Verify power voltages, check ground connections, use serial monitor for debug.
An interactive simulator will be available here — simulate circuits and run code in-browser without hardware.