Logicalwebhost Cheatsheet

Linux & Open Source Cheatsheets & Howto's

Skip to: Content | Sidebar | Footer

Elasticsearch Logstash Kibana ELK server howto

If you want to search large volumes of network traffic, syslog, and other kinds of data and sort through and visualize them, ELK stack is a pretty good way to start. You can also add Graphana if you want to visualize stuff.

This howto is using Debian Stretch (9.x), though you can adapt this to whatever you have.

ELK setup on Debian Stretch (9)

You have to install Java first, then add repositories from Elasticsearch.co website. You need lots of memory and disk space to build this, I used 4GB RAM on a 64 bit system, but 1GB wouldn’t work, Java wouldn’t start.

wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/jdk-8u171-linux-x64.tar.gz
java -version
  java version "1.8.0_171"
  Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
  Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elk.list
apt-get update
apt-get install elasticsearch
vi /etc/elasticsearch/elasticsearch.yml
  network.host: localhost <-- change from an IP to this
systemctl enable elasticsearch
systemctrl start elasticsearch
curl -X GET http://localhost:9200
{
  "name" : "VYxIwOT",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "RGs9BYi-RZudJtV0htkRyA",
  "version" : {
    "number" : "5.6.9",
    "build_hash" : "877a590",
    "build_date" : "2018-04-12T16:25:14.838Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

That means elasticsearch is working, which means java is working. Now install logstash:

apt-get install logstash

Now you install kibana

apt-get install kibana
vi /etc/kibana/kibana
  server.host: "localhost" <-- change to your actual IP
systemctl restart kibana
systemctl enable kibana
  Synchronizing state of kibana.service with SysV service script with /lib/systemd/systemd-sysv-install.
  Executing: /lib/systemd/systemd-sysv-install enable kibana

Now install filebeat, the thing that sends information to your ELK box to look at.

apt-get install nginx
echo "admin:$(openssl passwd -apr1 YourStrongPassword)" | tee -a /etc/nginx/htpasswd.kibana
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/certs/nginx.crt
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/nginx.key -out /etc/ssl/private/nginx.crt
rm -f /etc/nginx/sites-enabled/default
vi /etc/nginx/sites-available/kibana
  server {
    listen 80 default_server;
    server_name _;
    return 301 https://$server_name$request_uri;
  }
 
  server {
    listen 443 default_server ssl http2;
 
    server_name _;
 
    ssl_certificate /etc/ssl/certs/nginx.crt;
    ssl_certificate_key /etc/ssl/private/nginx.key;
    ssl_session_cache shared:SSL:10m;
 
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.kibana;
 
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
  }
ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana
nginx -t
systemctl restart nginx
systemctl enable nginx
Now you should be able to view ELK stack in a browser like: 
http://your.server.ip.address
Now you can install grafana and filebeat: 
vi /etc/apt/sources.list
  deb https://packages.grafana.com/oss/deb stable main
curl https://packages.grafana.com/gpg.key | apt-key add -
apt-get update
apt install grafana
systemctl enable grafana-server.service
systemctl start grafana-server
netstat -plunt
...
tcp        0      0 127.0.0.1:3000          0.0.0.0:*               LISTEN      3021/grafana-server
cd /usr/share/elasticsearch/bin
./elasticsearch-plugin install ingest-geoip
apt install filebeat
vi /etc/filebeat/filebeat.yml
  uncomment port number
filebeat modules enable system
filebeat setup
/etc/init.d/filebeat start
ps aux | grep filebeat
root      4007  1.3  1.7 1618020 35156 ?       Ssl  21:33   0:08 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat