ELK STACK INSTALLATION ON HDP 2.4 SANDBOX (CENTOS 6.7)


Our Goal:

                The installation of the Elasticsearch ELK Stack on CentOS 7—that is, Elasticsearch 2.2.x, Logstash 2.2.x, and Kibana 4.4.x. We will also show you how to configure it to gather and visualize the syslogs of your systems in a centralized location, using Filebeat 1.1.x. Logstash is an open source tool for collecting, parsing, and storing logs for future use. Kibana is a web interface that can be used to search and view the logs that Logstash has indexed. Both of these tools are based on Elasticsearch, which is used for storing logs.
It is possible to use Logstash to gather logs of all types, but we will limit the scope to gather syslog.

Our ELK stack setup has four main components:
  • Logstash: The server component of Logstash that processes incoming logs
  • Elasticsearch: Stores all of the logs
  • Kibana: Web interface for searching and visualizing logs, which will be proxied through Nginx
  • File Beat: Installed on client servers that will send their logs to Logstash, File Beat serves as a log shipping agent that utilizes the lumberjack networking protocol to communicate with Logstash
ELK Infrastructure
We will install the first three components on a single server, which we will refer to as our ELK Server. Filebeat will be installed on all of the client servers that we want to gather logs for, which we will refer to collectively as our Client Servers.

Prerequisites:
  • OS: CENTOS 6.7 AND UBUNTU 14.04
  • RAM: 8 GB
  • CPU: 2 (NOTE: Server Machine is Centos 6.7(Hdp 2.4 sandbox) and Client Machine is Ubuntu 14.04)
In Server Machine :
Install Java 8
We will install a recent version of Oracle Java 8 because that is what Elasticsearch recommends.Change to your home directory and download the Oracle Java 8 (Update 73, the latest at the time of this writing) JDK RPM with these commands:
  • cd ~
  • wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.rpm"

Then install the RPM with this yum command (if you downloaded a different release, substitute the filename here):
sudo yum -y localinstall jdk-8u73-linux-x64.rpm
Now Java should be installed at /usr/java/jdk1.8.0_73/jre/bin/java, and linked from /usr/bin/java.
You may delete the archive file that you downloaded earlier:
  • rm ~/jdk-8u*-linux-x64.rpm

Now that Java 8 is installed, let's install ElasticSearch.

Install Elasticsearch
Elasticsearch can be installed with a package manager by adding Elastic's package repository.
Run the following command to import the Elasticsearch public GPG key into rpm:
  • sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch
Create a new yum repository file for Elasticsearch. Note that this is a single command:
  • echo '[elasticsearch-2.x]
  • name=Elasticsearch repository for 2.x packages
  • baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
  • gpgcheck=1
  • gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
  • enabled=1
  • ' | sudo tee /etc/yum.repos.d/elasticsearch.repo
Install Elasticsearch with this command:
  • sudo yum -y install elasticsearch
Elasticsearch is now installed. Let's edit the configuration:
  • sudo vi /etc/elasticsearch/elasticsearch.yml
You will want to restrict outside access to your Elasticsearch instance (port 9200), so outsiders can't read your data or shutdown your Elasticsearch cluster through the HTTP API. Find the line that specifies network.host, uncomment it, and replace its value with "localhost" so it looks like this:
network.host: localhost
Save and exit elasticsearch.yml.
Now start Elasticsearch:
  • sudo service elasticsearch start
Then run the following command to start Elasticsearch automatically on boot up:
  • sudo systemctl enable elasticsearch
Now that Elasticsearch is up and running, let's install Kibana.

ERRORS For Elasticsearch Installation:
To start Elasticsearch : sudo /etc/init.d/elasticsearch start
To enable elasticsearch : virtualbox -> settings -> network -> port forwarding -> add a new port(name=elasticsearch,port=9200)

Install Kibana
The Kibana package shares the same GPG Key as Elasticsearch, and we already installed that public key.
Create and edit a new yum repository file for Kibana:
  • sudo vi /etc/yum.repos.d/kibana.repo
Add the following repository configuration:
  • [kibana-4.4]
  • name=Kibana repository for 4.4.x packages
  • baseurl=http://packages.elastic.co/kibana/4.4/centos
  • gpgcheck=1
  • gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
  • enabled=1
Save and exit.
Install Kibana with this command:
  • sudo yum -y install kibana
Open the Kibana configuration file for editing:
  • sudo vi /opt/kibana/config/kibana.yml
In the Kibana configuration file, find the line that specifies server.host, and replace the IP address ("0.0.0.0" by default) with "localhost":
                server.host: "localhost"
Save and exit. This setting makes it so Kibana will only be accessible to the localhost. This is fine because we will install an Nginx reverse proxy, on the same server, to allow external access.
Now start the Kibana service, and enable it:
  • sudo service kibana start
  • sudo chkconfig kibana on
Before we can use the Kibana web interface, we have to set up a reverse proxy. Let's do that now, with Nginx.

Install Nginx
Because we configured Kibana to listen on localhost, we must set up a reverse proxy to allow external access to it. We will use Nginx for this purpose.
Add the EPEL repository to yum:
  • sudo yum -y install epel-release
Now use yum to install Nginx and httpd-tools:
  • sudo yum -y install nginx httpd-tools
Use htpasswd to create an admin user, called "kibanaadmin" (you should use another name), that can access the Kibana web interface:
  • sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin
Enter a password at the prompt. Remember this login, as you will need it to access the Kibana web interface.
Now open the Nginx configuration file in your favorite editor. We will use vi:
  • sudo vi /etc/nginx/nginx.conf
Find the default server block (starts with server {), the last configuration block in the file, and delete it. When you are done, the last two lines in the file should look like this:

  include /etc/nginx/conf.d/*.conf;
}
Save and exit.
Now we will create an Nginx server block in a new file:
                  sudo vi /etc/nginx/conf.d/kibana.conf
Paste the following code block into the file. Be sure to update the server_name to match your server's name:
server {
  listen 80;
  server_name example.com;
  auth_basic "Restricted Access";
  auth_basic_user_file /etc/nginx/htpasswd.users;
  location / {
      proxy_pass http://localhost:5601;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;        
  }
}
Save and exit. This configures Nginx to direct your server's HTTP traffic to the Kibana application, which is listening on localhost:5601. Also, Nginx will use the htpasswd.users file, that we created earlier, and require basic authentication.
Now start and enable Nginx to put our changes into effect:
  • sudo  service nginx start
  • sudo systemctl enable nginx
ERROR For nginx installation:
Error shows:[root@sandbox nginx]# sudo service nginx start
Starting nginx: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] still could not bind()
                                                          [FAILED]
solution:
To a kill a specific port in Linux use below command
sudo fuser -k 80/tcp
---
server_name localhost;

Install Logstash:
The Logstash package shares the same GPG Key as Elasticsearch, and we already installed that public key, so let's create and edit a new Yum repository file for Logstash:
  • sudo vi /etc/yum.repos.d/logstash.repo
Add the following repository configuration:
  • [logstash-2.2]
  • name=logstash repository for 2.2 packages
  • baseurl=http://packages.elasticsearch.org/logstash/2.2/centos
  • gpgcheck=1
  • gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
  • enabled=1
Save and exit.
Install Logstash with this command:
  • sudo yum -y install logstash
Logstash is installed but it is not configured yet.

Error For logstash:
In the openssl conf section:
sudo vi /etc/pki/tls/openssl.cnf
Find the [ v3_ca ] section in the file, and add this line under it (substituting in the ELK Server's private IP address):
subjectAltName = IP: ELK_server_private_ip
The ELK_server_private_ip is ambariserver_ip(127.0.0.1)

Generate SSL Certificates:
Since we are going to use Filebeat to ship logs from our Client Servers to our ELK Server, we need to create an SSL certificate and key pair. The certificate is used by Filebeat to verify the identity of ELK Server. Create the directories that will store the certificate and private key with the following commands:
Now you have two options for generating your SSL certificates. If you have a DNS setup that will allow your client servers to resolve the IP address of the ELK Server, use Option 2. Otherwise, Option 1 will allow you to use IP addresses.
Option 1: IP Address
If you don't have a DNS setup—that would allow your servers, that you will gather logs from, to resolve the IP address of your ELK Server—you will have to add your ELK Server's private IP address to the subjectAltName (SAN) field of the SSL certificate that we are about to generate. To do so, open the OpenSSL configuration file:
  • sudo vi /etc/pki/tls/openssl.cnf
Find the [ v3_ca ] section in the file, and add this line under it (substituting in the ELK Server's private IP address):
  • subjectAltName = IP: ELK_server_private_ip
Save and exit.
Now generate the SSL certificate and private key in the appropriate locations (/etc/pki/tls/), with the following commands:
  • cd /etc/pki/tls
  • sudo openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
The logstash-forwarder.crt file will be copied to all of the servers that will send logs to Logstash but we will do that a little later. Let's complete our Logstash configuration. If you went with this option, skip option 2 and move on to Configure Logstash.

Configure Logstash
Logstash configuration files are in the JSON-format, and reside in /etc/logstash/conf.d. The configuration consists of three sections: inputs, filters, and outputs.
Let's create a configuration file called 02-beats-input.conf and set up our "filebeat" input:
  • sudo vi /etc/logstash/conf.d/02-beats-input.conf
Insert the following input configuration:
input {
beats {
  port => 5044
   ssl => true
   ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
   ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
 }
}
Save and quit. This specifies a beats input that will listen on tcp port 5044, and it will use the SSL certificate and private key that we created earlier.
Now let's create a configuration file called 10-syslog-filter.conf, where we will add a filter for syslog messages:
  • sudo vi /etc/logstash/conf.d/10-syslog-filter.conf
Insert the following syslog filter configuration:
filter {
if [type] == "syslog" {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    add_field => [ "received_at", "%{@timestamp}" ]
     add_field => [ "received_from", "%{host}" ]
   }
   syslog_pri { }
   date {
     match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
   }
 }
}
Save and quit. This filter looks for logs that are labeled as "syslog" type (by Filebeat), and it will try to use grok to parse incoming syslog logs to make it structured and query-able.
Lastly, we will create a configuration file called 30-elasticsearch-output.conf:
  • sudo vi /etc/logstash/conf.d/30-elasticsearch-output.conf
Insert the following output configuration:
output {
 elasticsearch {
   hosts => ["localhost:9200"]
   sniffing => true
   manage_template => false
   index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
   document_type => "%{[@metadata][type]}"
 }
}
Save and exit. This output basically configures Logstash to store the beats data in Elasticsearch which is running at localhost:9200, in an index named after the beat used (filebeat, in our case).
If you want to add filters for other applications that use the Filebeat input, be sure to name the files so they sort between the input and the output configuration (i.e. between 02- and 30-).
Test your Logstash configuration with this command:
  • sudo service logstash configtest
It should display Configuration OK if there are no syntax errors. Otherwise, try and read the error output to see what's wrong with your Logstash configuration.
Restart and enable Logstash to put our configuration changes into effect:
  • Sudo service logstash restart
  • sudo chkconfig logstash on
Next, we'll load the sample Kibana dashboards.

Load Kibana Dashboards:
Elastic provides several sample Kibana dashboards and Beats index patterns that can help you get started with Kibana. Although we won't use the dashboards in this tutorial, we'll load them anyway so we can use the Filebeat index pattern that it includes.
First, download the sample dashboards archive to your home directory:
  • cd ~
  • curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip
Install the unzip package with this command:
  • sudo yum -y install unzip
Next, extract the contents of the archive:
  • unzip beats-dashboards-*.zip
And load the sample dashboards, visualizations and Beats index patterns into Elasticsearch with these commands:
  • cd beats-dashboards-*
  • ./load.sh
These are the index patterns that we just loaded:
  • [packetbeat-]YYYY.MM.DD
  • [topbeat-]YYYY.MM.DD
  • [filebeat-]YYYY.MM.DD
  • [winlogbeat-]YYYY.MM.DD
When we start using Kibana, we will select the Filebeat index pattern as our default.

Error For Kibana Installation:( localhost:5601)
user = kibanaadmin
password = kibanaadmin

when run kibana through http:http://localhost:5601/status#/

Error:This version of Kibana requires Elasticsearch ^2.2.0 on all nodes. I found the following incompatible nodes in your cluster: Elasticsearch v1.2.1 @ inet[/10.0.2.15:9200] (10.0.2.15)
solution :  su -c 'yum update'
When I did the 'yum update' it uninstalled elasticsearch 2.2.1 and installed 2.3.1.1. I retried the Kibana UI but it still showed the error, but I figured that was probably because something needed to be restarted. Rather than fiddle with it I just restarted the machine, and now it's fine. Visiting the Kibana start page I do not see the error, and 'rpm -qa | grep elasticsearch' shows the expected version (2.3).

Load Filebeat Index Template in Elasticsearch
Because we are planning on using Filebeat to ship logs to Elasticsearch, we should load a Filebeat index template. The index template will configure Elasticsearch to analyze incoming Filebeat fields in an intelligent way.
First, download the Filebeat index template to your home directory:
  • cd ~
  • curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json
Then load the template with this command:
  • curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@filebeat-index-template.json
If the template loaded properly, you should see a message like this:
Output:
{
  "acknowledged" : true
}

Set Up Filebeat (Add Client Servers)
Copy SSL Certificate
On your ELK Server, copy the SSL certificate—created in the prerequisite tutorial—to your Client Server(substitute the client server's address, and your own login):
  • scp /etc/pki/tls/certs/logstash-forwarder.crt user@client_server_private_address:/tmp
NOTE:
ip configuration:
In server: scp /etc/pki/tls/certs/logstash-forwarder.crt rajesh@192.168.0.11:/tmp
In client: filebeat configuration:
$ sudo vi /etc/filebeat/filebeat.yml
### Logstash as output
 logstash:
   # The Logstash hosts
   # hosts: ["ELK_server_private_IP:5044"]
   hosts: ["127.0.0.1:5044"]

On client Machine:
open the terminal and go to root : $sudo suword:
Sudo dpkg -i filebeat_1.2.3_amd64.dep
On Client Server, create and edit Filebeat configuration file:
sudo vi /etc/filebeat/filebeat.yml
Near the top of the file, you will see the prospectors section, which is where you can define prospectors that specify which log files should be shipped and how they should be handled. Each prospector is indicated by the - character.
We'll modify the existing prospector to send syslog and auth.log to Logstash. Under paths, comment out the - /var/log/*.log file. This will prevent Filebeat from sending every .log in that directory to Logstash. Then add new entries for syslog and auth.log. It should look something like this when you're done:
...
      paths:
        - /var/log/auth.log
        - /var/log/syslog
#        - /var/log/*.log
...
            Input_type: syslog
Then find the line that specifies document_type:, uncomment it and change its value to "syslog". It should look like this after the modification:
...
      document_type: syslog
...
This specifies that the logs in this prospector are of type syslog
Next, under the output section, find the line that says elasticsearch:, which indicates the Elasticsearch output section (which we are not going to use). Delete or comment out the entire Elasticsearch output section (up to the line that says #logstash:).
Find the commented out Logstash output section, indicated by the line that says #logstash:, and uncomment it by deleting the preceding #. In this section, uncomment the hosts: ["localhost:5044"] line. Change localhost to the private IP address (or hostname, if you went with that option) of your ELK server:
### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["ELK_server_private_IP:5044"]
This configures Filebeat to connect to Logstash on your ELK Server at port 5044
And uncomment the file: and path: “/tmp/filebeat” present under ### File as Output
It looks like:
file:
   #path -----
     path: “/tmp/filebeat”
Save and quit.
Service filebeat restart
Service filebeat status
Check all the service are running
Then reboot your system
Then goto your browser and enter the url localhost:5601
And create a timestamp after that under the discover,it shows a graph that you want


Error :
If the index file is not load in your logstash,follow the bellow command
Load a index file in logstash:
sample index pattern in persent in the below link
https://www.elastic.co/guide/en/kibana/current/tutorial-load-dataset.html

Reference :
Youtube links: In youtube the video is split into two
1:https://www.youtube.com/watch?v=61bGPqVVLHk
Website link:

Comments

Post a Comment

Popular posts from this blog

Running Hive Queries Using Spark SQL

Image
Running Hive Queries Using Spark SQL Hive queries using Spark SQL. This instructional blog post explores how it can be done. Prerequisite: Hadoop Cluster configured in your system Hive installed and configured with Hadoop Spark installed on the top of Hadoop eco-system Some more configurations need to be done after the successful configuration of Hadoop, Hive, and Spark. Open your spark-env.sh file which is present in the $SPARK_HOME/conf directory and open the spark-env.sh file. Here, add the HIVE_HOME as shown below. 1 export HIVE_HOME=/path_to_hive_installed_directory Now copy the hive-site.xml file present in the $ HIVE_HOME/conf directory to the $ SPARK_HOME/conf directory. In hive-site.xml at   $ SPARK_HOME/conf directory Change hive.execution engine from tez to mr <property>      <name> hive.execution.engine </name>      <value> mr </value>  ...
Read more

Insert Postgresql database into Elasticsearch Using Logstash

Insert Postgresql database into Elasticsearch Using Logstash Ever want to search your database entities from Elasticsearch? Now you can use Logstash to do just that! In this blog we introduce the JDBC input , which has been created to import data from any database that supports the JDBC interface . Below, we show you few examples of using this plugin. Getting Started Installation bin / plugin install logstash - input - jdbc Driver Support Popular databases like Oracle, Postgresql, and MySQL have compatible JDBC drivers that can be used with this input. This plugin does not come packaged with any of these JDBC drivers out of the box, but is straightforward to download. You can then configure the plugin to use the desired jdbc driver library. The setting jdbc_driver_library and jdbc_driver_class are used to load the library path and the driver's class name. Lets get started with the examples! Example 1: Simple Postgres Input Here is an example of how you get st...
Read more

Rest API Java + Postgresql

Image
  REST API DOCUMENTATION Rest API REST (REpresentational State Transfer) is an architectural style, and an approach to communications that is often used in the development of Web services . The REST style emphasizes that interactions between clients and services is enhanced by having a limited number of operations (verbs). Flexibility is provided by assigning resources (nouns) their own unique Universal Resource Identifiers (URIs). Because each verb has a specific meaning (GET, POST, PUT and DELETE), REST avoids ambiguity. REST API IDE At first I choose Eclipse for developing RestApi sample application but there is lots of disadvantage are there while using eclipse.some sample things are below                    Disadvantage: You have to install lots of application to run restapi.I install Eclipse Mars.2 Release (4.5.2), JDk 1.7,tomcat 8.5.6 and jersey Framew...
Read more