BASIC ELASTICSEARCH-KIBANA SETUP

BASIC ELASTICSEARCH-KIBANA SETUP

Prerequisite:

•  HDP 2.4 or 2.5 

Elasticsearch Setup:

*****************

Download Elasticsearch:

----------------------------

We use curl to download Elasticsearch from sandbox.(Elasticsearch version 2.4.0)

$ cd ~

$ curl -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.0/elasticsearch-2.4.0.tar.gz

Install Elasticsearch:

--------------------------

Next we need to extract Elasticsearch to the /opt directory, which is where we'll run it.

$ mv elasticsearch-2.4.0.tar.gz /opt/

$ cd /opt

$ sudo tar xvfz elasticsearch-2.4.0.tar.gz

Configure Elasticsearch:

------------------------------

We need to make a couple of changes to the Elasticsearch configuration file /opt/elasticsearch-2.4.0/config/elasticsearch.yml.

$ cd config

$ vi elasticsearch.yml

We need to set the cluster.name setting to "elasticsearch".

cluster.name: elasticsearch

We need to set the network.host setting to our sandbox hostname or ip.

network.host: 0.0.0.0

Make sure you have removed the # character at the start of the line for these two settings. Once you have completed these two changes, save the file:

Press the esc key

!wq

Create Elasticsearch user:

*********************

We are going to create an elastic user to run the application.

$ sudo useradd elastic -d /home/elastic

Change Ownership of Elasticsearch directories

We are going to change the ownership of the elastic directories to the elastic user:

$ sudo chown -R elastic:elastic /opt/elasticsearch-2.4.0

Start Elasticsearch

We want to run Elasticsearch as the elastic user so first we'll switch to that user.

$ sudo su - elastic

We want to run Elasticsearch as the elastic user so first we'll switch to that user.

$ cd /opt/elasticsearch-2.4.0

$ bin/elasticsearch

Note:

How to add Port in Docker HDP 2.5 follow the below link,

http://rajeshaj.blogspot.in/2017/11/add-ports-to-hdp-25-virtualbox-sandbox.html

Kibana Setup:

************

Download Kibana:

-----------------------

We use curl to download Kibana from sandbox.(Kibana version 4.6.1)

$ cd ~

$ curl -O https://download.elastic.co/kibana/kibana/kibana-4.6.1-linux-x86_64.tar.gz

Install Kibana:

-------------------

We will be running Kibana out of the /opt directory. So extract the archive there:

$ mv kibana-4.6.1-linux-x86_64.tar.gz /opt/

$ cd /opt

$ sudo tar xvfz kibana-4.6.1-linux-x86_64.tar.gz

We will be using the elastic user,we need to change ownership of the Kibana files to the elastic user:

$ sudo chown -R elastic:elastic /opt/kibana-4.6.1-linux-x86_64

Configure Kibana:

------------------------

Before making any configuration changes, switch over to the elastic user:

$ sudo su - elastic

The Kibana configuration file is kibana.yml and it's located in the config directory. We need to edit this file:

$ cd /opt/kibana-4.6.1-linux-x86_64

$ vi config/kibana.yml

Kibana defaults to port 5601, but we want to set it explicitly. This port should not conflict with anything on the sandbox.

Uncomment this line:

#server.port: 5601

It should look like this:

server.port: 5601

We want to explicitly tell Kibana to listen to the host ip address:

Uncomment this line:

#server.host: "0.0.0.0"

Change it to this:

server.host: 0.0.0.0

We also want to explicitly set the Elasticsearch host.

Uncomment this line:

#elasticsearch.url: "http://localhost:9200"

Change it to this:

elasticsearch.url: "http://50.232.31.41:9200"

Save the file

Press Esc

!wq

Start Kibana

Now we can start Kibana. The archive file does not provide a service script, so we'll run it by hand.

$ bin/kibana

##############################################################################

ADDITIONAL INFORMATION:

*************************

CONTENTS:

**********

1. DATA

2. SPARK

2.1 ADDIONAL JARS

2.2 QUERY

3. ELASTICSEARCH

3.1 ELASTICSEARCH INDEX

3.2 ELASTICSEARCH API

4. KIBANA

4.1 KIBANA LINK

1. DATA:

**********

DATA PATH: /tmp/dr_loc_cause_population.csv

2. SPARK:

************

2.1 ADDIONAL JARS:

--------------------

elasticsearch-hadoop-2.3.2.jar

2.2 QUERY:

-------------

pyspark --master local[4] --jars /tmp/elasticsearch-hadoop-2.3.2.jar

dmo=sc.textFile("/tmp/filename.csv").map(lambda line: line.split("|"))

dmo1=dmo.map(lambda (a,b,c): ('key', {'firstname':str(b), 'secondname' : str(c), 'lastname' : str(d)}))

dmo1.take(3)

dmo1.saveAsNewAPIHadoopFile(path='-',

outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",

keyClass="org.apache.hadoop.io.NullWritable",

valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",

conf={"es.nodes" : "localhost","es.port" : "9200","es.resource" : "elas/data"})

3. ELASTICSEARCH API:

---------------------------------

http://localhost:9200/elas/_search?pretty

4.1 KIBANA LINK:

---------------------------

http://50.232.31.41:5601