HIVE An Introduction

Hive was founded by Facebook in August 2007 and later made open source in 2008 .The main idea behind the creation  of HIVE was to provide a SQL like flavor for the Hadoop.

The problem faced were that the Hadoop Map Reduce needs a lot of code for the simple programs and lack the expressability of the SQL.

HIVE has a SQL like dialect called HQL (Hive Query Language).It has made the solution very easy as anyone having the knowledge of SQL can easily work upon it.

Hive is best suited for data warehouse applications, where a large data set is maintained and mined for insights, reports, etc.

However , HIVE is not a proper database as it lacks some basic properties of the database.

1.)The record level update is not possible in the HIVE.
2.)HIVE does not provide transactions
3.)Even small data set required a large latency .

HIVE consist mainly of three parts:-

1.) It contains of multiple JAR files and each having some different functionality.They are normally available in the $HIVE_HOME/lib directory.
2.)The second part has executable scripts present in the $HIVE_HOME/bin directory.CLI (Command Line Interface) is invoked with the help of this scripts.
3.)HIVE has also a thrift services to access it's services via ODBC / JDBC driver .It is normally used by the reporting solution like Tableau and Qlikview.

HIVE Architecture:-



Apart from this HIVE also has a meta store that is built-in  DERBY database.It is used to store table schema and other metadata.

The DERBY database is normally used for learning purpose and we cannot run two instances of the HIVE CLI as derby is a single process storage.


Starting with HIVE:-

Just type HIVE in the prompt and a hive session will open with HIVE prompt hive> and a secondary prompt comes like this >.

Hive Prompt:-
 

Secondary Prompt:-
 

A Simple Query in HIVE :-



In the above example , we have created a table and try to see the data but we do not have any data .Finally we have dropped the table.

We should note that whenever our query is correct ,OK should be there and later the query result.

Map Reduce Part - 1

History :-  Map Reduce was first implemented by GOOGLE .Initially ,It was coded in C++ language but later it has been  re coded into Java.

Phases of Map Reduce :-  The Map and Reduce is divided into two main parts Map and Reduce stage. But it has various sub stages inside these two stages.

1.) Record Reader
2.) Mapper
3.)In Memory Sorting
4.)Merge
5.)Shuffle
6)Reducer



Let us start our journey with the first part Mapper .
   `
Record Reader :- Let us suppose we have a file.txt having few lines embedded inside it.

 
So, it will be divided into two parts key and value .

Key  =0                               Value = How are you ?
Key = 15                             Value = I am good.

Here , the key refers to the byte offset and in first case the byte offset is 0 but in the second value . The byte offset is 15 To have the byte offset , we need to count the letters "How" -3 , Space -1 , are -3 ,Space -1, you  -3 space-1  ?-1  we will have a    /n after the first line also  -1-

Adding all these things we have     3+1+3+1+3+1+1+1 = 14
so ,the next line will start from the 15th character .


Later , the required key,value is fed into the Mapper part .

Mapper :-   Once the input is passes it's processing from the Record reader .it is fed into the Mapper .The language in a mapper is a program dependent .It can be any programming language based upon our convenience.

To understand this problems ,let us take an example of the word count problem.
Suppose we have a sentence " how are you  I am good".

The output of the mapping process will be something like this.

how    1
are      1
you     1
i          1
am      1
good   1

 So, the output of mapper will be something like the above output.

 

Sorting :- Once all the above processes are done then in memory sorting will take place on the data part .

We will discuss other components in our next tutorials.