Map Reduce Part - 1

History :-  Map Reduce was first implemented by GOOGLE .Initially ,It was coded in C++ language but later it has been  re coded into Java.

Phases of Map Reduce :-  The Map and Reduce is divided into two main parts Map and Reduce stage. But it has various sub stages inside these two stages.

1.) Record Reader
2.) Mapper
3.)In Memory Sorting
4.)Merge
5.)Shuffle
6)Reducer



Let us start our journey with the first part Mapper .
   `
Record Reader :- Let us suppose we have a file.txt having few lines embedded inside it.

 
So, it will be divided into two parts key and value .

Key  =0                               Value = How are you ?
Key = 15                             Value = I am good.

Here , the key refers to the byte offset and in first case the byte offset is 0 but in the second value . The byte offset is 15 To have the byte offset , we need to count the letters "How" -3 , Space -1 , are -3 ,Space -1, you  -3 space-1  ?-1  we will have a    /n after the first line also  -1-

Adding all these things we have     3+1+3+1+3+1+1+1 = 14
so ,the next line will start from the 15th character .


Later , the required key,value is fed into the Mapper part .

Mapper :-   Once the input is passes it's processing from the Record reader .it is fed into the Mapper .The language in a mapper is a program dependent .It can be any programming language based upon our convenience.

To understand this problems ,let us take an example of the word count problem.
Suppose we have a sentence " how are you  I am good".

The output of the mapping process will be something like this.

how    1
are      1
you     1
i          1
am      1
good   1

 So, the output of mapper will be something like the above output.

 

Sorting :- Once all the above processes are done then in memory sorting will take place on the data part .

We will discuss other components in our next tutorials.

Hadoop : Architecture

Hadoop employs a master-slave architecture for both distributed storage and distributed computation.

In layman terms , The master-slave architecture means there will be some authority who will divide the work, do some checks at regular interval,keep the log of work that has been assigned to it's employee.In the same way, hadoop keep track of all the data division, processing,job tracking with the help of some of it's components.

On a fully running cluster , we can have some daemons that run on this cluster and completes a master slave architecture in hadoop.

Suppose we have a cluster of 100 nodes then ,97 will work in slave mode and act as a data node.Rest 3 will be Name Node ,Secondary Name Node and Job Tracker.It can also depend upon the configuration of the individual cluster.

Five daemons that are present in the Hadoop :-
1.) Name Node
2.) Data Node
3.) Secondary Name Node
4.) Job Tracker
5.)Task tracker

Name Node :- Hadoop employs a master/slave architecture for both distributed storage and distributed computation. The distributed storage system is called the Hadoop File System , or HDFS. The NameNode is the master of HDFS that directs the slave DataNode daemons to perform the low-level I/O tasks. The NameNode is the bookkeeper of HDFS;

it keeps track of how your files are broken down into file blocks, which nodes store those blocks, and the overall health of the distributed  filesystem.It is the single point of failure. Normally, the server having the Name Node does not contain any data or computation process.

Data Node:- When you want to read or write a HDFS fi le, the fi le is broken into blocks and the NameNode will tell your client which DataNode each block resides in.Your client communicates directly with the DataNode daemons to process the local files corresponding to the blocks. Furthermore, a DataNode may communicate with other DataNodes to replicate its data blocks for redundancy.DataNodes are constantly reporting to the NameNode.

Upon initialization, each of  the DataNodes informs the NameNode of the blocks it’s currently storing

Secondary Name Node :-The Secondary NameNode (SNN) is an assistant daemon for monitoring the state of the cluster HDFS. Like the NameNode, each cluster has one SNN, and it typically resides on its own machine as well. No other DataNode or TaskTracker daemons run on the same server. The SNN differs from the NameNode in that this process doesn’t receive or record any real-time changes to HDFS. Instead, it communicates with the NameNode to take snapshots of the HDFS metadata at intervals defined by the cluster configuration.

Job Tracker :-The JobTracker daemon is the liaison between your application and Hadoop. Once you submit your code to your cluster, the JobTracker determines the execution plan by determining which fi les to process, assigns nodes to different tasks, and monitors all tasks as they’re running. Should a task fail, the JobTracker will automatically relaunch the task, possibly on a different node, up to a predefi ned limit of retries. There is only one JobTracker daemon per Hadoop cluster. It’s typically run on a server as a master node of the cluster.

Task Tracker:- Each TaskTracker is responsible for executing the individual tasks that the JobTracker assigns. Although there is a single TaskTracker per slave node,One responsibility of the TaskTracker is to constantly communicate with the JobTracker. If the JobTracker fails to receive a heartbeat from a TaskTracker within a specifi ed amount of time, it will assume the TaskTracker has crashed and will resubmit the corresponding tasks to other nodes in the cluster.

 



How to see the deamons in Hadoop environemt ?

once you start the Hadoop by typing :  start-all.sh

All the daemons will be up and running .We can verify this with the help of the jps command.

 

Support Vector Machines

SVM (support vector machine) is one of the variant of  supervised learning algorithm.It is considered as one of the best classification algorithm.It works well with the binary classifier but with little modification  can also work with the multiple classes as well.The complete idea of Support Vectors resides on finding the best hyperplane that classifies our data set.



What is HyperPlane ?

The line which divided the data set and classifies them.In the above example , the line(decision Boundary) classifies the data set into two different classes.
It should be noted that farther the datapoint from the Hyperplane the more correctly it has been classified.

In case of two-dimensional architecture , it can be easy to find a plane that classifies our data but in real time ,we have to face the multi-dimensional data and there is no clear classification then  it is inefficient to use the above mentioned concept.In such scenario, the concept of kernel comes into the picture which we will dicuss in our coming tutorials.

What are Support Vectors ?

Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane.

Advantages of SVM :-

1.) Works well with smaller data set.
2.) Have a good accuracy

Disadvantages of SVM :-

1.) Not so efficient with enormous data set
2.) Noisier data set is difficult to handle.

We will learn more about SVM kernel and it's python implementation in our upcoming tutorials.