Hadoop : Architecture

Hadoop employs a master-slave architecture for both distributed storage and distributed computation.

In layman terms , The master-slave architecture means there will be some authority who will divide the work, do some checks at regular interval,keep the log of work that has been assigned to it's employee.In the same way, hadoop keep track of all the data division, processing,job tracking with the help of some of it's components.

On a fully running cluster , we can have some daemons that run on this cluster and completes a master slave architecture in hadoop.

Suppose we have a cluster of 100 nodes then ,97 will work in slave mode and act as a data node.Rest 3 will be Name Node ,Secondary Name Node and Job Tracker.It can also depend upon the configuration of the individual cluster.

Five daemons that are present in the Hadoop :-
1.) Name Node
2.) Data Node
3.) Secondary Name Node
4.) Job Tracker
5.)Task tracker

Name Node :- Hadoop employs a master/slave architecture for both distributed storage and distributed computation. The distributed storage system is called the Hadoop File System , or HDFS. The NameNode is the master of HDFS that directs the slave DataNode daemons to perform the low-level I/O tasks. The NameNode is the bookkeeper of HDFS;

it keeps track of how your files are broken down into file blocks, which nodes store those blocks, and the overall health of the distributed  filesystem.It is the single point of failure. Normally, the server having the Name Node does not contain any data or computation process.

Data Node:- When you want to read or write a HDFS fi le, the fi le is broken into blocks and the NameNode will tell your client which DataNode each block resides in.Your client communicates directly with the DataNode daemons to process the local files corresponding to the blocks. Furthermore, a DataNode may communicate with other DataNodes to replicate its data blocks for redundancy.DataNodes are constantly reporting to the NameNode.

Upon initialization, each of  the DataNodes informs the NameNode of the blocks it’s currently storing

Secondary Name Node :-The Secondary NameNode (SNN) is an assistant daemon for monitoring the state of the cluster HDFS. Like the NameNode, each cluster has one SNN, and it typically resides on its own machine as well. No other DataNode or TaskTracker daemons run on the same server. The SNN differs from the NameNode in that this process doesn’t receive or record any real-time changes to HDFS. Instead, it communicates with the NameNode to take snapshots of the HDFS metadata at intervals defined by the cluster configuration.

Job Tracker :-The JobTracker daemon is the liaison between your application and Hadoop. Once you submit your code to your cluster, the JobTracker determines the execution plan by determining which fi les to process, assigns nodes to different tasks, and monitors all tasks as they’re running. Should a task fail, the JobTracker will automatically relaunch the task, possibly on a different node, up to a predefi ned limit of retries. There is only one JobTracker daemon per Hadoop cluster. It’s typically run on a server as a master node of the cluster.

Task Tracker:- Each TaskTracker is responsible for executing the individual tasks that the JobTracker assigns. Although there is a single TaskTracker per slave node,One responsibility of the TaskTracker is to constantly communicate with the JobTracker. If the JobTracker fails to receive a heartbeat from a TaskTracker within a specifi ed amount of time, it will assume the TaskTracker has crashed and will resubmit the corresponding tasks to other nodes in the cluster.

 



How to see the deamons in Hadoop environemt ?

once you start the Hadoop by typing :  start-all.sh

All the daemons will be up and running .We can verify this with the help of the jps command.

 

Support Vector Machines

SVM (support vector machine) is one of the variant of  supervised learning algorithm.It is considered as one of the best classification algorithm.It works well with the binary classifier but with little modification  can also work with the multiple classes as well.The complete idea of Support Vectors resides on finding the best hyperplane that classifies our data set.



What is HyperPlane ?

The line which divided the data set and classifies them.In the above example , the line(decision Boundary) classifies the data set into two different classes.
It should be noted that farther the datapoint from the Hyperplane the more correctly it has been classified.

In case of two-dimensional architecture , it can be easy to find a plane that classifies our data but in real time ,we have to face the multi-dimensional data and there is no clear classification then  it is inefficient to use the above mentioned concept.In such scenario, the concept of kernel comes into the picture which we will dicuss in our coming tutorials.

What are Support Vectors ?

Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane.

Advantages of SVM :-

1.) Works well with smaller data set.
2.) Have a good accuracy

Disadvantages of SVM :-

1.) Not so efficient with enormous data set
2.) Noisier data set is difficult to handle.

We will learn more about SVM kernel and it's python implementation in our upcoming tutorials.

Map vs FlatMap In Pyspark

Map() :- Return a new distributed dataset formed by passing each element of the source through a function.

Map can be considered as 1: 1 relationship in spark It means each element is associated with it's corresponding element.

The input and output will be RDD for this function.



FlatMap():-Similar to map, but each input item can be mapped to 0 or more output items (so func should return a Seq rather than a single item).

The input and output will be RDD for this function also.

It flattens multiple list into single list. 

Python Code :-

OUTPUT OF MAP:-



OUTPUT OF FLATMAP:-


Difference between MAP and FLATMAP :-

map() output is an RDD whereas flatMap() output is RDD containing elements of all iterators.

You can find the datafile and related code on my github id :-
 https://github.com/sangam92/Spark_tutorials

Hadoop - An Introduction

Hadoop is an open source java based network that can be used for writing and running distributed application that runs on a large amount of data.

What makes hadoop different from the  normal distributed system ;

• Hadoop runs on large clusters of commodity hardware and also on the cloud like Amazon (EC2)
• Since Hadoop runs on commodity hardware so the failure will be frequent.It can handle the failure easily.
• It is highly scalable means it can handle large data by adding more nodes on the cluster.
• Hadoop allows user to write simple parallel code.
  

 

Hadoop simplicity and accessibility has given it an edge over the existing technologies.

Hadoop has two main component :

Storage , can store a huge amount of data structured or unstructured data.

Processing ,has a parallel processing framework.

The storage part is handled by the HDFS (Hadoop Distributed File System) while the Processing is managed by MapReduce.



 

Introduction to Big Data

Why we need Big Data ?

It is really the question of the century.well! we do not have facebook,twitter,linkedin,youtube in the past.

if we notice the data trend , there is a huge increase in the data in the past few years.

90 % of the data has been generated in the last 2-3 years.

Our traditional RDBMS system can handle good amount of data but to handle such a huge data is currently out of the scope of such systems.


RDBMS cannot accommodate such a huge and unstructured data.

Why Big Data become so important suddenly ?

The data has variety and amount also which can be useful for mining lot of information. It contains a lot of business value and contain a lot of insight.

 The current technology is sufficient to handle such a huge amount of data .

Challenges of Big Data 

How to scale up storage methodology.

How to analyze such a huge data.

Sources of Big Data

As the data is growing day by day , we need to check the sources that are producing such an enormous amount of data.

Source of Big Data is mainly coming from two sources :-

1.) Machine Generated Data :- Data is generated with minimal human intervention with the computer system.
example :- Sensors , Gaming Data.

2.)Human Generated Data :- Data that is generated by the computer system with the human intervention.
example :- Facebook,LinkedIn etc.

Major Sources of Data are

•  Stock exchanges 
•  Banks 
•  Social Networking sites.

Types of Data

Data is primarily classified into three types :-

1.) Structured Data
2.) Unstructured Data
3.) Semi Structured Data

Structured Data :-  It includes those data which fits perfectly with traditional RDBMS concept.
                                 The data is perfectly carved into rows and columns.
                                 The data can easily fit into the tabular format.
Example :- mysql database

Unstructured Data :- These kind of data do not have a fixed format or structure.
                                     It cannot be stored in traditional rows and column format.
                                     It cannot support any particular schema.
                                     We can put into tabular format.
Example :- Video streaming data,pdf,whatsapp messages etc.

Semi-Structured Data :-It is in between structured and unstructured one.
                                         It has some tags and schema associated with it.

Example:-  XML files