Hadoop - Master Slave Architecture

Suppose we have a hadoop cluster of 1000 nodes and let us say out of which 3 nodes would predominantly be operating in a master mode that is the name node, secondary name node, and the Job tracker .The remaining 1000 minus 3, that is your 997 machines are going to be working in the slave mode and they are going to be the data nodes.We should note that the machines which is working in a slave mode is your data nodes and the machines which is working in the master mode which will be your Name node, Secondary Name Node and Job Tracker.

However, They actually vary slightly in terms of their hardware configurations. For example, the name node, the secondary name node and Job tracker need not have a very high hard drive storage space whereas your data nodes will be very high in terms of your Hard Drive storage space because there the once which actually is going to bare all the data loads in terms of all your big data storage is going to happen on these data nodes which is going to bare the bulk of the data operations.

It is also important to understand that a host operating system or a native operating system would be install on all of these machines, both which is working as data nodes in slave mode as well as the machines in master mode. The native operating system 90% of the time it‘s going to be a Linux based operating system and in some cases we do have Windows based operating system which is installed on these machines.

It is on this native operating system that Hadoop as a piece of software framework is going to be installed. We should note that , Hadoop would be installed on all of these machines on name node, secondary name node, job tracker as well as in each one of the data nodes. And the differentiating factor between the master mode and the machines in the slave mode is that it is the software configuration after installing Hadoop which would enable each of these machines to actually perform responsibilities associated with that of a name node, and that of a secondary name node and that of a job tracker.

Big Data - Commodity Hardware and Cluster

Commodity Hardware

Commodity hardware basically means the PCs which is used to actually make up the cluster. Here commodity hardware essentially means it’s not gone be a cheap piece of machine. It means that when you are actually procuring the machine to build up your Hadoop cluster the hardware need not be confined or tied up to a single vendor meaning you can actually source different component of your PCs from different vendors and still actually build your systems and it easy to maintain and easy to procure and not tied up to single vendor. That’s what actually is meant by commodity hardware.

Cluster

Cluster or a grid is basically an interconnected system in a network and node here refers to a single piece of computer or single instance of computer.A distributed system is basically a cluster which is made up of multiple nodes and each of them are going to be connected by means of the network equipment which can be either your Ethernet cable or it can be fiber optic cables or it can also be WI-Fi, but typically in the industry is going to be high speed Ethernet cables all it's going to be fiber optic cable so this is what distributed system actually means basically a cluster of interconnected machines which can communicate over network.

Hadoop - Why Hadoop need vendors like Cloudera,Hortonworks and MAPR ?

Hadoop is an open source Technology stack provided by the Apache software Foundation, now why are there major vendors out here so it is open source, it’s should be completely free and why do we have all these companies like Cloudera, Hortonworks, MAPR, the IBM the Microsoft.

 

The answer for this question is however even though the major components of Hadoop and the Hadoop itself is going to be completely open source and free, it is not so easy for large companies to rely on the open source framework unless they deploy a large number of administrators and engineers to fit everything together and make sure Hadoop cluster is running smoothly and

troubleshoot whenever there is a down time. It's not so easy.

What these companies do? The Cloudera, Hortonworks, MAPR and other big players, what they try to do is that they basically use the same open source Hadoop from Apache and they try to bundle everything together and try to add wrapper on the top of that, which is basically their management tools and a lot of GUI based troubleshooting and management tools and to certain extent also automate the cluster setup and troubleshooting. It not completely automated but to a large extent the cluster setup and troubleshooting is very very easy with a few numbers of administrators in place.

Actually, not all the companies , specially the mid-sized ones cannot afford to have an army of engineer working 24 x7.Performance that's going to be a herculean task for a lot of mid-sized companies and they have a budget crunch and the number of resources what they hire is going to be strictly limited. In such cases these vendors come to rescue while they have tools at their disposal

of the administrator where managing the cluster , fine tuning the cluster , optimizing it and troubleshooting it is becoming really easy because of the lot of tools what this company try to bundle along with the open source Hadoop which is provided by Apache. So they might also add some of their own touch to the open source apache Hadoop but it will be mostly the open source one.

Cloudera, Hortonworks, MAPR these are the three major players , now cloudera and Hortonworks are merged. The foot print of Cloudera is pretty high over here.

Hadoop is not a full-fledged distributed operating system. It’s a piece of software which is sitting on top of an existing standalone desktop operating system, and some of the common operating system what Hadoop supports are mostly the flavors of Linux. So we have the CentOS, the Oracle Linux, the Ubuntu.

Big Data - Hadoop vs RDBMS

Traditional RDBMS is schema on write whereas your Hadoop is schema on read while writing the data there is no checking , basically you can dump all kinds of data, only when you want to perform the analysis you can actually impose the schema. That's why it calls scheme on read.



Coming back to the processing, it's gone be both interactive  and batch processing is supported for traditional RDBMS, whereas in Hadoop it’s strictly batch processing. However, there are third party tools can sit on top of Hadoop that can query the HDFS data to give up close to real time interactive

processing but not as close as what the traditional RDBMS provides.

RDBMS is strictly used for storing structured data whereas Hadoop can store all kinds of data structured, semi- structured and unstructured and everything.

Typically your existing traditional RDBMS systems can start choking if data exceeds of few terabytes . But in your Hadoop there is literally no limit. We say it can handle data up to petabytes and can go beyond that also.

And the scaling model when it comes to data storage it's kind of like nonlinear meaning it's not easy to horizontally scale your traditional RDBMS system when your data loads are increasing exponentially, whereas your Hadoop can very smoothly expand in terms of its data storage capability without affecting the query performance or without affecting the data analysis performance. That is the key feature out here. So it’s kind of like smooth linear scalability in terms of data storage can be achieved using Hadoop.

Schema yes in your traditional data base management system the data has to strictly comply with schema specification you cannot change this during run time but whereas the Hadoop is big relief here is that it is very very accommodating to the changing schema needs.



And a Traditional RDBMS systems are computational intensive. While in Hadoop It’s like simple pieces which actually built together and work in large numbers, so it is more data intensive. In the sense it is data intensive, it's also CPU intensive or it's also computational intensive but when compared to traditional RDBMS system, all these computation does not happen on a single server. It happens on multiple machines which are distributed in a cluster. That's the major difference out here.

We have Shared file storage in our traditions RDBMS based systems but in Hadoop we actually move the code to the data.

We would discuss about this part of what is the concept of moving the code to the data when we discuss about the MapReduce Framework when we go forward.

Big Data - Challenges of parallel computing

The parallel way of trying to solve a problem by the parallel processing or grid processing was known by the term called Super Computing.This idea of super computing is pretty old and date backs to 1950s and 1960s .

The major vendors of supercomputers include the IBM, the FUJITSU, the CRAY , the INTEL and have a dozen of companies to actually add up to the list of super computer vendors throughout the globe.

Basically super computer is nothing but a cluster of computers which is interconnected by means of networking hard drive. You can see the picture
out here where we have the various computing resources , actually is going
to be stacked in the rack and you have the network engineers and the system administrators have to physically wire them up and enable them to actually communicate over a network.

Now having spoken about super computers , what are the use cases of super computers? What was the real need for super computer and who are the companies or who are the organizations who are actually using super computers?

Mostly supercomputers were restricted to University research labs and research lab owned by individual organization mainly , they were used in the areas of Computational fluid dynamics research Bioinformatics and a lot more.

The general purpose operating system like Framework did not exist for parallel computing need, meaning if a company actually is selling supercomputer it did not have a ready to sell off the shelf operating system that can be readily installed as a super computer goes live. It was not as simple as that. The companies procuring the supercomputers well locked to specific vendors for Hardware support. Suppose you are buying the super computer from IBM than you have to go back to IBM for any kind of hardware support. The high initial cost of hardware the supercomputer literally cost at millions of dollars . and you have to develop the custom software for each individual use case.

For example, if your organization has procured supercomputer you have to write a full-fledged operating system.However, the basic Framework support probably could be available to an open source, a source but for most part you will have to actually customize it for your use cases.

So heavily you have to depend upon the internal software engineering team to tailor the software for each kind of problem what you want to solve using the super computer. This actually let to the high cost of software maintenance, upgrades, bug fixes, and everything has to be taken care in house.
It is not simple to actually scale the cluster horizontally. Meaning if you want to actually increase the computing capacity of a supercomputer or the storage capacity in your supercomputer, you cannot do it very easily. You require some kind of support from the super computer vendors itself. So these were all the challenges of super computing.