Commodity hardware basically means the PCs which is used to actually make up the cluster. Here commodity hardware essentially means it’s not gone be a cheap piece of machine. It means that when you are actually procuring the machine to build up your Hadoop cluster the hardware need not be confined or tied up to a single vendor meaning you can actually source different component of your PCs from different vendors and still actually build your systems and it easy to maintain and easy to procure and not tied up to single vendor. That’s what actually is meant by commodity hardware.
Cluster
Cluster or a grid is basically an interconnected system in a network and node here refers to a single piece of computer or single instance of computer.A distributed system is basically a cluster which is made up of multiple nodes and each of them are going to be connected by means of the network equipment which can be either your Ethernet cable or it can be fiber optic cables or it can also be WI-Fi, but typically in the industry is going to be high speed Ethernet cables all it's going to be fiber optic cable so this is what distributed system actually means basically a cluster of interconnected machines which can communicate over network.
Hadoop is an open
source Technology stack provided by the Apache software Foundation,
now why are there major vendors out here so it is open source, it’s
should be completely free and why do we have all these companies
like Cloudera, Hortonworks, MAPR, the IBM the Microsoft.
The answer for this
question is however even though the major components of Hadoop and
the Hadoop itself is going to be completely open source and free, it
is not so easy for large companies to rely on the open source
framework unless they deploy a large number of administrators and
engineers to fit everything together and make sure Hadoop cluster is
running smoothly and
troubleshoot
whenever there is a down time. It's not so easy.
What these companies
do? The Cloudera, Hortonworks, MAPR and other big players, what they try to do is that
they basically use the same open source Hadoop from Apache and they
try to bundle everything
together and try to add wrapper on the top of that, which is
basically their management tools and a lot of GUI based
troubleshooting and management tools and to certain extent also
automate the cluster setup and troubleshooting. It not completely
automated but to a large extent the cluster setup and troubleshooting
is very very easy with a few numbers of administrators in place.
Actually, not all
the companies , specially the mid-sized ones cannot afford to have
an army of engineer working 24
x7.Performance that's going to be a herculean task for a lot of
mid-sized companies and they have a budget crunch and the number of
resources what they hire is going to be strictly limited. In such
cases these vendors come to rescue while they have tools at their
disposal
of the administrator
where managing the cluster , fine tuning the cluster , optimizing it
and troubleshooting it is becoming really easy because of the lot of
tools what this company try to bundle along with the open source
Hadoop which is provided by Apache. So they might also add some of
their own touch to the open source apache Hadoop but it will be
mostly the open source one.
Cloudera,
Hortonworks, MAPR these are the three major players , now cloudera
and Hortonworks are merged. The foot print of Cloudera is pretty high
over here.
Hadoop is not a
full-fledged distributed operating system. It’s a piece of software
which is sitting on top of an existing standalone desktop operating
system, and some of the common operating system what Hadoop supports
are mostly the flavors of Linux. So we have the CentOS, the Oracle
Linux, the Ubuntu.
Traditional RDBMS
is schema on write whereas your Hadoop is schema on read while
writing the data there is no checking , basically you can dump all
kinds of data, only when you want to perform the analysis you can
actually impose the schema. That's why it calls scheme on read.
Coming back to the
processing, it's gone be both interactive and batch processing is
supported for traditional RDBMS, whereas in Hadoop it’s strictly
batch processing. However, there are third party tools can sit on top
of Hadoop that can query the HDFS data to give up close to real time
interactive
processing but not
as close as what the traditional RDBMS provides.
RDBMS is strictly
used for storing structured data whereas Hadoop can store all kinds
of data structured, semi- structured and unstructured and everything.
Typically your
existing traditional RDBMS systems can start choking if data exceeds
of few terabytes . But in your Hadoop there is literally no limit. We
say it can handle data up to petabytes and can go beyond that also.
And the scaling
model when it comes to data storage it's kind of like nonlinear
meaning it's not easy to horizontally scale your traditional RDBMS
system when your data loads are increasing exponentially, whereas
your Hadoop can very smoothly expand in terms of its data storage
capability without affecting the query performance or without
affecting the data analysis performance. That is
the key feature out here. So it’s kind of like smooth linear
scalability in terms of data storage can be achieved using Hadoop.
Schema yes in your
traditional data base management system the data has to strictly
comply with schema specification you cannot change this during run
time but whereas the Hadoop is big relief here is that it is very
very accommodating to the changing schema needs.
And a Traditional
RDBMS systems are computational intensive. While in Hadoop It’s
like simple pieces which actually built together and work in large
numbers, so it is more data intensive. In the sense it is data
intensive, it's also CPU intensive or it's also computational
intensive but when compared to traditional RDBMS system, all these
computation does not happen on a single server. It happens on
multiple machines which are distributed in a cluster. That's the
major difference out here.
We have Shared file
storage in our traditions RDBMS based systems but in Hadoop we
actually move the code to the data.
We would discuss
about this part of what is the concept of moving the code to the data
when we discuss about the MapReduce Framework when we go forward.
The parallel way of trying to solve a problem by the parallel processing or grid processing was known by the term called Super Computing.This idea of super computing is pretty old and date backs to 1950s and 1960s .
The major vendors of supercomputers include the IBM, the FUJITSU, the CRAY , the INTEL and have a dozen of companies to actually add up to the list of super computer vendors throughout the globe.
Basically super computer is nothing but a cluster of computers which is interconnected by means of networking hard drive. You can see the picture
out here where we have the various computing resources , actually is going
to be stacked in the rack and you have the network engineers and the system administrators have to physically wire them up and enable them to actually communicate over a network.
Now having spoken about super computers , what are the use cases of super computers? What was the real need for super computer and who are the companies or who are the organizations who are actually using super computers?
Mostly supercomputers were restricted to University research labs and research lab owned by individual organization mainly , they were used in the areas of Computational fluid dynamics research Bioinformatics and a lot more.
The general purpose operating system like Framework did not exist for parallel computing need, meaning if a company actually is selling supercomputer it did not have a ready to sell off the shelf operating system that can be readily installed as a super computer goes live. It was not as simple as that. The companies procuring the supercomputers well locked to specific vendors for Hardware support. Suppose you are buying the super computer from IBM than you have to go back to IBM for any kind of hardware support. The high initial cost of hardware the supercomputer literally cost at millions of dollars . and you have to develop the custom software for each individual use case.
For example, if your organization has procured supercomputer you have to write a full-fledged operating system.However, the basic Framework support probably could be available to an open source, a source but for most part you will have to actually customize it for your use cases.
So heavily you have to depend upon the internal software engineering team to tailor the software for each kind of problem what you want to solve using the super computer. This actually let to the high cost of software maintenance, upgrades, bug fixes, and everything has to be taken care in house.
It is not simple to actually scale the cluster horizontally. Meaning if you want to actually increase the computing capacity of a supercomputer or the storage capacity in your supercomputer, you cannot do it very easily. You require some kind of support from the super computer vendors itself. So these were all the challenges of super computing.
Sys.argv
deals with the no of arguments that will be passed through the
command line arguments.The command line arguments represent all the
arguments coming through the command line.The no of command line
arguments start with 0.
We will write a sample code check the usefulness of this command.
We need to import the sys module for it.
Let
us run the program and check the output.
Output
:-
The first print statement gives us the name of the program as we are giving the argument as 0 (sys.argv[0]).
The second print statement will print the total number of arguments passed in the above program.
We
should note the below points regarding command line arguments :-
The sys.argv takes the
command line arguments in the form of a list.
The
first element in the list is the name of the file.
The arguments always
come in the form of a string even if we type an integer in the
argument list. We need to use int() function to convert the string
to integer.
