Transfer file in Hadoop

Spark Interview Question

1.) What are the different phases that took place when we submit spark job?

2.) What is Data Lineage?

3.) What is Lazy Evaluation?

4.) What is RDD ?

5.) What is Stage in Spark?

6.) What is DAG?

7.) Difference between Coalesce and Repartition.

8.) What is co group in Spark?

9.) Difference between Map and Flatmap.

10.)Difference between Datasets and Data Frames.

11.)Difference between MR and SPark.

12.) Difference between Cache and Persist.

13.) What do you mean by in memory calculation in spark?

14.)What is Kryo serializer ?

15.)What is a driver program?

16.)What is the default resource manager in spark?

17.)How to kill a job in Hadoop?

18.)How to deploy spark job in cluster mode?

19.) In which langiage spark is built

20.) What is the current Spark version ?

Python - Command Line Arguments

Sys.argv deals with the no of arguments that will be passed through the command line arguments.The command line arguments represent all the arguments coming through the command line.The no of command line arguments start with 0.

We will write a sample code check the usefulness of this command.

We need to import the sys module for it.

Let us run the program and check the output.

Output :-

The first print statement gives us the name of the program as we are giving the argument as 0 (sys.argv[0]).

The second print statement will print the total number of arguments passed in the above program.


We should note the below points regarding command line arguments :-

• The sys.argv takes the command line arguments in the form of a list.

• The first element in the list is the name of the file.

• The arguments always come in the form of a string even if we type an integer in the argument list. We need to use int() function to convert the string to integer.
  

Spark - Basic Statistics

We have already gone through the tutorial on Measure of Central Tendency.Now we will do it’s implementation in Pyspark.We need to import statistics module from pyspark.mlib.stat.

Once the spark job is submitted , we will get the below output as the result.

The below code is available in my Github library :- https://github.com/sangam92/Spark_tutorials

Spark - Correlation

Correlation explains us the association between two or more variables .The movement of one variable will impact the movement of another variable .
It is normally used in situation where we want to explore how two variables are related with each other.

Types of Correlation :-
Correlation can be classified in several ways but in most generic way it is divided into three ways.

(i) Positive and Negatives
(ii) Linear and Non Linear
(iii) Simple,partial and multiple

Positive and Negative : - Correlation can be positive and negative .If both the variable are moving in the same direction , we can termed them as positive correlation else it can be termed as negative correlation.

Linear and Non Linear :- If the change in one variable is accompanied by change in another variable in a constant ratio, it is a case of linear correlation.On the other hand, if the amount of change in one variable does not follow a constant ratio with the change in another variable, it is a case of non-linear or curvilinear correlation.

Simple,partial and multiple :- If only two variables are involved in a study, then the correlation is said to be simple correlation.When three or more variables are involved in a study, then it is a problem of either partial or multiple correlation. In multiple correlation, three or more variables are studied simultaneously. But in partial correlation we consider only two variables influencing each other while the effect of other variable(s) is held constant.


Let us implement a simple python code :-

from pyspark import SparkContext,SparkConf

from pyspark.mllib.stat import Statistics

import numpy as np

conf=SparkConf().setAppName("test")

sc =SparkContext(conf=conf)

seriesX = sc.parallelize([1.0, 2.0, 3.0, 3.0, 5.0]) # a series

# seriesY must have the same number of partitions and cardinality as seriesX

seriesY = sc.parallelize([11.0, 22.0, 33.0, 33.0, 555.0])

# Compute the correlation using Pearson's method. Enter "spearman" for Spearman's method.

# If a method is not specified, Pearson's method will be used by default.

print("Correlation is: " + str(Statistics.corr(seriesX, seriesY, method="pearson")))

data = sc.parallelize(

[np.array([1.0, 10.0, 100.0]), np.array([2.0, 20.0, 200.0]), np.array([5.0, 33.0, 366.0])]) # an RDD of Vectors

# calculate the correlation matrix using Pearson's method. Use "spearman" for Spearman's method.

# If a method is not specified, Pearson's method will be used by default.

print(Statistics.corr(data, method="pearson"))

Neural Networks - An Introduction

Neural Networks in computer system are quite analogues to what the neurons we have in our brain. Our brain that is the collection of millions of neurons can be considered as the most advanced system. The decision taken by brain are impeccable and are accompanied with a lot of decision making that involved a lot of permutation and combination.

Scientist were working day and night to produce an AI enabled system that can solve majority of problem. Few AI system were developed also with the capability to solve the problem in a formal way but do not have a broader prospect. The IBM Deep Blue was one of such invention that defeated World Chess Champion Garry Kasprov.

Initially the AI system based their knowledge on some set of rules and regulation. Such kind of system were not so robust and had certain limitation. The human need to feed every possible situation which a cumbersome job was. Such approach was called Knowledge Base approach. The scientist devised a new methodology which does not require any specific rules and work according to the raw data fed into the system.

The scientist works upon the machine learning approaches and have many successes, one of them was logistic regression which was able to predict the cesarean delivery by feeding the system with certain inputs which was called feature extraction. Similarly, Naïve Bayes was used to classify the email as Spam and Non-Spam. The feature extraction was quite a tough job and require lots of human effort and time, moreover it requires a top-level domain expertise.

The ground-breaking innovation came into the year 1960 when Frank Rosenblatt has discovered an artificial neuron called Perceptron. Later, the evolution take place in such a perceptron and lead the way for the Multilayer Perceptron (MLP).

We will cover about Perceptron and Multilayer Perceptron in our next blog post.

Python - Keywords

Python Keywords

Keywords are the reserved words and we cannot use a keyword as variable name, function name or an Identifier. Almost 33 keywords are available in Python which is quite lesser than other languages like Java, C++ etc.

Method	Description
and	A logical operator
as	To create an alias
assert	For debugging
break	To break out of a loop
class	To define a class
continue	To continue to the next iteration of a loop
def	To define a function
del	To delete an object
elif	Used in conditional statements, same as else if
else	Used in conditional statements
except	Used with exceptions, what to do when an exception occurs
False	Boolean value, result of comparison operations
finally	Used with exceptions, a block of code that will be executed no matter if there is an exception or not
for	To create a for loop
from	To import specific parts of a module
global	To declare a global variable
if	To make a conditional statement
import	To import a module
in	To check if a value is present in a list, tuple, etc.
is	To test if two variables are equal
lambda	To create an anonymous function
None	Represents a null value
nonlocal	To declare a non-local variable
not	A logical operator
or	A logical operator
pass	A null statement, a statement that will do nothing
raise	To raise an exception
return	To exit a function and return a value
True	Boolean value, result of comparison operations
try	To make a try...except statement
while	To create a while loop
with	Used to simplify exception handling
yield	To end a function, returns a generator

---

                                                  List of Key Words

There are some basic rules that need to be followed with the keywords.No identifiers should have the same name as any of the keywords.Names that begin with __ underscore should not be used.