Apache Spark Introduction

Spark started in 2009 as a research project in the UC Berkeley RAD Lab, later to become the AMPLab. The researchers in the lab had previously been working on Hadoop Map‐Reduce, and observed that MapReduce was inefficient for iterative and interactive computing jobs. Thus, from the beginning, Spark was designed to be fast for interactive queries and iterative algorithms, bringing in ideas like support for in-memory storage and efficient fault recovery.

Apache Spark is a cluster computing platform designed to be fast and general purpose.The main purpose of the Apache spark was to handle the map reduce efficiently in the speed side.
To handle the speed ,spark has in memory calculation but it can handle the complex problem better than map reduce in the disk.
Spark is highly accessible means it can offer a wide range of API's for scala, python ,sql ,java etc.It has a wide variety of libraries also.
It can be integrated with most of the Big  data tool.
 


Spark Core :- Spark Core  can handle many basic functionality like task scheduling,memory management,fault tolerance,storage system.it is the home for the most sought API in spark (RDD).
SparkSQL:- Spark supports SQL  as well as HQL(Hive Query Language).It can support many source of data including HIVE tables, JSON ,paraquet .Moreover,Spark SQL gives a blending of RDD with the SQL
MLIB:-It provides complete machine learning flavour with all the required algorithms like classification,regression ,clustering ,collaborative filtering.
GraphX:- GraphX extends the Spark RDD API, allowing us to create a directed graph with arbitrary
properties attached to each vertex and edge. GraphX also provides various operators
for manipulating graphs.

It’s important to remember that Spark does not require Hadoop; it simply has support for storage systems implementing the Hadoop APIs. Spark supports text files, SequenceFiles, Avro, Parquet, and any other Hadoop InputFormat.

Normal Distribution

It is one of the most common form of probability distribution and can take any values within a range.
The normal distribution is a type of continuous probability distribution.One of the most common graph bell curve occurs in this one.

The two parameters that characterized the Normal Distribution are :- 

1.) Mean
2.) Variance

The normal distribution can take any values from -infinity to + infinity.There is infinite number of normal distribution ,varying upon their mean and variance.

Standard Distribution / Z-distribution :- The normal distribution with mean =0 and standard deviation =1 is called standard distribution /Z distribution.

Characteristics of Normal Distribution :-
1.)Symmetry
2.)A single most common value (uni modality)
3.)Range from -infinity to + infinity
4.)Area under curve is 1
5.)A common value for the mean,median and mode.

Examples to illustrate it :-
 
A normal distribution is perfectly symmetrical around its center. That is, the right side of the center is a mirror image of the left side. There is also only one mode, or peak, in a normal distribution. Normal distributions are continuous and have tails that are asymptotic, which means that they approach but never touch the x-axis. The center of a normal distribution is located at its peak, and 50% of the data lies above the mean, while 50% lies below. It follows that the mean, median, and mode are all equal in a normal distribution.

Some of the common uses of normal distribution :-
1.) Height
2.)IQ
3.)Blood pressure
4.)Salaries etc.

The empirical rule tells you what percentage of your data falls within a certain number of standard deviations from the mean:

• 68% of the data falls within one standard deviation of the mean.
• 95% of the data falls within two standard deviations of the mean.
• 99.7% of the data falls within three standard deviations of the mean.


Z-scores/Normalized Scores:-

The formula for Z-score is :

Let us take an example to understand the concept of Z-scores .

A wild pack of Chihuahuas terrorizing the countryside has a mean height of 7.5 inches, with a standard deviation of 1.5 inches. We feel sorry for the person who had to measure that. What proportion of these Chihuahuas are between 6 and 9 inches tall?
When we want to know something about probabilities or proportions of normal distributions, we need to work with Z-scores. We use them to convert a value into the number of standard deviations it is from the mean. The formula is:
 

μ is another fancy code name for the mean of the normal distribution, while σ is its standard deviation. We can find the Z-scores for 6 and 9 inches now.

 
How much of the normal distribution falls within 1 standard deviation above or below the mean? According to the Empirical Rule, that's 68% of the distribution.







Further Reading :-https://en.wikipedia.org/wiki/Normal_distribution
                             http://www.statisticshowto.com/probability-and-statistics/normal-distributions/

Probability Distribution

A probability distribution is a table or an equation that links each outcome of a statistical experiment with its probability of occurrence.

It defines what will be the probability of an event to take place and how common are the values in a given range.

Example 1 :- Suppose we flip a coin ,the probability of Head and Tail is 0.5 and 0.5 respectively.
This can considered as the simplest probability distribution.
Example 2:-  Suppose we flip two coin and we need to find the probability of occurrence of head in each coin.
probability of 0 head ---->0.25
probability of 1 head ---->0.50
probabilty of 2 head ---->0.25

So, the above table represents our probabilty distribution.

Probability Distribution are commonly classified into two parts:-

1.) Continuous , data can take any value within a specified range.
2.) Discrete , data can take only certain values.

Rules which make the probability Distribution Valid:-

1.) The probabilities must lie in between 0 and 1 .
2.) The sum of all the probabilities must equal to 1.

Linear Regression

Linear Regression is one of the simplest supervised machine learning algorithm and the predicted output is continuous.It is used to determine the extent to which there is a linear relationship between a dependent variable and one or more independent variables.

So , just dive into the depth of the Linear Regression.

We will start with basic linear equation model.

The general linear model contains a relation between the independent Variable (x) and dependent variable (Y) .

So, what does it mean ,

case 1 .)  Y= x
 if ,
       x=3  then Y=3
       x=6  then Y=6
       x=9  then Y=9

case 2 .) Now multiply the independent variable x with a coefficient (a).
So , the equation will become   Y = ax and assume that a=2.

if,
       x=3  then Y=6
       x=6  then Y=12
       x=9  then Y=18


case 3 .) Now add a constant value b into the equation , we have got in case 2. So, the new equation will be Y=ax + b and b =3.

if,
      x=3  then Y=9
      x=6  then Y=15
      x=9  then Y=21

The linear regression can be represented by the below equation :-
                              Y= ax + b
Here ,

Y =Dependent Variable
a  =Regression Coefficient
x  =Independent Variable
b  =Constant

Now , let us understand the basic linear regression where the dependent variable x is called "feature"  while y is called "response".

We will create  the scatter plot of the above feature and response.

 Now ,the best fit line is the one which fits this scatter plot .

 The above line is called is Linear Regression.

  Equation to represent the above the Point.





here,
h(x_i) represent the predicted variable.
b_0  and b_1 represent the regression coefficient.
Now, Let us consider that the every observation contains some residual error associated with it.
 
The above equation can be written as   
 Hence , the residual error is :-

Hence , the residual error is :-

 
Now ,the focus should be on minimizing this error and so we need to take the sum of squared error (SSE)
The cost function or squared error can be defined by the below equation :-
 
After doing the calculation of the above equation , the result will be like :-

Python Implementation of the above code :-

import matplotlib.pyplot as plt
from sklearn import linear_model
x_train= [[2],[4],[6],[8],[10],[12],[14],[16]]
y_train =[[4],[6],[7],[10],[12],[15],[17],[19]]
regr = linear_model.LinearRegression()
regr.fit(x_train,y_train)
y_output =regr.predict(22)
print(regr.predict(22))
Output :- [[ 25.64285714]]

Further Reading :- https://en.wikipedia.org/wiki/Simple_linear_regression

You can download the code from my github id :- https://github.com/sangam92/Machine-Learning-tutorials

Bytecode

Bytecode is also known as the Intermediate code ,portable code or p-code.It is a code that is not read by the CPU but by the VM.The VM converts the bytecode into the machine language for the platform specific portability and that converted code is understandable by the CPU.
It means the VM converts our byte code according to the type of platform we are using.

The earlier programming language like C,C++ have cross platform compilation issues.Now with the concept of the VM and bytecode , the cross platform portability has been solved.

Range and Interquartile Range

In the most simplest term , range referred as the difference between the maximum and the minimum value .

Example :- For the dataset [23,25,34,27,38]
The maximum value is 38 and the minimum value is 23.
The range wil be  (38 - 23) = 15.
However ,if there will be an outliers in the data , range cannot be considered as the effective measure of the data inspection.

Example :- In the above dataset ,[23,25,34,27,38]
Let us suppose a new value 92.
So ,the new dataset will be like [23,25,34,27,38,92]
The maximum value is 92  and the minimum value is 23.
The range wil be  (92 - 23) = 69.
As ,we can see most of the data lies between the value 23 to 38.


To overcome this issue ,the interquartile range is used and which is the range of the 50 % values in the dataset.
It can be calculated as the difference between the 75th and 25th percentile values.It is the place where majority of data lies.And can provide the central tendency of data.
Let us understand this with an example:-
Assume a dataset having the value as
[1, 3, 4, 5, 5, 6, 7, 11]
so, we will take the range from the value 4,5,5,6.
so , the IQR =6-4 =2.


Further Reading :- https://en.wikipedia.org/wiki/Range_(mathematics)

FOR loop in python

The for loop iterate over the items of any  sequence like list or a string.

Syntax:

for iter_var in sequence:
    statement(s)

Example :

a = [2,3,4,5,6]
for i in a:
    print(i)

#output :
2
3
4
5
6

For Loop with range :.

Example :
for i in list(range(5)):
    print(i)

#output :-
0
1
2
3
4