Spark - Correlation

Correlation explains us the association between two or more variables .The movement of one variable will impact the movement of another variable .
It is normally used in situation where we want to explore how two variables are related with each other.

Types of Correlation :-
Correlation can be classified in several ways but in most generic way it is divided into three ways.

(i) Positive and Negatives
(ii) Linear and Non Linear
(iii) Simple,partial and multiple

Positive and Negative : - Correlation can be positive and negative .If both the variable are moving in the same direction , we can termed them as positive correlation else it can be termed as negative correlation.

Linear and Non Linear :- If the change in one variable is accompanied by change in another variable in a constant ratio, it is a case of linear correlation.On the other hand, if the amount of change in one variable does not follow a constant ratio with the change in another variable, it is a case of non-linear or curvilinear correlation.

Simple,partial and multiple :- If only two variables are involved in a study, then the correlation is said to be simple correlation.When three or more variables are involved in a study, then it is a problem of either partial or multiple correlation. In multiple correlation, three or more variables are studied simultaneously. But in partial correlation we consider only two variables influencing each other while the effect of other variable(s) is held constant.


Let us implement a simple python code :-

from pyspark import SparkContext,SparkConf

from pyspark.mllib.stat import Statistics

import numpy as np

conf=SparkConf().setAppName("test")

sc =SparkContext(conf=conf)

seriesX = sc.parallelize([1.0, 2.0, 3.0, 3.0, 5.0]) # a series

# seriesY must have the same number of partitions and cardinality as seriesX

seriesY = sc.parallelize([11.0, 22.0, 33.0, 33.0, 555.0])

# Compute the correlation using Pearson's method. Enter "spearman" for Spearman's method.

# If a method is not specified, Pearson's method will be used by default.

print("Correlation is: " + str(Statistics.corr(seriesX, seriesY, method="pearson")))

data = sc.parallelize(

[np.array([1.0, 10.0, 100.0]), np.array([2.0, 20.0, 200.0]), np.array([5.0, 33.0, 366.0])]) # an RDD of Vectors

# calculate the correlation matrix using Pearson's method. Use "spearman" for Spearman's method.

# If a method is not specified, Pearson's method will be used by default.

print(Statistics.corr(data, method="pearson"))

Neural Networks - An Introduction

Neural Networks in computer system are quite analogues to what the neurons we have in our brain. Our brain that is the collection of millions of neurons can be considered as the most advanced system. The decision taken by brain are impeccable and are accompanied with a lot of decision making that involved a lot of permutation and combination.

Scientist were working day and night to produce an AI enabled system that can solve majority of problem. Few AI system were developed also with the capability to solve the problem in a formal way but do not have a broader prospect. The IBM Deep Blue was one of such invention that defeated World Chess Champion Garry Kasprov.

Initially the AI system based their knowledge on some set of rules and regulation. Such kind of system were not so robust and had certain limitation. The human need to feed every possible situation which a cumbersome job was. Such approach was called Knowledge Base approach. The scientist devised a new methodology which does not require any specific rules and work according to the raw data fed into the system.

The scientist works upon the machine learning approaches and have many successes, one of them was logistic regression which was able to predict the cesarean delivery by feeding the system with certain inputs which was called feature extraction. Similarly, Naïve Bayes was used to classify the email as Spam and Non-Spam. The feature extraction was quite a tough job and require lots of human effort and time, moreover it requires a top-level domain expertise.

The ground-breaking innovation came into the year 1960 when Frank Rosenblatt has discovered an artificial neuron called Perceptron. Later, the evolution take place in such a perceptron and lead the way for the Multilayer Perceptron (MLP).

We will cover about Perceptron and Multilayer Perceptron in our next blog post.

Support Vector Machines

SVM (support vector machine) is one of the variant of  supervised learning algorithm.It is considered as one of the best classification algorithm.It works well with the binary classifier but with little modification  can also work with the multiple classes as well.The complete idea of Support Vectors resides on finding the best hyperplane that classifies our data set.



What is HyperPlane ?

The line which divided the data set and classifies them.In the above example , the line(decision Boundary) classifies the data set into two different classes.
It should be noted that farther the datapoint from the Hyperplane the more correctly it has been classified.

In case of two-dimensional architecture , it can be easy to find a plane that classifies our data but in real time ,we have to face the multi-dimensional data and there is no clear classification then  it is inefficient to use the above mentioned concept.In such scenario, the concept of kernel comes into the picture which we will dicuss in our coming tutorials.

What are Support Vectors ?

Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane.

Advantages of SVM :-

1.) Works well with smaller data set.
2.) Have a good accuracy

Disadvantages of SVM :-

1.) Not so efficient with enormous data set
2.) Noisier data set is difficult to handle.

We will learn more about SVM kernel and it's python implementation in our upcoming tutorials.

Linear Regression

Linear Regression is one of the simplest supervised machine learning algorithm and the predicted output is continuous.It is used to determine the extent to which there is a linear relationship between a dependent variable and one or more independent variables.

So , just dive into the depth of the Linear Regression.

We will start with basic linear equation model.

The general linear model contains a relation between the independent Variable (x) and dependent variable (Y) .

So, what does it mean ,

case 1 .)  Y= x
 if ,
       x=3  then Y=3
       x=6  then Y=6
       x=9  then Y=9

case 2 .) Now multiply the independent variable x with a coefficient (a).
So , the equation will become   Y = ax and assume that a=2.

if,
       x=3  then Y=6
       x=6  then Y=12
       x=9  then Y=18


case 3 .) Now add a constant value b into the equation , we have got in case 2. So, the new equation will be Y=ax + b and b =3.

if,
      x=3  then Y=9
      x=6  then Y=15
      x=9  then Y=21

The linear regression can be represented by the below equation :-
                              Y= ax + b
Here ,

Y =Dependent Variable
a  =Regression Coefficient
x  =Independent Variable
b  =Constant

Now , let us understand the basic linear regression where the dependent variable x is called "feature"  while y is called "response".

We will create  the scatter plot of the above feature and response.

 Now ,the best fit line is the one which fits this scatter plot .

 The above line is called is Linear Regression.

  Equation to represent the above the Point.





here,
h(x_i) represent the predicted variable.
b_0  and b_1 represent the regression coefficient.
Now, Let us consider that the every observation contains some residual error associated with it.
 
The above equation can be written as   
 Hence , the residual error is :-

Hence , the residual error is :-

 
Now ,the focus should be on minimizing this error and so we need to take the sum of squared error (SSE)
The cost function or squared error can be defined by the below equation :-
 
After doing the calculation of the above equation , the result will be like :-

Python Implementation of the above code :-

import matplotlib.pyplot as plt
from sklearn import linear_model
x_train= [[2],[4],[6],[8],[10],[12],[14],[16]]
y_train =[[4],[6],[7],[10],[12],[15],[17],[19]]
regr = linear_model.LinearRegression()
regr.fit(x_train,y_train)
y_output =regr.predict(22)
print(regr.predict(22))
Output :- [[ 25.64285714]]

Further Reading :- https://en.wikipedia.org/wiki/Simple_linear_regression

You can download the code from my github id :- https://github.com/sangam92/Machine-Learning-tutorials

Euclidean Distance using Python

As per Wikipedia , the Euclidean distance or Euclidean metric is the "ordinary" straight-line distance between two points in Euclidean space.

The euclidean distance between the point A and B will be the line segment joining the point AB.

In a simple 2 dimensional space  , euclidean distance can be calculated by the below formula:

Let P is a point  with co-ordinates as (p1,p2) and Q is another point with co-ordinates as (q1,q2).
Then , the Euclidean Distance PQ,.



Similarly , for a point in 3 dimensional space , with coordinates as P(p1,p2,p3) and Q(q1,q2,q3)
 the distance is,


Let us take a simple example to understand it :-

(Euclidean) distance between points (2, -1) and (-2, 2) is found like this:-

Python implementation of Euclidean Distnace using scipy:-

from scipy.spatial import distance
a = (1,2,3)
b = (4,5,6)
dst = distance.euclidean(a,b)
print(dst)

#output :-5.19615242271

Further reading :-

https://en.wikipedia.org/wiki/Euclidean_distance

https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.spatial.distance.euclidean.html

Classification versus Regression

“Machine Learning” is the new devil in the market. Every day we used to listen some fancy words and algorithms .We have heard that people are able to classify the animal images and able to predict the stock prices.

We need to understand few jargon related to ML.

 So , let us start our discussion with the topic Classification Vs Regression. 

Classification :-

The main goal of classification is to predict the target class (Yes/ No). .Suppose,  we want to know whether a student fails or pass. It is a polarity based algorithm.

Examples :- To find whether a mail is spam or not.

                     Whether an image belongs to a cat or a dog.

Types of Classification:-

Binary Classification :- When we have only two target class labels to predict.

Example :- Pass or Fail 

Multi-Class Classification :- When there are more than two class label to predict.

Example :- image classification problems where there are more than thousands classes(cat, dog, fish, car,…). 

Algorithms For Classification:-

•   KNN(K Nearest Neighbor)
•   SVC(Support Vector Classifier)
•  Decision Tree etc.

Regression:- In regression problems, we are trying to predict continuous valued output,.

Given a stock and to predict its value in next few months.

Algorithms For Regression :-

•  Linear Regression
•  SVR( Support Vector Regression) etc

Whenever we find machine learning problem first define whether we are dealing with a classification or regression problem and we can get to know that by analyzing the target variable (Y).