Spark - Basic Statistics

We have already gone through the tutorial on Measure of Central Tendency.Now we will do it’s implementation in Pyspark.We need to import statistics module from pyspark.mlib.stat.

Once the spark job is submitted , we will get the below output as the result.

The below code is available in my Github library :- https://github.com/sangam92/Spark_tutorials

Spark - Correlation

Correlation explains us the association between two or more variables .The movement of one variable will impact the movement of another variable .
It is normally used in situation where we want to explore how two variables are related with each other.

Types of Correlation :-
Correlation can be classified in several ways but in most generic way it is divided into three ways.

(i) Positive and Negatives
(ii) Linear and Non Linear
(iii) Simple,partial and multiple

Positive and Negative : - Correlation can be positive and negative .If both the variable are moving in the same direction , we can termed them as positive correlation else it can be termed as negative correlation.

Linear and Non Linear :- If the change in one variable is accompanied by change in another variable in a constant ratio, it is a case of linear correlation.On the other hand, if the amount of change in one variable does not follow a constant ratio with the change in another variable, it is a case of non-linear or curvilinear correlation.

Simple,partial and multiple :- If only two variables are involved in a study, then the correlation is said to be simple correlation.When three or more variables are involved in a study, then it is a problem of either partial or multiple correlation. In multiple correlation, three or more variables are studied simultaneously. But in partial correlation we consider only two variables influencing each other while the effect of other variable(s) is held constant.


Let us implement a simple python code :-

from pyspark import SparkContext,SparkConf

from pyspark.mllib.stat import Statistics

import numpy as np

conf=SparkConf().setAppName("test")

sc =SparkContext(conf=conf)

seriesX = sc.parallelize([1.0, 2.0, 3.0, 3.0, 5.0]) # a series

# seriesY must have the same number of partitions and cardinality as seriesX

seriesY = sc.parallelize([11.0, 22.0, 33.0, 33.0, 555.0])

# Compute the correlation using Pearson's method. Enter "spearman" for Spearman's method.

# If a method is not specified, Pearson's method will be used by default.

print("Correlation is: " + str(Statistics.corr(seriesX, seriesY, method="pearson")))

data = sc.parallelize(

[np.array([1.0, 10.0, 100.0]), np.array([2.0, 20.0, 200.0]), np.array([5.0, 33.0, 366.0])]) # an RDD of Vectors

# calculate the correlation matrix using Pearson's method. Use "spearman" for Spearman's method.

# If a method is not specified, Pearson's method will be used by default.

print(Statistics.corr(data, method="pearson"))

Spark - cogroup

Spark cogroup is a transformation that works with key,value pairRDD ’s and primarily use to group these RDD’s.

Syntax :- cogroup(dataset,[numpartitions])

when we have a data set of (k,v) and (k,w) , then the resulting output will be

(k,iterable(v),iterable(w)) tuples.

Let us take an example and try to understand it.

We will create two data sets.Here first data set will be batsman_rank.txt while the second dataset will be bowler_rank.txt.

We will write a simple code two apply cogroup on these datasets and will try to check the result.

Output :-

The cogroup is conceptually equivalent to full outer join. If an RDD does not have an element for a given key that is present in other RDD , then the corresponding iterable will be empty.

The code and dataset is available in my Git repository :-

https://github.com/sangam92/Spark_tutorials