Spark - Correlation

Correlation explains us the association between two or more variables .The movement of one variable will impact the movement of another variable .
It is normally used in situation where we want to explore how two variables are related with each other.

Types of Correlation :-
Correlation can be classified in several ways but in most generic way it is divided into three ways.

(i) Positive and Negatives
(ii) Linear and Non Linear
(iii) Simple,partial and multiple

Positive and Negative : - Correlation can be positive and negative .If both the variable are moving in the same direction , we can termed them as positive correlation else it can be termed as negative correlation.

Linear and Non Linear :- If the change in one variable is accompanied by change in another variable in a constant ratio, it is a case of linear correlation.On the other hand, if the amount of change in one variable does not follow a constant ratio with the change in another variable, it is a case of non-linear or curvilinear correlation.

Simple,partial and multiple :- If only two variables are involved in a study, then the correlation is said to be simple correlation.When three or more variables are involved in a study, then it is a problem of either partial or multiple correlation. In multiple correlation, three or more variables are studied simultaneously. But in partial correlation we consider only two variables influencing each other while the effect of other variable(s) is held constant.


Let us implement a simple python code :-

from pyspark import SparkContext,SparkConf

from pyspark.mllib.stat import Statistics

import numpy as np

conf=SparkConf().setAppName("test")

sc =SparkContext(conf=conf)

seriesX = sc.parallelize([1.0, 2.0, 3.0, 3.0, 5.0]) # a series

# seriesY must have the same number of partitions and cardinality as seriesX

seriesY = sc.parallelize([11.0, 22.0, 33.0, 33.0, 555.0])

# Compute the correlation using Pearson's method. Enter "spearman" for Spearman's method.

# If a method is not specified, Pearson's method will be used by default.

print("Correlation is: " + str(Statistics.corr(seriesX, seriesY, method="pearson")))

data = sc.parallelize(

[np.array([1.0, 10.0, 100.0]), np.array([2.0, 20.0, 200.0]), np.array([5.0, 33.0, 366.0])]) # an RDD of Vectors

# calculate the correlation matrix using Pearson's method. Use "spearman" for Spearman's method.

# If a method is not specified, Pearson's method will be used by default.

print(Statistics.corr(data, method="pearson"))

Neural Networks - An Introduction

Neural Networks in computer system are quite analogues to what the neurons we have in our brain. Our brain that is the collection of millions of neurons can be considered as the most advanced system. The decision taken by brain are impeccable and are accompanied with a lot of decision making that involved a lot of permutation and combination.

Scientist were working day and night to produce an AI enabled system that can solve majority of problem. Few AI system were developed also with the capability to solve the problem in a formal way but do not have a broader prospect. The IBM Deep Blue was one of such invention that defeated World Chess Champion Garry Kasprov.

Initially the AI system based their knowledge on some set of rules and regulation. Such kind of system were not so robust and had certain limitation. The human need to feed every possible situation which a cumbersome job was. Such approach was called Knowledge Base approach. The scientist devised a new methodology which does not require any specific rules and work according to the raw data fed into the system.

The scientist works upon the machine learning approaches and have many successes, one of them was logistic regression which was able to predict the cesarean delivery by feeding the system with certain inputs which was called feature extraction. Similarly, Naïve Bayes was used to classify the email as Spam and Non-Spam. The feature extraction was quite a tough job and require lots of human effort and time, moreover it requires a top-level domain expertise.

The ground-breaking innovation came into the year 1960 when Frank Rosenblatt has discovered an artificial neuron called Perceptron. Later, the evolution take place in such a perceptron and lead the way for the Multilayer Perceptron (MLP).

We will cover about Perceptron and Multilayer Perceptron in our next blog post.

Python - Keywords

Python Keywords

Keywords are the reserved words and we cannot use a keyword as variable name, function name or an Identifier. Almost 33 keywords are available in Python which is quite lesser than other languages like Java, C++ etc.

Method	Description
and	A logical operator
as	To create an alias
assert	For debugging
break	To break out of a loop
class	To define a class
continue	To continue to the next iteration of a loop
def	To define a function
del	To delete an object
elif	Used in conditional statements, same as else if
else	Used in conditional statements
except	Used with exceptions, what to do when an exception occurs
False	Boolean value, result of comparison operations
finally	Used with exceptions, a block of code that will be executed no matter if there is an exception or not
for	To create a for loop
from	To import specific parts of a module
global	To declare a global variable
if	To make a conditional statement
import	To import a module
in	To check if a value is present in a list, tuple, etc.
is	To test if two variables are equal
lambda	To create an anonymous function
None	Represents a null value
nonlocal	To declare a non-local variable
not	A logical operator
or	A logical operator
pass	A null statement, a statement that will do nothing
raise	To raise an exception
return	To exit a function and return a value
True	Boolean value, result of comparison operations
try	To make a try...except statement
while	To create a while loop
with	Used to simplify exception handling
yield	To end a function, returns a generator

---

                                                  List of Key Words

There are some basic rules that need to be followed with the keywords.No identifiers should have the same name as any of the keywords.Names that begin with __ underscore should not be used.

Python - Datatype

Every thing in python has a datatype. There are various types of datatypes in python. Broadly they are divided into numbers and strings. Let's have a look at both.

Python Numbers

Integers, float and complex numbers fall under python’s number category. They are represented in python in following ways:

Int

Integers can be of any length, it is only limited by the memory available.

Float

A floating point number is accurate up to 15 decimal places. Integer and floating points are separated by decimal points.

Complex

Complex numbers are written in the form, x + yj, where x is the real part and y is the imaginary part.

How would you know a data type of a variable?

type() function to know which class a variable or a value belongs.

type at work

Python Strings

String is sequence of unicode characters. We can use single quotes or double quotes to represent strings. Multiline strings can be denoted using triple quotes''' or """.

Conversion of data type

We can convert between different data types by using different type conversion functions like int(), float(), str() etc.

Python - Introduction

As per 2018 survey conducted by Stackoverflow.com , Python is the 7^th most sought language and most wanted technology.The question is why python is so demanding ? Here are the few points that i have observed and gathered through books and internet.

Python is hot

According to research by Dice Python is also one of the hottest skills to have and the most popular programming language in the world based on the Popularity of Programming Language Index.

Python is interpreted

Most of the languages that are in use are compiled one , it means the language are first converted into machine code , the language of our processors before it can run.

However, python is an interpreted language and the codes are not converted

into machine code.

Python is free

The Python interpreter is developed under an OSI-approved open-source license, making it free to install, use, and distribute, even for commercial purposes.

Python is portable

Python code is interpreted and not compiled into native machine instructions, code written for one platform will work on any other platform that has the Python interpreter installed.

Python is simple

Python 3 has 33 keywords, and Python 2 has 31. By contrast, C++ has 62, Java has 53, and Visual Basic has more than 120, though these latter examples probably vary somewhat by implementation or dialect.

Python - Match vs Search

Regular Expression are very useful and can be used by developers in numerous of ways. It proved handy when we are encountering with the data having some sort of patterns. There are numerous of operation that can be done using regular expression. The re module can be used to import the regular expression module.

Match Operation in Python :-   Python match operation will look for the first matching string in a given set of data. If the searched data is not available at the first position, then it will return NONE.

Example :-  We will check what will be the output when we look for the data set that is available at the first place and when it is not at the first index position.

Python Code :- (when at the first position)

import re

data = "this is good"

res_match = re.match("this",data)

print("The result set when it is at first place",res_match)

Output :-

The result set when it is at first place <re.Match object; span=(0, 4), match='this'>

Python Code :- (when not at first position)

import re

data = "this is good"

res_mat_pos = re.match("good",data)

print("The result set when it is not in first place",res_mat_pos)

Output :- 

The result set when it is not in first place None.

Search Operation in Python :- Search operation works in the similar way as match works , except that it can search the word that are in different index position.

Sample Python Code :-

import re

data = "this is good"

result = re.search("good",data)

print("This is the result for the search",result)

Output :-

This is the result for the search <re.Match object; span=(2, 4), match='is'>

Thanks !!!

Sqoop - Eval

Eval is normally used to run simple sql queries against the database server.It will also preview the result on the console level.Eval can be used to evaluate any type of query like DDL/DML.

EVAL is only to check the database connection and also to preview the small set of data.It will provide the user to test the simple queries.

Sqoop Syntax :-

sqoop --eval {generic-args} {eval args}

Generic args :-

Argument	Description
–connect <jdbc-uri>	Specify JDBC connect string
–connection-manager <class-name>	Specify connection manager class to use
–driver <class-name>	Manually specify JDBC driver class to use
–hadoop-mapred-home <dir>	Override $HADOOP_MAPRED_HOME
–help	Print usage instructions
–password-file	Set path for a file containing the authentication password
-P	Read password from console
–password <password>	Set authentication password
–username <username>	Set authentication username
–verbose	Print more information while working
–connection-param-file <filename>	Optional properties file that provides connection parameters
–relaxed-isolation	Set connection transaction isolation to read uncommitted for the mappers.

Eval Args :-

e ,query Execute statement in SQL.

query examples :-

sqoop eval \
–connect jdbc:mysql://localhost/test \
–username root \
–query “SELECT * FROM sqoop_test ”