Spark - Reading JSON File

We have understood the spark capability to read the CSV file in our previous post. In this blog post , we will learn the how spark reads the JSON files and convert it into data frames.

Step 1:- We will import all the necessary packages and do the required configuration which is almost common in most of the Spark Program.

from pyspark import SparkConf,SparkContext
from pyspark.sql import SQLContext
conf = SparkConf().setAppName("read_json")
sc = SparkContext(conf=conf)
sqlcontext = SQLContext(sc)

Step 2:- We will try to read the JSON file with the command read.json

read_json = sqlcontext.read.json('/home/hduser/sangam/employee.json')

Step 3:- We can display the records and schemas .

read_json.show()
read_json.printSchema()

Step 4:- We can even select the columns which we like to display.

read_json.select("id","age").show()

Once the JSON file is converted into dataframes , all the operation will be similar to what we did it for the dataframes.

Complete Code Snippet :-






Output :- 








The complete code file and data file is available in my GitHub repository.

https://github.com/sangam92/Spark_tutorials

Python -SwapCase

In python , we have a functionality by which we can swap the cases from upper to lower and vice versa. This can be achieved by the function called swapcase .

Example :- The input that will be provided here will get converted into

the alternative case

data = input()

print(data.swapcase())

We can convert the case with writing a function also.

Example :-

def swap_case(s):

result1=""

for i in s:

if i.isupper():

newletter= i.lower()

result1 += ''.join(newletter)

else:

newletter=i.upper()

result1 +=''.join(newletter)

return result1

if __name__ == '__main__':

s = input()

result = swap_case(s)

print(result)

Statistics – Data and Type of Data Sources

We have gone through the introduction of statistics in our previous blogs.In this blog, we will learn about the types of data and data sources that are used to generate this statistics.

Data is broadly classified into two main parts :-

• Quantitative Data
• Qualitative Data

Quantitative data are the data that can quantified in definite units of measurements .The Quantitative data are further divided into two more categories :

• Continuous Data
• Discrete Data

Continuous data can take any values on the line segment It can include data like height , weight , Temperature etc. The continuous data can have the highest degree of precision. They can be represented on a number line .

Discrete  are the one whose outcome is measured in fixed numbers . For example :- No of Student in the class, How many matches played by Virat Kohli .

As per wiki , Qualitative data is a categorical measurement expressed not in terms of numbers, but rather by means of a natural language description. In statistics, it is often used interchangeably with "categorical" data.

Example :- Favorite Color = ‘Blue’

Height = ‘Tall’

Qualitative data are further divided into two parts :-

• Nominal Data
• Ordinal Data

Nominal data can be simply be called “labels.”.Nominal data simply names something without assigning it to an order in relation to other numbered objects or pieces of data. An example of nominal data might be a "pass" or "fail" , Male or Female etc.

Ordinal data, unlike nominal data, involves some order; ordinal numbers stand in relation to each other in a ranked fashion. Example :- How do you feel today ? Very happy , happy ,ok , Unhappy , Very Unhappy . Here , we know that happy is better than Inhappy but we can not quantify it , like how much is the difference .

Data Sources are of two types :-

Primary Data :- Those data which do not already exist in any form,and thus

have to be collected for the first time from the primary source(s). Example :- Collecting the data for the census.

Secondary Data :- 

Such data are already exist in some form .Example :- Using the Census data to find how many women are employed in India .

Spark - Saving Dataframes

In my last blog, we have gone through the dataframes and some of their operations.we should note that we need to save the dataframes for further operations.

In this blog post , we will understand how to save the dataframes (Generic/Manual) form.

Generic Load/Save function

The default data format that is used in loading and saving the data source is paraquet.we can save the dataframe in parquet format by giving the dataframe name. Let us check the code for the same.

#saving the dataframes in the default location

read_file.select("name","age").write.save("dataframe_save.parquet",format=”parquet”)

We can change the default settings and can save the dataframes in other format like csv.

Let us start with our previous code that we have written for spark dataframes2. The code is available in my github repository .

Code Snippet :

from pyspark import SparkConf,SparkContext

from pyspark.sql import SQLContext

conf=SparkConf().setAppName("dataframe")

sc=SparkContext(conf=conf)

sqlcontext= SQLContext(sc)

read_file=sqlcontext.read.csv('/home/hduser/sangam/test.csv',header='true')

read_file.show()

print("The number of rows in the file are ",read_file.count())

read_file.head(2)

#Below command describe the no of columns in the dataframe and the respective columns.

print("no of columns and name of the columns",len(read_file.columns),read_file.columns)

#provides the complete statistics of the numerical columns available in the dataframe

read_file.describe().show()

#Provides the statistics of a particular column

read_file.describe('salary').show()

#Select specific column from the dataframe

read_file.select('salary','age').show()

#saving the dataframes in the default location

read_file.select("name","age").write.save("dataframe_save.csv",format="csv")

After submitting the code , we can get the output in our default location in a directory called dataframe_save.csv

once we enter the directory , we will get the file “part-00000-e90b751f-b7b9-4093-93b0-b014ef2012a8.csv”

All the related code is available in my github repository :-https://github.com/sangam92/Spark_tutorials

Python - Word Count Problem

A word count problem is one of the basic program that we come across in python.Being a programmer , we should be able to understand such kind of program and able to execute it.

Step 1 : Create the dataset , here it is done using the list.

words = ['A','boy','is','a','the','boy','is','the','the','india']

Step 2: Create an empty dictionary

wordcount = {}

Step 3: Traverse the loop and check for the word in list whether it is available in wordcount dictionary

for word in words:

if word not in wordcount:

wordcount[word] =1

else:

wordcount[word] +=1

Step 4 : Print the value using key,value pair

for k,v in wordcount.items():

print(k,v)

Complete code snippet

words = ['A','boy','is','a','the','boy','is','the','the','india']

wordcount = {}

for word in words:

if word not in wordcount:

wordcount[word] =1

else:

wordcount[word] +=1

for k,v in wordcount.items():

print(k,v)

Output :

you can find these codes in my github repository :- https://github.com/sangam92/python_tutorials

Four steps to create our first graph using matplotlib

In the field of data science, it is mandatory for us to have an understanding of the graph and we should know how to create one using the python. The idea of this blog is to make you familiar with matplotlib and  help you to  create your first graph.

Python provides us with a library called matplotlib to do the same. It is a plotting library used for 2D graphics in python programming language.

With the help of matplotlib , we can create different types of graphs such as:-

    • Bar Graph
    • Histogram
    • Scatter Plot
    • Area Plot
    • Pie Plot
we will create our first graph using the matplotlib and will move to other graphs in our upcoming blogs.

Step 1:- we need to import the matplotlib library in our python editor.
 

#importing the matplot library

import matplotlib.pyplot as pt

Step 2 :- we need to create the data sets for both the x as well as y axis.

#Creating the data

x = [2,4,6,8,10]

y = [1,2,3,4,5]

Step 3 :- Plotting the data on x as well as y axis .

#plotting the data on x and Y axis

pt.plot(x,y,label='linear')

Step 4 :- Displaying the graph on Python console .

#Displaying the graph

pt.show()

Snippet of the complete program :-

#!/usr/bin/env python3

# -*- coding: utf-8 -*-

"""

Created on Wed Nov 28 23:22:00 2018

@author: sangam

"""

#importing the matplot library

import matplotlib.pyplot as pt

#Creating the data

x = [2,4,6,8,10]

y = [1,2,3,4,5]

#plotting the data on x and Y axis

pt.plot(x,y,label='linear')

#Displaying the graph

pt.show()

Output :-