Persist Vs Cache in Spark

Whenever there is a RDD creation ,it is accompanied with a lot of query fired by us on that RDD.We can re frame like that a number of action take place on that RDD.

What can be done in such scenario ?

Spark can save the partial result and reuse it a number of times.This can reduce the extra process of creating the RDD again and again.
By default , Apache Spark do not persist the data and we need to instruct the spark that we need persistence.In fact Persisting is one of the optimization technique .

When you persist an RDD, each node stores any partitions of it that it computes in memory and reuses them in other actions on that dataset (or datasets derived from it). This allows future actions to be much faster (often by more than 10x).

How to Persist the RDD ?

from pyspark import SparkContext,SparkConf
conf =SparkConf().setAppName("Count").setMaster("local")
sc =SparkContext(conf=conf)
rdd_create = sc.textFile('test.txt')
rdd_create.persist
rdd_count = rdd_create.first()
rdd_count2= rdd_create.count()
print('The first line is' ,rdd_count)
print('The count of lines is ',rdd_count2)

Difference between Persist and cache:-

Cache can use only the default memory while in persist we can use different memory management technique.

Persist:-


 Caching :-










We will learn more about the memory management in our different post.

Further Reading :-https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html

Code Download :- https://github.com/sangam92/Spark_tutorials

RDD Creation using Pyspark

In the last post ,we have understood how RDD works in Apache spark.Now we focus on how to create RDD in Apache Spark.

So ,Let us start our discussion on creating a RDD with a simple text file.

so , our first code is below:-.

from pyspark import SparkContext,SparkConf
conf =SparkConf().setAppName("Count").setMaster("local")
sc =SparkContext(conf=conf)
rdd_create = sc.textFile('test.txt')
rdd_first = rdd_create.first()
print(rdd_first)


Let us decode this piece of code

Line 1 : We need to import the sparkcontext and sparkconfiguration packages.

Line 2:  The application name is count and it can be given any name .it is just need to find the application program on the cluster .The  application is set here local as we are running this piece of code on our local machine.

Line 3 : The SparkContext has been assigned to the variable sc .

Line 4: We read the file test.txt  and in simple words it is a pointer to the file (We need to keep in mind the Lazy evaluation technique of Spark)

Line 5: Here ,we read the first line of the text file and store it in the variable rdd_count

Line 6: The output has been displayed here.

You can find the code in my github id :- https://github.com/sangam92/Spark_tutorials

filter() in python

filter method filter out the iterable with the help of a given function and return the result in the form of a iter(list,set).

Basic Syntax :- filte(func,iter)

Example :-
a = [1,2,3,4,5]
c = list(filter(lambda x : x%2==0 ,a))
print(c)

It filter out the all those elements that are divisble by 2 from the above iterable a.
Hence  the output will be [2,4]

map() in python

The map() function applies a given function to each item of an iterable (list, tuple etc.) and returns a list of the results.

Basic Syntax :- map(func,iter)

Note :- we can provide more than one iterable in the map function.

Example :-  a = [1,2,3,4,5]
c =list(map(lambda x : x+1 ,a))
print(c)
The output is [2, 3, 4, 5, 6]

Example :- More than one iterable

a = [1,2,3,4,5]
b=[1,2,3,4,5]
c =list(map(lambda x,y : x+y ,a,b))
print(c)

#The output is [2, 4, 6, 8, 10]

lambda in Python

So ,what is lambda function ? In every python code , we come across this devil .Let us crack this nut .
lambda is an anonymous function or throw away function, without a name .It is normally used when we do not want a function twice in the same program.

Basic Syntax :

    lambda arguments : expression

Note :- lambda can contain 'n'  number of arguments but it can have only one expression.

Examples  :- c = lambda x :x +1
        print(c(92))

The output will be 93.

Example :- a = [(1, 2), (4, 1), (9, 10), (13, -3),(-2,5)]
a.sort(key=lambda x: x[0])
print(a)

Output: [(13, -3), (4, 1), (1, 2), (9, 10)]

During the preprocessing of data , The lambda function is used in a wide range.Normally to map the delimiter in the csv files.