RDD Creation using Pyspark

In the last post ,we have understood how RDD works in Apache spark.Now we focus on how to create RDD in Apache Spark.

So ,Let us start our discussion on creating a RDD with a simple text file.

so , our first code is below:-.

from pyspark import SparkContext,SparkConf
conf =SparkConf().setAppName("Count").setMaster("local")
sc =SparkContext(conf=conf)
rdd_create = sc.textFile('test.txt')
rdd_first = rdd_create.first()
print(rdd_first)


Let us decode this piece of code

Line 1 : We need to import the sparkcontext and sparkconfiguration packages.

Line 2:  The application name is count and it can be given any name .it is just need to find the application program on the cluster .The  application is set here local as we are running this piece of code on our local machine.

Line 3 : The SparkContext has been assigned to the variable sc .

Line 4: We read the file test.txt  and in simple words it is a pointer to the file (We need to keep in mind the Lazy evaluation technique of Spark)

Line 5: Here ,we read the first line of the text file and store it in the variable rdd_count

Line 6: The output has been displayed here.

You can find the code in my github id :- https://github.com/sangam92/Spark_tutorials

filter() in python

filter method filter out the iterable with the help of a given function and return the result in the form of a iter(list,set).

Basic Syntax :- filte(func,iter)

Example :-
a = [1,2,3,4,5]
c = list(filter(lambda x : x%2==0 ,a))
print(c)

It filter out the all those elements that are divisble by 2 from the above iterable a.
Hence  the output will be [2,4]

map() in python

The map() function applies a given function to each item of an iterable (list, tuple etc.) and returns a list of the results.

Basic Syntax :- map(func,iter)

Note :- we can provide more than one iterable in the map function.

Example :-  a = [1,2,3,4,5]
c =list(map(lambda x : x+1 ,a))
print(c)
The output is [2, 3, 4, 5, 6]

Example :- More than one iterable

a = [1,2,3,4,5]
b=[1,2,3,4,5]
c =list(map(lambda x,y : x+y ,a,b))
print(c)

#The output is [2, 4, 6, 8, 10]

lambda in Python

So ,what is lambda function ? In every python code , we come across this devil .Let us crack this nut .
lambda is an anonymous function or throw away function, without a name .It is normally used when we do not want a function twice in the same program.

Basic Syntax :

    lambda arguments : expression

Note :- lambda can contain 'n'  number of arguments but it can have only one expression.

Examples  :- c = lambda x :x +1
        print(c(92))

The output will be 93.

Example :- a = [(1, 2), (4, 1), (9, 10), (13, -3),(-2,5)]
a.sort(key=lambda x: x[0])
print(a)

Output: [(13, -3), (4, 1), (1, 2), (9, 10)]

During the preprocessing of data , The lambda function is used in a wide range.Normally to map the delimiter in the csv files.

RDD In Apache Spark

RDD is considered as the core and heart of the Spark.For any program to run in the Spark, the RDD is there inside it .It can be created , transformed or called in any operation .Spark automatically distributes the data set within RDD across the cluster.

RDD is an immutable distributed collection of object  and is fault tolerant , can be operated in parallel.
RDD can be created in two different ways :-

1.) Parallelizing existing object like list or set in the driver program.
2.) Referencing an external dataset like HDFS, textfile,csv fille etc.

Once the RDD is created, two important processes can take place :-

1.) Transformation :- Generating one RDD from another RDD , can be commonly used in the    filtering process.
2.) Action :- Compute a result on the basis of a RDD like count, first.

Lazy Evaluation in Spark :-

Spark works on the concept of Lazy evaluation .It means that the RDD cannot be created until spark finds an action.Initially it seems weird but in handling Big data ,this concept is quite useful. All transformations in Spark are lazy, in that they do not compute their results right away. Instead, they just remember the transformations applied to some base data set (e.g. a file). The transformations are only computed when an action requires a result to be returned to the driver program. This design enables Spark to run more efficiently.
By default, each transformed RDD may be recomputed each time you run an action on it. However, you may also persist an RDD in memory using the persist (or cache) method, in which case Spark will keep the elements around on the cluster for much faster access the next time you query it. There is also support for persisting RDDs on disk, or replicated across multiple nodes.

Standalone Application in Spark

Spark can be run interactively as well as in the standalone program .The major difference  between the interactive shell and standalone application is that we need to define SparkContext in the case of standalone but in the interactive shell it is available through sc variable.
We will learn the spark through python implementation.In Python, you simply write applications as Python scripts, but you must run them using the bin/spark-submit script included in Spark. The spark-submit script includes the Spark dependencies for us in Python. This script sets up the environment for Spark’s Python API to function.

Intialization in Standalone Program;-

from pyspark import SparkContext,SparkConf
conf =SparkConf().setAppName("Count").setMaster("local")
sc =SparkContext(conf=conf)

The Line 1 will import all the Spark API for the Python.
Then in Line 2 , we are giving a name to identify this program on the cluster as count and which tells Spark how to connect to a cluster. local is a special value that runs Spark on one thread on the local machine, without connecting to a cluster.
In Line we initialize the SparkContext with sc Variable.

Apache Spark Core

In spark , the architecture is mainly divided between the Driver and Executor node.The Driver node is the part of the program where the main program get executed.The Driver take the main program and distribute the data sets into the worker nodes and also the operation that the worker nodes are suppose to do.
In layman words, the driver is the manager and executor are the developer .Driver distributes the resources and tasks to be performed by each developer.
The driver program access spark through sparkcontext  object which is connected to a computing cluster.


In spark shell ,you can connect to  the sparkcontext via sc variable .

If we are running our program on our local machine ,then it will run on a single cluster .But when we run the same program on cluster , different part of the program is run on different cluster.