Spark - DataFrames 2

Spark DataFrames 2

In the previous blog , we have gone through the basic of dataframes and also created a dataframe from the sample test.csv file.In this post, we go ahead and will see how to do different operations on dataframes.

We will see the complete program and find out the different operations that can be done on the dataframes.

We will start the program by reading the csv file and try to display the count of  rows in the dataframes.

After the submitting the spark jobs , we will get the below output.

  Display the number of the columns and their name.

describe operation is use to calculate the summary statistics of numerical column(s) in DataFrame. If we don’t specify the name of columns it will calculate summary statistics for all numerical columns present in DataFrame.

 

 

     









Selecting specific columns in the dataframes .

The specific columns in a dataframe can be selected by invoking the dataframe and specifying the required columns.

 

 

         







  Displaying the statistics of a specific column.

   

Spark - DataFrames 1

DataFrames are the combination of rows and columns and are quite similar to the tables in sql. They have combination of rows and columns Rows can have a variety of data formats (heterogeneous), whereas a column can have data of the same data type (homogeneous).

DataFrames are built in order to make the spark understand the schema of the dataframes.This can help the spark to optimize it’s execution plan.

It will further help in the slicing and dicing of the data. It will help where we have a lot of columns in the data source and we need very few columns out of it.

These dataframes are able to support various kinds of data sources like csv , Json, hive tables etc.


Let us read a csv and create the data frame out of it.


We can further check the schema of the dataframe which tell us the column by column detail of the data.

Let us some other stuff like counting of the data .

We will go through some more data frames commands in our next blog post.

Thanks !!!

Kafka - Introduction

Apache Kafka is a data extraction tool for real-time streaming data. It is also distributed, partitioned, and a replicated system for extracting real time streaming data extraction. Kafka was developed and open sourced by LinkedIn in 2011. It is getting widely adopted by companies for efficient real time streaming of data.

Kafka is being used by tens of thousands of organizations, including over a third of the Fortune 500 companies. It’s among the fastest growing open source projects and has spawned an immense ecosystem around it. It’s at the heart of a movement towards managing and processing streams of data.

Kafka is often compared to a couple of existing technology categories: enterprise messaging systems, big data systems like Hadoop, and data integration or ETL tools. Each of these comparisons has some validity but also falls a little short.

Kafka is like a messaging system in that it lets you publish and subscribe to streams of messages. In this way, it is similar to products like ActiveMQ, RabbitMQ, IBM’sMQSeries, and other products. But even with these similarities, Kafka has a number of core differences from traditional messaging systems that make it another kind of animal entirely.

Here are the big three differences: first, it works as a modern distributed system that runs as a cluster and can scale to handle all the applications in even the most massive of companies. Rather than running dozens of individual messaging brokers, hand wired to different apps, this lets you have a central platform that can scale elastically to handle all the streams of data in a company. 

Secondly, Kafka is a true storage system built to store data for as long as you might like. This has huge advantages in using it as a connecting layer as it provides real delivery guarantees—its data is replicated, persistent, and can be kept around as long as we like