Statistics - Skewness

What is a Symmetric data ?

A data set where the left and right hand sides of the distribution are roughly equal .In a histogram ,the tails of the left as well as the right part of the distribution are equally balanced.data Such kinds of data are often referred as symmetric data.

In below case if we calculate the Mean and Median both are approximately equal to 5.5

Mean =5.5

Median =5.5

If we see the tail on the left of the center value is almost equal to the value on the right side of the center.Skewness refers to the asymmetry of lack of symmetry in the frequency distribution.

However, a distribution which is asymmetrical is skewed .skewed can be both positive and negative.

Negative Skewness :- When a distribution is skewed to the left (red dashed curve), the tail on the curve's left-hand side is longer than the tail on the right-hand side, and the mean is less than the mode. This situation is also called negative skewness.

                                    Mean < Mode

Positive Skewness :- When a distribution is skewed to the right the tail on the curve's right-hand side is longer than the tail on the left-hand side, and the mean is greater than the mode. This situation is also called positive skewness.

                                    Mean > Mode

Tests of Skewness :-

1. The values of mean, median and mode do not coincide.

2. When the data are plotted on a graph they do not give the normal bell-

shaped form i.e. when cut along a vertical line through the centre the two

halves are not equal.

3.The sum of the positive deviations from the median is not equal to the sum

of the negative deviations.

4. Quartiles are not equidistant from the median.

5. Frequencies are not equally distributed at points of equal deviation from

the mode

Karl Pearson’s Measure :-

The formula for measuring skewness as given by Karl Pearson is as follows:

                     Skewness = Mean - Mode

Coefficient of Skewness = Mean – Mode / SD

Flume - Architecture

Flume is a highly reliable tool for aggregating and transporting large amount of streaming data such as log files,events from various sources to centralized data source.

Flume chiefly consist of three components :-

Source 

Channel  

Sink

Source: It accepts the data from the incoming streamline and stores the data in the channel.
Channel: Normally, the reading speed is faster than the writing speed. Thus, we need some buffer to match the read & write speed difference. Basically, the buffer acts as a intermediary storage that stores the data being transferred temporarily and therefore prevents data loss. Similarly, channel acts as the local storage or  a temporary storage between the source of data and persistent data in the HDFS.
Sink: Then, our last component i.e. Sink, collects the data from the channel and commits or writes the data in the HDFS permanently.

Advantages of Flume :-

• It is reliable, salable, fault tolerant and customizable for different sources and sinks.
• Flume provides a steady flow of data between read and write operations.
• Flume feed online streaming data from various sources like network traffic, social media, email messages, and log files into HDFS.
• Supports multiple data flow like multiple-hop, fan-in, and fan-out.

Disadvantages of Flume :-

• Flume has complex topology.
• It does not support for data replication.It does not guarantee 100% unique message delivery (duplicate messages might enter at any times).
  

Population Vs Sample

If population is a Pizza then sample is a slice of it.

Suppose the CEC of India has announced the election date , now every news channel will try to predict the result of the election . They will start conducting the opinion polls .But do they conduct opinion polls on complete population of India and take out this inference .

The answer is NO.

They will visit cities , towns and villages , interact with a set of people and find out their opinion about the incumbent government and opposition .They will ask a set of question to these people who are in this sample and later analyze this data and present us this detail.They predict the winner of the next election.

population denotes a large group consisting of elements having at least one common feature. The term is often contrasted with the sample, which is nothing but a part of the population that is so selected to represent the entire group.

In simple terms, population means the aggregate of all elements under study having one or more common characteristic, for example, all people living in India constitutes the population.

• The population of tax payers in India.
• The population of all workers working in the factory.

By the term sample, we mean a part of population chosen at random for participation in the study. The sample so selected should be such that it represent the population in all its characteristics, and it should be free from bias.

Statistics _ Introduction

STATISTICS involves only four major part :-

Let us take an example and try to understand the underlying meaning of the statistics.

Consider the census of 2011 conducted in India , many volunteers went from house to house in order to collect the data and this data was sent to CM chandramauli , the census commissioner .

Mr. Chandramauli and his colleague will start cooking this data so that they will come on the prime time .They sat for hours and started analyzing the data and finally come out with a report SRS Statistical Report 2011.

Since this report is so huge that it will cause problem to our politician and bureaucrats, they don’t have time to read the complete story .Then , Mr.Chandramauli will creates beautiful charts and graphs out of it .So , everyone will be happy . Chandramauli is on prime time.

Flume - Introduction

Flume is a highly reliable tool for aggregating and transporting large amount of streaming data such as log files,events from various sources to centralised data source.

What is streaming data ?

As per AWS , Streaming Data is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). Streaming data includes a wide variety of data such as log files generated by customers using your mobile or web applications, e commerce purchases, in-game player activity, information from social networks, financial trading floors, or geo spatial services, and telemetry from connected devices or instrumentation in data centers.

Basic Architecture Of Flume :-
 
The data from data sources like Twitter,Facebook and Web server are passed through flume and stored in a centralized data store like HDFS, HBASE.

But a question arises why PUT and HDFS -CopyFromLocal not efficient ?
⦁    'PUT' command can transfer one data at a time .However , the log data are generated at a much higher rate than the expected.
⦁    For PUT command the data need to be packaged and should be ready for the upload.but with the case of web server logs it is not possible at all.

Apache Flume is a tool for data ingestion in HDFS. It collects, aggregates and transports large amount of streaming data such as log files, events from various sources like network traffic, social media, email messages etc. to HDFS. Flume is a highly reliable & distributed.
The main idea behind the Flume’s design is to capture streaming data from various web servers to HDFS. It has simple and flexible architecture based on streaming data flows. It is fault-tolerant and provides reliability mechanism for Fault tolerance & failure recovery.