Understanding AST (Abstract Syntax Tree) in Spark

Understanding AST (Abstract Syntax Tree) in Spark

An Abstract Syntax Tree (AST) is a way to break down a query into a tree-like structure so that the system can process it more efficiently. This happens in Apache Spark, SQL databases (like Snowflake, PostgreSQL, MySQL), and other data processing engines.

1. AST in Apache Spark
When you write a PySpark DataFrame operation or an SQL query, Spark follows these steps:
Step 1: Parsing (Building the AST)
Spark reads the query and converts it into an AST, which represents its structure.

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("AST_Example").getOrCreate()
df = spark.read.csv("data.csv", header=True)
df_filtered = df.select("name", "age").filter(df["age"] > 30)
df_filtered.explain(mode="extended") # Show execution plan

Step 2: Logical Plan (Understanding the Query)
Spark converts the AST into a Logical Plan, which describes what needs to be done, without deciding how to do it.
Project [name, age]
├── Filter (age > 30)
├── Read data.csv

Step 3: Optimization (Making it Faster)
Spark improves the Logical Plan to make the query run faster and cheaper.
Some common optimizations:
Predicate Pushdown → Moves filters closer to data to reduce scanning.
Projection Pruning → Removes unnecessary columns.
Constant Folding → Simplifies calculations before execution (e.g., 5 + 10 is replaced with 15).

Step 4: Physical Plan (Deciding How to Execute)
Finally, Spark chooses the best way to run the query and creates a Physical Plan.
Example Physical Plan:
*(1) Project [name, age]
*(2) Filter (age#1 > 30)
*(3) Read CSV file

Why is AST Important?

Helps Spark and databases understand queries.

Optimizes performance before running the query.

Makes sure the query runs efficiently on large datasets.

Azure - databricks - storing vs registering a DF in Delta Lake

 

When to Use What? ✅ Use "Storing DataFrame as Delta Table" if: You don’t need SQL access and will only use Spark for reading/writing. The table is temporary or an intermediate dataset in a pipeline. You want to manually control schema enforcement and versioning. ✅ Use "Registering DataFrame in Delta Lake" if: You want the table available for SQL queries in Databricks or external BI tools. You need governance, access control, and metadata tracking. You plan to use Delta Sharing, Unity Catalog, or ADF for integration.

Azure : Azure Data Lake vs. Azure Blob Storage

 Azure Data Lake vs. Azure Blob Storage: Which One Should You Use?

Both Azure Data Lake Storage (ADLS) and Azure Blob Storage are powerful cloud storage solutions, but they serve different purposes. Let’s break it down:

 Purpose & Use Cases

ADLS: Built for big data & analytics, perfect for handling massive datasets and integrating with tools like Databricks, Synapse, and Spark.

Blob Storage: A general-purpose object store, great for storing images, videos, backups, and web content.

Storage Structure & Performance

ADLS: Uses a hierarchical namespace, meaning files and folders act like a traditional file system—this makes operations like renaming and deleting much faster!

Blob Storage: Uses a flat structure, treating everything as an object, which is simpler but less efficient for managing large-scale structured data.

Access & Security

ADLS: Offers POSIX-like ACLs (fine-grained permissions) alongside Azure RBAC (Role-Based Access Control).

Blob Storage: More flexible, allowing access via SAS tokens, RBAC, and anonymous public access if needed.

Cost Considerations

ADLS: Slightly more expensive because of its advanced capabilities.

Blob Storage: Cheaper and more cost-effective for basic storage needs.

Which One Should You Choose?

 Pick ADLS if you're dealing with big data, analytics, or machine learning workloads.

 Pick Blob Storage if you need scalable storage for backups, media files, and general object storage.

Spark - Constant Folding

 Constant Folding: Spark’s Hidden Efficiency Booster 🚀

Imagine this: you write a query in Spark that includes something like (2 + 3) * column_name. Now, wouldn’t it be smarter to compute (2 + 3) just once, rather than doing the math every single time the query is run? That’s exactly what constant folding does for you!

Spark’s Catalyst Optimizer recognizes constant expressions like (2 + 3), evaluates them during the query optimization phase, and replaces them with their computed value—in this case, 5. So, the query is transformed into SELECT 5 * column_name FROM table_name. Simple, efficient, and ready to blaze through execution! 🔥

This clever optimization reduces the computation Spark needs to perform when processing data, ensuring faster and more efficient query execution. It’s like giving your queries a little brainpower boost before they hit the big leagues.

Join Strategies in Apache Spark

🔹 Broadcast Hash Join

This join is ideal when one dataset is significantly smaller than the other. The smaller dataset is sent to all executor nodes, allowing each node to join its partitions of the larger dataset with the broadcasted data locally. Since no shuffle is required, this method is highly efficient for joins where one table is small enough to fit in memory.

🔹 Sort Merge Join

The default strategy when dealing with large datasets that cannot fit into memory. Both datasets are first sorted based on the join key, followed by a merge operation to match records efficiently. This approach requires shuffling for sorting, which can be computationally expensive. It is best used when neither table is small enough for broadcasting.

🔹 Shuffle Hash Join

Used when one dataset is smaller than the other, but not small enough for a broadcast join. The data is hashed and repartitioned based on the join keys, ensuring that matching records are processed within the same partition. This method is more efficient than Sort Merge Join but requires partitioning overhead. It is useful when working with medium-sized datasets.

🔹 Broadcast Nested Loop Join

This method is chosen when no other join strategy applies, often as a last resort. It broadcasts the smaller dataset and iterates through it for each row in the larger dataset, applying filtering conditions to retain only matching records. Since this approach does not rely on keys, it is commonly used in cross joins or joins with non-equi conditions where standard join algorithms are inefficient.

🔹 Cartesian Product Join

A cross join that generates all possible combinations between two datasets. Each row from the first dataset is paired with every row in the second dataset. This join strategy is explicitly used when a cross join is needed or when no join condition is provided. Since it produces an exponential number of rows, it is typically avoided for large datasets due to high computational cost.

🔹 Skew Join

Designed to handle data skew, where certain keys have a disproportionately large number of records. Spark detects skewed keys and splits their partitions into smaller chunks, distributing the workload evenly across the cluster. This ensures that the join operation runs efficiently without bottlenecking specific partitions. Skew join optimization is particularly useful when a few keys dominate the dataset, causing slow performance in standard joins.

✔ Broadcast joins are fastest when one table is small.

✔ Sort Merge Join is default for large, unsorted datasets.

✔ Shuffle Hash Join is effective for medium-sized tables.

✔ Broadcast Nested Loop Join is a fallback when other joins aren’t feasible.

✔ Cartesian Join should be used cautiously due to high computation.

✔ Skew Join prevents bottlenecks caused by uneven data distribution.

Example:

👉 Broadcast Hash Join – 🚀 When one table is small, broadcast it! Eliminates shuffle, boosts speed!

👉 Sort Merge Join – 📊 For large datasets. Sort both sides before merging. Expensive shuffle, but reliable.

👉 Shuffle Hash Join – 🔄 Medium-sized tables? Hash & partition them efficiently. Faster than Sort Merge.

👉 Broadcast Nested Loop Join – ⚠️ Last resort! When no keys exist, broadcast small table & loop through rows. Costly!

👉 Skew Join – 🎯 Fix data skew by splitting large keys into smaller partitions. No more stragglers!

 

Hadoop - What is a Job in Hadoop ?

In the field of computer science , a job just means a piece of program and the same rule applies to the Hadoop ecosystem as well. Here, the job essentially consists of a set of program or programs that needs to manipulate a certain piece of data.So, a job is usually associated with certain parameters like one important thing of a job is that it is going to be a set of programs. Here in the Hadoop ecosystem, it’s going to be a Map Reduce program typically written in Java. However, there are also non-Java based jobs which can be submitted into the Hadoop cluster which we shall discuss in detail later.

The important parameters which actually identify or make up a job is going to be the programs and the piece of data on which the program has to work on , which is going to be the input file path and the output file or the output directory to which the output of the program’s execution needs to be stored. So, a job in a Hadoop ecosystem is going to be a combination of all the three.

When we specifically speak about a job in terms of a Java MapReduce job, the programs are going to be submitted into the cluster in the form of a jar file which is going to be a packaging of all the classes which constitute the program which needs to execute on a particular set of data.

The two major features what Hadoop actually provides to the users.

The first important feature is a distributed file system which is called as the Hadoop Distributed File System or the HDFS. Basically, once you submitted into the Hadoop cluster it’s going to be residing on multiple data nodes where the original file would be divided into smaller pieces. So, this service actually is called as the Hadoop Distributed File System.

Second most important feature of Hadoop is that it provides a robust parallel processing framework which is called as a MapReduce Framework. So, once you submit a job into the Hadoop cluster, the program which is supposed to execute on a piece of data is now going to be running on multi machines wherever the data corresponding to the input file on which the Hadoop program is supposed to manipulate. So, the two core services or the two core features what Hadoop actually comes very handy is that the Hadoop Distributed File System and then the Parallel Processing Framework called as a MapReduce. p { margin-bottom: 0.25cm; line-height: 115% }

Hadoop - Cluster Set Up and RAID

The Hadoop cluster mainly resides in steel frame that resembles the modern day cup boards.The only difference between these rack and cupboard is that the rack does not have metal enclosure on all the sides like your traditional metal cupboards. Indeed, there would just like open cupboards where you would have holes on these pillar like structures in the racks for you to be able to bolt on the data nodes or basically the data node over here are just a chassis only machines.

A rack would be consisting a form like anywhere between 25 to 50 nodes and it’s not these racks just accommodate the data node. These racks can also accommodate the name node, the secondary name node or your job tracker.

It is just that is an open frame like structure where these chassis can be bolted upon and all of these machines within a rack will be able to communicate with the help of a switch at the speed with usually be around like 10 gigabytes per second and multiple racks would be able to communicate with the help of a multi-layer switch or an uplink switch which also does the functionality of that of a router.Usually the data transfer speed between the machine in the same rack is much higher than the data transfer speed between the machines across different racks.

Most of the data nodes would be having 2 hex core processor or 2 OctaCore processor meaning 8 Cores. There will be 2 CPU’s, each of them 8 cores and processing speed anywhere between 2.4 to 3.5 gigahertz CPU. the highest speed and the amount of RAM required typically varies according to the organizational needs.Usually your name node and Job tracker would be having higher RAM than that of your data nodes so the data node over here typically would be having anywhere between 50 to 500 GB of high speed RAM . And the frequency speed is somewhere around greater than 2000 megahertz and for the storage, the general formula like how much of hard drive storage you need for each data nodes is done by the rule of thumb that. If for every single core of CPU, you would be requiring at least 2 Terabyte of hard drive.

The question is why don’t we use RAID redundant array in expensive disc which has been since more than a decade? Why can't be used that in place of a distributed storage system as in the case of HDFS in Hadoop.

The reason being HDFS cluster do not benefit from using RAID for data nodes especially because your data nodes handle replication across several nodes which is a built in functionality of your HDFS and although RAID can actually be used for name node disc and secondary name node disc to just act as a backup against the fail over or the corruption of the meta data. Even if a disc fails in case of your Hadoop cluster your HDFS can continue to operate without the fail disc but that's not the case with the RAID, so due to these reasonsRAID is not the preferred choice out here in HDFS. p { margin-bottom: 0.25cm; line-height: 115% }