Welcome to Day 7 of your Spark Mastery journey!Today is one of the most practical days because joins, unions, and aggregations are used in almost every pipeline you will ever build — be it feature engineering, building fact tables, or aggregating transactional data.Let’s master the fundamentals with clarity and real-world examples.🌟 1. Joins in PySpark — The Heart of ETL PipelinesA join merges two DataFrames based on keys, similar to SQL.df.join(df2, df.id == df2.id, “inner”) Join on same column name:df.join(df2, [“id”], “left”) 🔹 Join Type - Meaning inner - Matching rows left - All rows from left, match from right right - All rows from right full - All rows from both left_anti - Rows in left NOT in right left_semi - Rows in left WHERE match exists in right cross Cartesian productleft_semi = existence check left_anti = anti-join / unmatched rows🌟 2. Union — Stack DataFrames VerticallyUnion (same schema, same order)Why important? Because in real projects you combine:🌟 3. GroupBy + Aggregation — Business Logic LayerThis is how reports, fact tables, metrics are built.df.groupBy(“dept”).agg( sum(“salary”).alias(“total_salary”), avg(“age”).alias(“avg_age”) ) df.select(count(“id”)) df.select(countDistinct(“id”)) 🔹 approx_count_distinct (faster!)df.select(approx_count_distinct(“id”)) 🌟 4. Real ETL Example — Sales Aggregationdf_joined = sales.join(products, “product_id”, “left”) df_agg = df_joined.groupBy(“category”).agg( sum(“amount”).alias(“total_revenue”), count(“*”).alias(“transactions”) ) This is EXACTLY how business dashboards are built.🌟 5. Join Performance OptimizationUse Broadcast Join for small lookup tables:df.join(broadcast(df_small), “id”) Why?Avoids shuffle → runs much faster.broadcast join optimizationFollow for more such content. Let me know if I missed anything in comments. Thank you!!