Why Spark?
Traditional MapReduce processed data in sequential disk-bound stages. Spark stores intermediate results in memory, making iterative algorithms (like machine learning) up to 100x faster.
Core Abstractions
- RDD (Resilient Distributed Dataset): The low-level, fault-tolerant distributed collection.
- DataFrame: A higher-level, schema-aware abstraction similar to a SQL table — the recommended API.
- Dataset: A type-safe version of DataFrame for JVM languages.
# Reading and aggregating data with PySparkndf = spark.read.parquet("s3://bucket/events/")nresult = df.groupBy("country").agg({"revenue": "sum"})nresult.write.mode("overwrite").parquet("s3://bucket/output/")