Compare SparkCore and SparkSQL and 4 ways we use SparkSQL in production

SparkCore && SparkSQL - English Version

Introduction

Why Not Hive?

1. Performance Issues
• HiveQL translates queries into MapReduce jobs, which are disk-based. This makes it very slow.
• MapReduce is inefficient compared to in-memory computation.

2. Limited Functionality
• Some data processing tasks are hard to achieve using HiveQL alone. Developers must write UDFs (User Defined Functions), which increases complexity.

3. Structured Data Only
• Hive only supports structured data (Schema). It cannot handle semi-structured or unstructured data.

Why Use Spark?

1. High Performance
• Spark processes data in memory, making it 10 times faster than disk-based MapReduce.

2. Functional Programming with RDDs
• SparkCore provides RDD (Resilient Distributed Dataset) operations. This functional programming model allows flexible data transformations.

3. SQL Support with SparkSQL
• SparkSQL provides an SQL API, similar to HiveQL, but with much better performance.
• SparkSQL also supports DSL functions and RDD-based transformations, making it more flexible.

Understanding SparkCore and SparkSQL

SparkCore 【Call API function to process data: O-Object, O-Function】

• Works like MapReduce, using RDD transformations to process data.

• Use case: Often used to clean semi-structured data and convert it into structured data before analysis.

• Example: ETL (Extract, Transform, Load) processing.

from pyspark import SparkContext

# Initialize SparkContext
sc = SparkContext("local", "SparkCoreExample")

# Create an RDD from a list
data = ["Spark", "Hadoop", "Big Data", "Machine Learning"]
rdd = sc.parallelize(data)

# Transform the RDD by mapping each element to its length
length_rdd = rdd.map(lambda word: (word, len(word)))

# Collect and print the results
print(length_rdd.collect())

SparkSQL 【O-tables】- What we use most in real world

• Works like HiveSQL, using SQL queries for large-scale data analysis, and computation.

• Use case: Performing analytics and batch processing on structured data.

• Development methods:
1. DSL (Functional API with RDDs)
2. SQL (Using SQL queries on tables)

from pyspark.sql import SparkSession

# Initialize SparkSession
spark = SparkSession.builder.appName("SparkSQLExample").getOrCreate()

# Create a DataFrame from sample data
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35)]
df = spark.createDataFrame(data, ["Name", "Age"])

# Register DataFrame as a temporary SQL table
df.createOrReplaceTempView("people")

# Run SQL query
result = spark.sql("SELECT Name, Age FROM people WHERE Age > 28")

# Show results
result.show()

Conclusion

4 ways we use SparkSQL in production

1. Python/Jar: Using spark-submit

Usage Scenario: ETL (Extract, Transform, Load)

• Python: Submit a Spark job developed in Python.

• Java: Submit a compiled JAR file to Spark.

spark-submit --master yarn my_spark_job.py

spark-submit --class com.example.MyApp --master yarn my_app.jar

2. ThriftServer: Submitting SQL Only (Similar to Hive's HiveServer2)

Common Ways to Submit SQL

a) PyHive (Most Common in Production)

from pyhive import hive

conn = hive.Connection(host='spark-thriftserver-host', port=10000, database='default')
cursor = conn.cursor()
cursor.execute("SELECT * FROM my_table LIMIT 10")
for result in cursor.fetchall():
    print(result)

b) spark-sql -f (For Automated SQL Execution)

spark-sql -f my_query.sql

c) Beeline (Typically Used for Testing)

beeline -u jdbc:hive2://spark-thriftserver-host:10000/default

d) JDBC (Rarely Used)

Connection conn = DriverManager.getConnection("jdbc:hive2://spark-thriftserver-host:10000/default", "user", "password");
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT * FROM my_table");
while (rs.next()) {
    System.out.println(rs.getString(1));
}

SparkCore && SparkSQL - 中文版本

介绍

为什么不使用 Hive？

1. 性能问题
• HiveQL 需要将查询转换为 MapReduce 任务，而 MapReduce 是基于磁盘的计算，速度非常慢。
• 相比之下，内存计算的 Spark 速度要快得多。

2. 功能有限
• HiveQL 无法满足复杂的数据处理需求，必须开发 UDF（用户自定义函数），增加了开发难度。

3. 只能处理结构化数据
• Hive 仅支持 结构化数据（Schema），无法处理 半结构化 或 非结构化 数据。

为什么要使用 Spark？

1. 高性能
• Spark 采用 内存计算，相比基于磁盘的 MapReduce，速度提高 10 倍。

2. 基于函数式编程的 RDD
• SparkCore 提供 RDD（弹性分布式数据集），可以灵活地进行数据转换。

3. SparkSQL 提供 SQL 计算能力
• SparkSQL 提供 SQL API，类似于 HiveQL，但效率更高。
• SparkSQL 还支持 DSL 函数 和 RDD 计算，更具灵活性。

理解 SparkCore 和 SparkSQL

SparkCore - 面向对象编程、面向函数编程

• 类似 MapReduce，使用 RDD 进行数据转换和计算。

• 应用场景：通常用于处理 半结构化数据，将其转换为结构化数据，以便进行分析。

• 示例：ETL（提取、转换、加载）数据处理。

from pyspark import SparkContext

# Initialize SparkContext
sc = SparkContext("local", "SparkCoreExample")

# Create an RDD from a list
data = ["Spark", "Hadoop", "Big Data", "Machine Learning"]
rdd = sc.parallelize(data)

# Transform the RDD by mapping each element to its length
length_rdd = rdd.map(lambda word: (word, len(word)))

# Collect and print the results
print(length_rdd.collect())

SparkSQL - 面向表，工作中最常用的

• 类似 HiveSQL，使用 SQL 语法来处理结构化数据。

• 应用场景：大规模数据分析和批量计算。

• 开发方式：
1. DSL（函数 API + RDD 计算）
2. SQL（使用 SQL 处理表数据）

from pyspark.sql import SparkSession

# Initialize SparkSession
spark = SparkSession.builder.appName("SparkSQLExample").getOrCreate()

# Create a DataFrame from sample data
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35)]
df = spark.createDataFrame(data, ["Name", "Age"])

# Register DataFrame as a temporary SQL table
df.createOrReplaceTempView("people")

# Run SQL query
result = spark.sql("SELECT Name, Age FROM people WHERE Age > 28")

# Show results
result.show()

总结

如何在工作中使用 SparkSQL

1. Python/Jar: 使用 spark-submit

应用场景: ETL（提取、转换、加载）

• Python: 提交基于 Python 开发的 Spark 作业。

• Java: 提交编译后的 JAR 文件到 Spark。

spark-submit --master yarn my_spark_job.py

spark-submit --class com.example.MyApp --master yarn my_app.jar

2. ThriftServer: 仅用于提交 SQL（类似于 Hive 的 HiveServer2）

常见的 SQL 提交方式

a) PyHive（生产环境中最常见）

from pyhive import hive

conn = hive.Connection(host='spark-thriftserver-host', port=10000, database='default')
cursor = conn.cursor()
cursor.execute("SELECT * FROM my_table LIMIT 10")
for result in cursor.fetchall():
    print(result)

b) spark-sql -f（用于 SQL 自动化执行）

spark-sql -f my_query.sql

c) Beeline（通常用于测试）

beeline -u jdbc:hive2://spark-thriftserver-host:10000/default

d) JDBC（较少使用）