Resilient Distributed Datasets (RDDs) are the core abstraction in PySpark, offering fault-tolerant, distributed data structures that can be operated on in parallel. Although the DataFrame API is more popular due to its higher-level abstractions, RDDs are still fundamental for certain low-level operations and are the building blocks of PySpark.

In this article, you’ll learn how to create RDDs in PySpark, the different ways to create them. There are three main ways to create RDDs in PySpark:

1) From an existing collection (e.g., a list or set).
2) From an external data source (e.g., a file).
3) By transforming an existing RDD.

1. Creating an RDD from an Existing Collection: The simplest way to create an RDD is by using an in-memory collection, like a list or set in Python. The parallelize() method is used to convert the collection into an RDD.

# Importing the necessary module
from pyspark import SparkContext

# Initialize the Spark Context
sc = SparkContext("local", "RDD Example")

# Creating an RDD from a Python list
data = [1, 2, 3, 4, 5]
rdd = sc.parallelize(data)

# Displaying the RDD contents
print(rdd.collect()) 
# O/P: [1,2,3,4,5]

Explanation:

• SparkContext: This is the entry point for all Spark functionality. It allows you to create an RDD.
• parallelize(): This method is used to parallelize a local collection (like a list) and distribute it across the Spark cluster.
• collect(): This action retrieves all the data from the RDD and brings it back to the driver program.

2. Creating an RDD from External Data (e.g., Files) :You can create RDDs by reading data from external sources like text files, CSVs, or other formats. The textFile() method is commonly used to load text files as RDDs.

# Creating an RDD from an external text file
rdd = sc.textFile("path/to/your/file.txt")

# Displaying the first few lines of the file
print(rdd.take(5))

Explanation:

• textFile(): This method reads a text file and creates an RDD where each line of the file becomes an element in the RDD.
• take(n): This action retrieves the first n elements from the RDD, allowing you to preview the data.

3. Creating an RDD by Transforming an Existing RDD: You can also create a new RDD by applying transformations (such as map(), filter(), and flatMap() etc.) to an existing RDD. These transformations return a new RDD and allow you to perform operations on your data in a distributed way.

# Creating a new RDD from an existing RDD
rdd = sc.parallelize([1, 2, 3, 4, 5])

# Apply a transformation to multiply each element by 2
new_rdd = rdd.map(lambda x: x * 2)

# Displaying the new RDD
print(new_rdd.collect()) 
# O/P: [2,4,6,8,10]

Explanation:

• map(): This is a transformation that applies a function to each element of the RDD, returning a new RDD.
• lambda x: x * 2: This is an anonymous function that multiplies each element in the RDD by 2.
• collect(): This action is used to retrieve the results from the RDD back to the driver.

Conclusion:
RDDs are a fundamental data structure in PySpark, offering flexibility, fault tolerance, and the ability to handle unstructured data. Though higher-level APIs like DataFrames are often preferred for efficiency and ease of use, understanding how to create and work with RDDs is essential for certain use cases.

In this article, we covered three different ways to create an RDD in PySpark: from an in-memory collection, from an external data source, and by transforming an existing RDD.