Renaming columns in a PySpark DataFrame is a common task when you’re cleaning, transforming, or organizing data. Whether you’re working with external datasets or need to make your DataFrame more readable, PySpark offers multiple ways to rename columns. In this article, we’ll cover three popular methods to rename columns in PySpark:

1) withColumnRenamed()
2) selectExpr()
3) select() with col()

Method 1: Using withColumnRenamed(): The most straightforward way to rename a column in PySpark is by using the withColumnRenamed() method. This method allows you to rename one column at a time.

Syntax: DataFrame.withColumnRenamed(existing, new)
Parameters:
a) existing: The name of the existing column that you want to rename.
b) new: The new name for the column.

Example 1: Renaming a Single Column: Let’s start with a simple example where we rename one column using withColumnRenamed().

# Importing necessary modules
from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder.appName("Rename Columns Example").getOrCreate()

# Sample DataFrame
data = [("Alice", 25), ("Bob", 30), ("Catherine", 29)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)

# Rename the 'Name' column to 'Full Name'
df_renamed = df.withColumnRenamed("Name", "Full Name")

# Show the result
df_renamed.show()

Output:

+----------+---+
| Full Name|Age|
+----------+---+
|     Alice| 25|
|       Bob| 30|
| Catherine| 29|
+----------+---+

In this example, the column Name has been renamed to ‘Full Name’ from ‘Name’.

Example 2: Renaming Multiple Columns: Although withColumnRenamed() allows renaming only one column at a time, you can chain it to rename multiple columns.

# Rename both columns: 'Name' to 'Full Name' and 'Age' to 'Years'
df_renamed_multiple = df.withColumnRenamed("Name", "Full Name").withColumnRenamed("Age", "Years")

# Show the result
df_renamed_multiple.show()

Output:

+----------+-----+
| Full Name|Years|
+----------+-----+
|     Alice|   25|
|       Bob|   30|
| Catherine|   29|
+----------+-----+

This example shows how you can rename multiple columns by chaining the withColumnRenamed() method calls.

Method 2: Using selectExpr(): Another method to rename columns is using selectExpr(). This method is more powerful because it allows you to rename multiple columns and apply expressions simultaneously. It uses SQL-like expressions to define the new column names.

Syntax:DataFrame.selectExpr(*exprs)
Parameter:
exprs: A list of SQL-like expressions that can include renaming columns.

Example: Renaming Multiple Columns: Let’s use selectExpr() to rename multiple columns in one step.

# Rename columns using selectExpr
df_renamed_expr = df.selectExpr("Name as Full_Name", "Age as Years")

# Show the result
df_renamed_expr.show()

Output:

+----------+-----+
| Full_Name|Years|
+----------+-----+
|     Alice|   25|
|       Bob|   30|
| Catherine|   29|
+----------+-----+

In this example:
“Name as Full_Name”: This expression renames the Name column to Full_Name.
“Age as Years”: This expression renames the Age column to Years.

Method 3: Using select() with col(): The third method to rename columns is by using the select() method along with the col(). This method allows you to select specific columns and apply renaming with the help of alias().

Syntax: DataFrame.select(*cols)
Parameter:
cols: A list of columns to select, which can be renamed using the alias() function.

Example: Renaming Multiple Columns: Let’s rename columns using select() along with col() and alias().

# Import col function
from pyspark.sql.functions import col

# Rename columns using select and col with alias
df_renamed_select = df.select(col("Name").alias("Full Name"), col("Age").alias("Years"))

# Show the result
df_renamed_select.show()

Output:

+----------+-----+
| Full Name|Years|
+----------+-----+
|     Alice|   25|
|       Bob|   30|
| Catherine|   29|
+----------+-----+

In this example:
a) col(“Name”).alias(“Full Name”): This renames the Name column to Full Name.
b) col(“Age”).alias(“Years”): This renames the Age column to Years.
This method provides more flexibility, especially if you need to perform additional transformations when selecting the columns.

Conclusion: Renaming columns in PySpark is a fundamental operation that can be done using several methods( withColumnRenamed(),selectExpr(), select() with col()), Each method has its strengths, and choosing the right one depends on your specific use case and the complexity of the task at hand. Understanding these approaches will help you manage your DataFrames more effectively when working with large datasets in PySpark.