How to Extract Date Part from Folder Name and Move it to Another Folder on HDFS using PySpark

Are you tired of manually extracting date parts from folder names and moving them to other folders on HDFS? Do you want to automate this process using PySpark? Look no further! In this article, we’ll guide you through a step-by-step process on how to extract date parts from folder names and move them to another folder on HDFS using PySpark.

Prerequisites

Before we dive into the solution, make sure you have the following prerequisites:

• PySpark installed on your system
• An HDFS cluster setup with write permissions
• A sample dataset with folder names containing date parts (e.g., “2022-01-01_data”, “2022-01-02_logs”, etc.)

Step 1: Import necessary libraries and create a PySpark session

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, substr, to_date

# create a PySpark session
spark = SparkSession.builder.appName("Extract Date Part from Folder Name").getOrCreate()

In this step, we’ve imported the necessary libraries, including `SparkSession` and `pyspark.sql.functions`. We’ve also created a PySpark session with an application name.

Step 2: Read the folder names from HDFS

# read folder names from HDFS
folder_names = spark.sparkContext.wholeTextFiles("hdfs:///path/to/folder/*").map(lambda x: x[0].split("/")[-1])

In this step, we’ve read the folder names from HDFS using the `wholeTextFiles` method. We’ve specified the path to the folder containing the folders with date parts in their names. The `map` function is used to extract the folder name from the full path.

Step 3: Extract the date part from the folder name

# extract the date part from the folder name
date_part = folder_names.map(lambda x: substr(x, 0, 10))

# convert the date part to a date format
date_part = date_part.map(lambda x: to_date(x, "yyyy-MM-dd"))

In this step, we’ve extracted the date part from the folder name using the `substr` function, which extracts a substring from the folder name. We’ve specified the starting position as 0 and the length as 10 to extract the date part in the format “yyyy-MM-dd”.

Next, we’ve converted the extracted date part to a date format using the `to_date` function.

Step 4: Create a new folder name with the date part

# create a new folder name with the date part
new_folder_name = date_part.map(lambda x: x.strftime("%Y/%m/%d"))

In this step, we’ve created a new folder name by formatting the date part using the `strftime` function. We’ve specified the format as “%Y/%m/%d” to create a folder structure with year, month, and day.

Step 5: Move the folder to the new location on HDFS

# move the folder to the new location on HDFS
new_folder_name.foreach(lambda x: spark.sparkContext.hadoopFileSystem.mkdirs("hdfs:///path/to/new/folder/" + x))

# move the files to the new location
folder_names.foreach(lambda x: spark.sparkContext.hadoopFileSystem.rename("hdfs:///path/to/old/folder/" + x, "hdfs:///path/to/new/folder/" + x[:10] + "/" + x))

In this step, we’ve moved the folder to the new location on HDFS using the `mkdirs` method to create the new folder structure. We’ve specified the new folder path and the date part as the folder name.

Next, we’ve moved the files to the new location using the `rename` method. We’ve specified the old folder path and the new folder path with the date part.

Step 6: Verify the results

Verify that the folders have been moved to the new location on HDFS with the correct date part in their names.

Old Folder Name	New Folder Name
2022-01-01_data	2022/01/01/data
2022-01-02_logs	2022/01/02/logs

In this step, we’ve verified that the folders have been moved to the new location with the correct date part in their names.

Conclusion

In this article, we’ve demonstrated how to extract date parts from folder names and move them to another folder on HDFS using PySpark. By following these steps, you can automate this process and save time and effort.

Additional Tips

Here are some additional tips to keep in mind:

• Make sure to replace the `hdfs:///path/to/folder/*` with the actual path to your folder on HDFS.
• Adjust the date format in the `to_date` function to match your folder naming convention.
• Use the `coalesce` method to write the data to HDFS in parallel.

By following these steps and tips, you can efficiently extract date parts from folder names and move them to another folder on HDFS using PySpark.

Frequently Asked Question

Get ready to unlock the secrets of extracting date parts from folder names and moving them to another folder on HDFS using PySpark!