Thursday, March 11, 2021

Add a uuid column to a spark dataframe

Recently, I came across a use case where i had to add a new column uuid in hex to an existing spark dataframe, here are two ways we can achieve that

import pyspark.sql.functions as f
from pyspark.sql.types import StringType

# method 1 use udf 
uuid_udf = f.udf(lambda : str(uuid.uuid4().hex), StringType())
df_with_uuid = df.withColumn('uuid', uuid_udf())

# method 2 use lit 
df_with_uuid = df.withColumn('uuid', f.lit(uuid.uuid4().hex))

2 comments:

  1. Elegant Data refers to the idea of presenting and working with data in a clean, organized, and meaningful way so that users can easily understand insights. In Data Analytics, elegant data practices include proper data cleaning, structured datasets, effective visualization, clear dashboards, and simple reporting. Tools such as Excel, SQL, Python, Power BI, and Tableau can be used to organize and visualize data. PySpark Course.The main goal is to transform raw and complex data into clear information that helps businesses identify patterns, trends, performance, and useful insights for better decision-making.

    ReplyDelete