import pyspark.sql.functions as f
from pyspark.sql.types import StringType
# method 1 use udf
uuid_udf = f.udf(lambda : str(uuid.uuid4().hex), StringType())
df_with_uuid = df.withColumn('uuid', uuid_udf())
# method 2 use lit
df_with_uuid = df.withColumn('uuid', f.lit(uuid.uuid4().hex))
GREAT
ReplyDeleteElegant Data refers to the idea of presenting and working with data in a clean, organized, and meaningful way so that users can easily understand insights. In Data Analytics, elegant data practices include proper data cleaning, structured datasets, effective visualization, clear dashboards, and simple reporting. Tools such as Excel, SQL, Python, Power BI, and Tableau can be used to organize and visualize data. PySpark Course.The main goal is to transform raw and complex data into clear information that helps businesses identify patterns, trends, performance, and useful insights for better decision-making.
ReplyDelete