topic Re: Show distinct column values in pyspark dataframe in Governance, Risk, Compliance

Show distinct column values in pyspark dataframe

ursyathi — Mon, 09 Oct 2023 10:41:26 GMT

With pyspark dataframe, how do you do the equivalent of Pandas df['col'].unique().

I want to list out all the unique values in a pyspark dataframe column.

Not the SQL type way (registertemplate then SQL query for distinct values).

Also I don't need groupby then countDistinct, instead I want to check distinct VALUES in that column.

Caute_cautim — Sun, 13 Aug 2023 06:22:21 GMT

@ursyathi Have you checked Google?

Not sure whether this helps or not, having never used this data framework myself.

Regards

Caute_Cautim