found: Unit required: org.apache.spark.sql.Dataset[_]

Solved2.92K viewsscala spark

found: Unit required: org.apache.spark.sql.Dataset[_]

I am splitting the data from a dataframe with this code and I am getting this error while using VectorAssembler

val newData = data.withColumn("_tmp", split($"datetime", "\\ ")).
 withColumn("date", $"_tmp".getItem(0)).
 withColumn("time", $"_tmp".getItem(1)).
drop("_tmp").
show
val output = assembler.transform(newData)

How to resolve this?

Share:
vito Selected answer as best September 21, 2021
1

You should not use the show method in the dataframe (newData). Remove the show method and the error will be resolved. If you want to display the output, you can execute the below statement separately after creating the dataframe

newData.show

Share:
vito Selected answer as best September 21, 2021
1