found: Unit required: org.apache.spark.sql.Dataset[_]
found: Unit required: org.apache.spark.sql.Dataset[_]
I am splitting the data from a dataframe with this code and I am getting this error while using VectorAssembler
val newData = data.withColumn("_tmp", split($"datetime", "\\ ")).
withColumn("date", $"_tmp".getItem(0)).
withColumn("time", $"_tmp".getItem(1)).
drop("_tmp").
show
val output = assembler.transform(newData)
How to resolve this?
vito Selected answer as best September 21, 2021
You should not use the show method in the dataframe (newData). Remove the show method and the error will be resolved. If you want to display the output, you can execute the below statement separately after creating the dataframe
newData.show
vito Selected answer as best September 21, 2021