Problem

You are trying to import timestamp_millis or unix_millis into a Scala notebook, but get an error message.

%scala

import org.apache.spark.sql.functions.{timestamp_millis, unix_millis}

error: value timestamp_millis is not a member of object org.apache.spark.sql.functions
import org.apache.spark.sql.functions.{timestamp_millis, unix_millis}

Cause

The functions timestamp_millis and unix_millis are not available in the Apache Spark DataFrame API.

These functions are specific to SQL and are included in Spark 3.1.1 and above.

Solution

You need to use selectExpr() with timestamp_millis or unix_millis if you want to use either one of them with a DataFrame.

selectExpr() takes a set of SQL expressions and runs them.

For example, this sample code returns an error message when run.

%scala

import sqlContext.implicits._
val df = Seq(
 (1, "First Value"),
 (2, "Second Value")
).toDF("int_column", "string_column")

import org.apache.spark.sql.functions.{unix_millis}
import org.apache.spark.sql.functions.col
df.select(unix_millis(col("int_column"))).show()

error: value unix_millis is not a member of object org.apache.spark.sql.functions
import org.apache.spark.sql.functions.{unix_millis}

While this sample code, using selectExpr(), successfully returns timestamp values.

%scala

import org.apache.spark.sql.functions._
import sqlContext.implicits._
val ndf = Seq(
 (1, "First Value"),
 (2, "Second Value")
).toDF("int_column", "string_column")

display(ndf.selectExpr("timestamp_millis(int_column)"))

Example notebook

Review the Cannot import timestamp_millis or unix_millis example notebook.