Problem

You are trying to SET the value of a Spark config in a notebook and get a Cannot modify the value of a Spark config error.

For example:

%sql

SET spark.serializer=org.apache.spark.serializer.KryoSerializer

Error in SQL statement: AnalysisException: Cannot modify the value of a Spark config: spark.serializer;

Cause

The SET command does not work on SparkConf entries. This is by design in Spark 3.0 and above.

Solution

You should remove SET commands for SparkConf entries from your notebook.

You can enter SparkConf values at the cluster level by entering them in the cluster’s Spark config (AWS | Azure | GCP) and restarting the cluster.