Trait

org.apache.spark.sql

Encoder

Related Doc: package sql

Permalink

trait Encoder[T] extends Serializable

Experimental

Used to convert a JVM object of type T to and from the internal Spark SQL representation.

Scala

Encoders are generally created automatically through implicits from a SparkSession, or can be explicitly created by calling static methods on Encoders.

import spark.implicits._

val ds = Seq(1, 2, 3).toDS() // implicitly provided (spark.implicits.newIntEncoder)

Java

Encoders are specified by calling static methods on Encoders.

List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());

Encoders can be composed into tuples:

Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

Or constructed from Java Beans:

Encoders.bean(MyClass.class);

Implementation

Annotations
@Experimental() @Evolving() @implicitNotFound( ... )
Source
Encoder.scala
Since

1.6.0

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Encoder
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def clsTag: ClassTag[T]

    Permalink

    A ClassTag that can be used to construct and Array to contain a collection of T.

  2. abstract def schema: StructType

    Permalink

    Returns the schema of encoding this type of object as a Row.