Package org.apache.cassandra.spark.data

Class LocalDataLayer

java.lang.Object

org.apache.cassandra.spark.data.DataLayer

org.apache.cassandra.spark.data.LocalDataLayer

All Implemented Interfaces:

java.io.Serializable

public class LocalDataLayer
extends DataLayer
implements java.io.Serializable

Basic DataLayer implementation to read SSTables from local file system. Mostly used for testing.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	LocalDataLayer.Serializer

Field Summary

Fields

Modifier and Type	Field	Description
static long	serialVersionUID

Constructor Summary

Constructors

Constructor	Description
LocalDataLayer(org.apache.cassandra.bridge.CassandraVersion version, java.lang.String keyspace, java.lang.String createStatement, java.lang.String... paths)
LocalDataLayer(org.apache.cassandra.bridge.CassandraVersion version, java.lang.String keyspace, java.lang.String createStatement, java.util.Set<java.lang.String> udtStatements, java.lang.String... paths)
LocalDataLayer(org.apache.cassandra.bridge.CassandraVersion version, org.apache.cassandra.spark.data.partitioner.Partitioner partitioner, java.lang.String keyspace, java.lang.String createStatement, java.util.Set<java.lang.String> udts, java.util.List<org.apache.cassandra.spark.config.SchemaFeature> requestedFeatures, boolean useBufferingInputStream, java.lang.String statsClass, org.apache.cassandra.spark.sparksql.filters.SSTableTimeRangeFilter sstableTimeRangeFilter, java.lang.String... paths)

Method Summary

Modifier and Type	Method	Description
org.apache.cassandra.bridge.CassandraBridge	bridge()
org.apache.cassandra.spark.data.CqlTable	cqlTable()
boolean	equals(java.lang.Object other)
protected java.util.concurrent.ExecutorService	executorService()	DataLayer implementation should provide an ExecutorService for doing blocking I/O when opening SSTable readers.
static LocalDataLayer	from(java.util.Map<java.lang.String,java.lang.String> options)	Builds a new DataLayer from the options map.
int	hashCode()
boolean	isInPartition(int partitionId, java.math.BigInteger token, java.nio.ByteBuffer key)
java.lang.String	jobId()
int	partitionCount()
org.apache.cassandra.spark.data.partitioner.Partitioner	partitioner()
java.util.List<org.apache.cassandra.spark.config.SchemaFeature>	requestedFeatures()
void	setDataFilePaths(java.util.Set<java.nio.file.Path> dataFilePaths)
org.apache.cassandra.spark.data.SSTablesSupplier	sstables(int partitionId, org.apache.cassandra.spark.sparksql.filters.SparkRangeFilter sparkRangeFilter, java.util.List<org.apache.cassandra.spark.sparksql.filters.PartitionKeyFilter> partitionKeyFilters)
org.apache.cassandra.spark.sparksql.filters.SSTableTimeRangeFilter	sstableTimeRangeFilter()	Returns SSTableTimeRangeFilter to filter out SSTables based on min and max timestamp.
org.apache.cassandra.analytics.stats.Stats	stats()	Override to plug in your own Stats instrumentation for recording internal events
org.apache.cassandra.spark.utils.TimeProvider	timeProvider()
LocalDataLayer	withMinimumReplicasPerMutation(int minimumReplicasPerMutation)

Methods inherited from class org.apache.cassandra.spark.data.DataLayer

bigNumberConfig, openCompactionScanner, openCompactionScanner, openPartitionSizeIterator, partitionKeyFiltersInRange, partitionSizeStructType, readIndexOffset, sparkRangeFilter, structType, typeConverter, unsupportedPushDownFilters, useIncrementalRepair, version

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

serialVersionUID

public static final long serialVersionUID

See Also:

Constant Field Values

Constructor Detail

LocalDataLayer

public LocalDataLayer(@NotNull
                      org.apache.cassandra.bridge.CassandraVersion version,
                      @NotNull
                      java.lang.String keyspace,
                      @NotNull
                      java.lang.String createStatement,
                      java.lang.String... paths)

LocalDataLayer

public LocalDataLayer(@NotNull
                      org.apache.cassandra.bridge.CassandraVersion version,
                      @NotNull
                      java.lang.String keyspace,
                      @NotNull
                      java.lang.String createStatement,
                      @NotNull
                      java.util.Set<java.lang.String> udtStatements,
                      java.lang.String... paths)

LocalDataLayer

public LocalDataLayer(@NotNull
                      org.apache.cassandra.bridge.CassandraVersion version,
                      @NotNull
                      org.apache.cassandra.spark.data.partitioner.Partitioner partitioner,
                      @NotNull
                      java.lang.String keyspace,
                      @NotNull
                      java.lang.String createStatement,
                      @NotNull
                      java.util.Set<java.lang.String> udts,
                      @NotNull
                      java.util.List<org.apache.cassandra.spark.config.SchemaFeature> requestedFeatures,
                      boolean useBufferingInputStream,
                      @Nullable
                      java.lang.String statsClass,
                      @NotNull
                      org.apache.cassandra.spark.sparksql.filters.SSTableTimeRangeFilter sstableTimeRangeFilter,
                      java.lang.String... paths)

Method Detail

from

public static LocalDataLayer from(java.util.Map<java.lang.String,java.lang.String> options)

Builds a new DataLayer from the options map. The keys for the map must be lower-cased to guarantee compatibility with maps where the keys are all lower-cased.

Parameters:

options - the map with options

Returns:

a new DataLayer

requestedFeatures

public java.util.List<org.apache.cassandra.spark.config.SchemaFeature> requestedFeatures()

Overrides:

requestedFeatures in class DataLayer

bridge

public org.apache.cassandra.bridge.CassandraBridge bridge()

Specified by:

bridge in class DataLayer

Returns:

version-specific CassandraBridge wrapping shaded packages

partitioner

public org.apache.cassandra.spark.data.partitioner.Partitioner partitioner()

Specified by:

partitioner in class DataLayer

partitionCount

public int partitionCount()

Specified by:

partitionCount in class DataLayer

isInPartition

public boolean isInPartition(int partitionId,
                             java.math.BigInteger token,
                             java.nio.ByteBuffer key)

Specified by:

isInPartition in class DataLayer

sstableTimeRangeFilter

@NotNull
public org.apache.cassandra.spark.sparksql.filters.SSTableTimeRangeFilter sstableTimeRangeFilter()

Description copied from class: DataLayer

Returns SSTableTimeRangeFilter to filter out SSTables based on min and max timestamp.

Overrides:

sstableTimeRangeFilter in class DataLayer

Returns:

SSTableTimeRangeFilter

timeProvider

public org.apache.cassandra.spark.utils.TimeProvider timeProvider()

Specified by:

timeProvider in class DataLayer

Returns:

a TimeProvider

cqlTable

public org.apache.cassandra.spark.data.CqlTable cqlTable()

Specified by:

cqlTable in class DataLayer

Returns:

CqlTable object for table being read, batch/bulk read jobs only

stats

public org.apache.cassandra.analytics.stats.Stats stats()

Description copied from class: DataLayer

Override to plug in your own Stats instrumentation for recording internal events

Overrides:

stats in class DataLayer

Returns:

Stats implementation to record internal events

setDataFilePaths

public void setDataFilePaths(java.util.Set<java.nio.file.Path> dataFilePaths)

sstables

public org.apache.cassandra.spark.data.SSTablesSupplier sstables(int partitionId,
                                                                 @Nullable
                                                                 org.apache.cassandra.spark.sparksql.filters.SparkRangeFilter sparkRangeFilter,
                                                                 @NotNull
                                                                 java.util.List<org.apache.cassandra.spark.sparksql.filters.PartitionKeyFilter> partitionKeyFilters)

Specified by:

sstables in class DataLayer

Parameters:

partitionId - the partitionId of the task

sparkRangeFilter - spark range filter

partitionKeyFilters - the list of partition key filters

Returns:

set of SSTables

withMinimumReplicasPerMutation

public LocalDataLayer withMinimumReplicasPerMutation(int minimumReplicasPerMutation)

jobId

public java.lang.String jobId()

Specified by:

jobId in class DataLayer

Returns:

a string that uniquely identifies this Spark job

executorService

protected java.util.concurrent.ExecutorService executorService()

Description copied from class: DataLayer

DataLayer implementation should provide an ExecutorService for doing blocking I/O when opening SSTable readers. It is the responsibility of the DataLayer implementation to appropriately size and manage this ExecutorService.

Specified by:

executorService in class DataLayer

Returns:

executor service

hashCode

public int hashCode()

Overrides:

hashCode in class java.lang.Object

equals

public boolean equals(java.lang.Object other)

Overrides:

equals in class java.lang.Object