templates

Type Members

1. abstract class InferredSparkDataFrameGUINode[Job <: InferredSparkDataFrameJob] extends SparkDataFrameGUINode[Job]

   

   Control the GUI of your Spark job, through this you can specify any visualization for the output of your job, and what params the user will need to specify.

   Control the GUI of your Spark job, through this you can specify any visualization for the output of your job, and what params the user will need to specify. Uses the provided operator to generate an updated schema, this should work for most operators but if not (e.g. your operator doesn't handle empty data or output schema depends on input data) then you will have to perform your own schema update.

2. abstract class InferredSparkDataFrameJob extends SparkDataFrameJob

   

   A class for plugins which will use Schema inference

3. abstract class SparkDataFrameGUINode[Job <: SparkDataFrameJob] extends TemplatedSparkDataFrameGUINode[HdfsTabularDataset]

   

   Control the GUI of your Spark job, through this you can specify any visualization for the output of your job, and what params the user will need to specify.

4. abstract class SparkDataFrameJob extends TemplatedSparkDataFrameJob[DataFrame, HdfsTabularDataset]

   

   Job base for non-inferred Spark plugin jobs taking and returning DataFrames.

   Job base for non-inferred Spark plugin jobs taking and returning DataFrames. Note: This WILL NOT work with hive.

5. abstract class SparkDataFrameRuntime[JobType <: SparkDataFrameJob] extends SparkRuntimeWithIOTypedJob[JobType, HdfsTabularDataset, HdfsTabularDataset]

   

   A class controlling the runtime behavior of your plugin.

   A class controlling the runtime behavior of your plugin. To use the default implementation, which launches a Spark job according to the default Spark settings you will not need to add any code beyond the class definition with the appropriate type parameters.

   JobType

   your implementation of SparkDataFrameJob

6. abstract class TemplatedSparkDataFrameGUINode[StorageType <: IOBase] extends OperatorGUINode[HdfsTabularDataset, StorageType]

   

   Control the GUI of your Spark job, through this you can specify any visualization for the output of your job, and what params the user will need to specify.

7. abstract class TemplatedSparkDataFrameJob[ReturnType, OutputType <: IOBase] extends SparkIOTypedPluginJob[HdfsTabularDataset, OutputType]

   

   Templated base for Spark plugin jobs operating on DataFrames.

   Templated base for Spark plugin jobs operating on DataFrames. Most jobs will want to use SparkDataFrameJob which takes and returns Spark DataFrames. This version does not support schema inference.

   ReturnType

   The return type of the transformation method (most commonly a DataFrame)

   OutputType

   The return type of the actual operator, extending IOBase. Most commonly will be an HDFS dataset of some flavor (see SparkDataFrame)

8. abstract class TemplatedSparkDataFrameRuntime[JobType <: TemplatedSparkDataFrameJob[_, O], O <: IOBase] extends SparkRuntimeWithIOTypedJob[JobType, HdfsTabularDataset, O]