spark

Type Members

1. class AlpineSparkEnvironment extends AnyRef

   

   This object is made available to the user during the spark job and has the relevant context of the spark job, including the spark session, the application conf and access to Alpine's runtime environment

2. case class AutoTunerOptions(options: Map[String, String]) extends Product with Serializable

   

3. final class OperatorFailedException extends Exception

   

   To be thrown if the operator encounters some exception at runtime (like a bad configuration), and the developer does not want the framework (in particular, Spark) to retry the operation.

4. trait SparkExecutionContext extends ExecutionContext

   

   :: AlpineSdkApi ::

   :: AlpineSdkApi ::

   Annotations

   @AlpineSdkApi()

5. abstract class SparkIOTypedPluginJob[I, O] extends AnyRef

   

   :: AlpineSdkApi ::

   :: AlpineSdkApi ::

   This is the Spark Job class. It gets submitted to the cluster by a Runtime class which is a derivative of SparkRuntime. The 'onExecution method' in this class serves as the driver function for the Spark Job.

   It handles the serialization/deserialization of Inputs and Outputs. It enables you to directly work with IOBase objects without needing to implement your own (de)serialization logic. This class is intended to be coupled with SparkRuntimeWithIOTypedJob, a descendant of SparkRuntime that takes a descendant of this class as a generic parameter.

   Note: It is possible to use this class with a runtime class that extends the generic SparkRuntime class (rather than the SparkRuntimeWithIOTypedJob class). However, by using SparkRuntimeWithIOTypedJob and overriding the onExecution method, you can get many of the benefits of the class while implementing more complex behavior. In taking the later approach you can use the SparkRuntimeWithIOTypedJob implementation of the 'onExecution' method as a utility function for submitting the Spark job by calling super.onExecution.

   I

   input type of your plugin must be consistent with the SparkRuntime implementation's type parameters.

   O

   output type of your plugin

   Annotations

   @AlpineSdkApi()

6. case class SparkJobConfiguration(userDefinedParameters: Map[String, String], autoTuneMissingValues: Boolean, autoTunerOptions: AutoTunerOptions) extends Product with Serializable

   

   Spark job configuration.

7. abstract class SparkRuntime[I <: IOBase, O <: IOBase] extends OperatorRuntime[SparkExecutionContext, I, O]

   

   Defines the behavior of your plugin after the end user 'runs' it from the GUI.

   Defines the behavior of your plugin after the end user 'runs' it from the GUI. This is a direct descendant of OperatorRuntime and it takes SparkExecutionContext as an argument. Its 'onExecution' runs on the local machine (Alpine machine). It is in this 'onExecution' method (call with'super.onExecution') that you could submit a Spark job. Unlike its less generic descendant, SparkRuntimeWithIOTypedJob, which automatically submits a SparkJob, you can use this class for more elaborate runtime behavior. For example, you can do some local processing here before submitting a SparkPluginJob to the cluster manually. Or you could can also do some purely local computations and just return without submitting a job.

   I

   the IOBase input type of your plugin (must be consistent with the input type of the GUINode class implementation, the plugin signature implementation.

   O

   the output of your plugin.

   Annotations

   @AlpineSdkApi()

8. abstract class SparkRuntimeWithIOTypedJob[J <: SparkIOTypedPluginJob[I, O], I <: IOBase, O <: IOBase] extends OperatorRuntime[SparkExecutionContext, I, O]

   

   A descendant of SparkRuntime which handles the most straightforward runtime behavior, submitting a Spark job with the input of your plugin and returning the output type.

   A descendant of SparkRuntime which handles the most straightforward runtime behavior, submitting a Spark job with the input of your plugin and returning the output type. It takes an implementation of SparkIOTypedPluginJob as a generic parameter, and it is in that class where you will define the logic of the Spark job. Use this class if all you want to do is submit a Spark job since it takes care of submitting the Spark job and serializing/ de-serializing the outputs.

   J

   your implementation of the SparkIOTypedPluginJob class, whose type parameters must align with I an O here

   I

   the IOBase input type of your plugin (must be consistent with the input type of the GUINode class implementation, and the plugin signature implementation.

   O

   the output of your plugin. Note: In order to do more at runtime than just submit a Spark job, but use our serialization logic, you could use this class with SparkIOTypedPluginJob and simply override the 'onExecution' method (see SparkIOTypedPluginJob documentation for more).

9. trait SubmittedSparkJob[O] extends AnyRef

   

   :: AlpineSdkApi ::

   :: AlpineSdkApi ::

   O

   Output type of the submitted Spark job.