com.alpine.plugin.core.spark.templates
This is called to generate the visual output for the results console.
This is called to generate the visual output for the results console. If the developer does not override it, we try OperatorGUINode#onOutputVisualization, which predated this, so we keep for compatibility.
Execution context of the operator.
The input to the operator.
The output from the execution.
The parameter values to the operator.
The listener object to communicate information back to the console.
Set the options passed to our Spark Auto Tuner which will choose optimal Spark configuration settings for values not provided by the user based on the size of the cluster, the input data and the type of computation.
Set the options passed to our Spark Auto Tuner which will choose optimal Spark configuration settings for values not provided by the user based on the size of the cluster, the input data and the type of computation. See documentation for the AutoTunerOptions object for more details on what the settings in this object mean. Set only the auto tuning options by overriding this method. To change the parameters passed the Spark Configuration more comprehensively override 'getSparkJobConfiguration' and this method will be ignored.
The default implementation looks for the parameter values that would be included by com.alpine.plugin.core.utils.SparkParameterUtils.addStandardSparkOptions.
The default implementation looks for the parameter values that would be included by com.alpine.plugin.core.utils.SparkParameterUtils.addStandardSparkOptions. If these are not provided we call out to Alpine's Spark Auto Tuning algorithm, which will determine them. The result of this method is an object which we will use to determine the Spark settings. The SparkJobConfiguration object contains three fields. 1. A map with the advanced parameters. This should be the parameters in the "Advanced Spark Parameters" box. However, if you would like to modify these values, or add your own Spark options here, you may do that by adding those values to this object. 2. A boolean "autoTuneMissingValues". If set to false, this will disable the auto tuning. In this case you must fill in the values of "spark.executor.memory" "spark.driver.memory" and "spark.executor.instances" in the "userDefinedParameters" object 3. Options that will be used for the Auto Tuning. See com.alpine.plugin.core.spark.SparkJobConfiguration for details
Parameters of the operator.
The input to the operator.
The Spark job configuration that will be used to submit the Spark job.
The runtime behavior of the plugin.
The runtime behavior of the plugin. This method is called when the user clicks 'run' or 'step run in the GUI'. The default implementation --configures the Spark job as defined by the getSparkJobConfiguration --submits a Spark job with the input dataType the parameters, the application context, and the listener --de-serializes the output returned by the Spark job --returns the de-serialized output of the Spark job as an IOBase output object.
A Spark specific execution context, includes Spark parameters.
The input to the operator.
The parameter values to the operator.
The listener object to communicate information back to the console or the Alpine UI.
The output from the execution.
This is called when the user clicks on 'stop'.
This is called when the user clicks on 'stop'. If the operator is currently running, this function gets called while 'onExecution' is still running. So it's the developer's responsibility to properly stop whatever is going within 'onExecution'.
Execution context of the operator.
The listener object to communicate information back to the console.
A class controlling the runtime behavior of your plugin. To use the default implementation, which launches a Spark job according to the default Spark settings you will not need to add any code beyond the class definition with the appropriate type parameters.
your implementation of SparkDataFrameJob