com.alpine.plugin.core.utils

SparkParameterUtils

object SparkParameterUtils

Convenience functions for directly adding Spark related options to the dialog window.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SparkParameterUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. def addStandardSparkOptions(operatorDialog: OperatorDialog, additionalSparkParameters: List[SparkParameter]): Unit

  7. def addStandardSparkOptionsWithStorageLevel(operatorDialog: OperatorDialog, defaultNumExecutors: Int, defaultExecutorMemoryMB: Int, defaultDriverMemoryMB: Int, defaultNumExecutorCores: Int, defaultStorageLevel: String, additionalSparkParameters: List[SparkParameter] = List.empty[SparkParameter]): Unit

    A more advanced method for adding SparkP Parameters.

    A more advanced method for adding SparkP Parameters. Will also add a "StorageLevel" Parameter which will indicate what level of persistence to use within a Spark job. NOTE: The storage level parameter cannot be set automatically during runtime. To have any effect the custom operator developer must implement RDD persistence with this value (retrievable with 'getStorageLevel' method) in the Spark Job class of their operator.

    defaultStorageLevel

    - default storage level e.g. NONE or "MEMORY_AND_DISK.

    additionalSparkParameters

    - a list of a additional Spark Parameters.

  8. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  10. val disableDynamicAllocationParamId: String

  11. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  12. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  13. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  14. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  15. def getStorageLevel(operatorParameters: OperatorParameters): Option[String]

  16. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  17. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  18. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  19. final def notify(): Unit

    Definition Classes
    AnyRef

  20. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  21. val sparkDriverMBElementId: String

  22. val sparkExecutorMBElementId: String

  23. val sparkNumExecutorCoresElementId: String

  24. val sparkNumExecutorsElementId: String

  25. val storageLevelParamId: String

  26. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  27. def toString(): String

    Definition Classes
    AnyRef → Any

  28. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  29. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  30. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def addStandardSparkOptions(operatorDialog: OperatorDialog, defaultNumExecutors: Int, defaultExecutorMemoryMB: Int, defaultDriverMemoryMB: Int, defaultNumExecutorCores: Int): Unit

    Annotations
    @deprecated
    Deprecated

  2. def addStandardSparkOptions(operatorDialog: OperatorDialog, defaultNumExecutors: Int, defaultExecutorMemoryMB: Int, defaultDriverMemoryMB: Int, defaultNumExecutorCores: Int, additionalSparkParameters: List[SparkParameter]): Unit

    Annotations
    @deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped