com.alpine.plugin.core.spark.utils

MLlibUtils

object MLlibUtils

Helper functions that are useful for working with MLlib.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. MLlibUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def anyToDouble(a: Any): Double

    Converts input of type Any to Double.

    Converts input of type Any to Double. Does this by casting to java.lang.Number, and then taking the double value.

    Will return Double.NaN in the case of bad input.

    a

    input to be converted to Double.

    returns

    Double representation of the number, or Double.NaN if impossible.

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  13. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  14. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  15. def mapSeqToCorrectType[T <: Seq[Any]](rowRDD: RDD[T], schema: StructType, nullable: Boolean = true): RDD[Row]

    An ugly function that handles the case where data that was a long, has been converted to a double and then can't be converted back i.

    An ugly function that handles the case where data that was a long, has been converted to a double and then can't be converted back i.e. 10 becomes 10.0 and then creating a dataFrame with an integer or long column fails. A non numeric input in a double or integer column will be converted to null, since round of Double.NAN is 0. Set the value of the nullable parameter to true in order to keep the behavior where null values in Integer and Long columns are converted to zeros.

    rowRDD

    and rdd of some sequence type that will be the body of the DataFrame

    schema

    the schema that we want tthe new RDD to conform to

    returns

    rowRDD with each value parsed so it won't cause failures when sqlContext.createDataFrame(rowRDD, schema) is called

  16. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  17. final def notify(): Unit

    Definition Classes
    AnyRef
  18. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  19. def rowToDoubleVector(row: Row): Vector

  20. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  21. def toLabeledPoint(dependentColumnIndex: Int, independentColumnIndices: Seq[Int]): (Row) ⇒ LabeledPoint

    dependentColumnIndex

    Index of the dependent column in the column set.

    independentColumnIndices

    Indices of the independent columns.

    returns

    Function mapping a Row of the correct schema to a labelled point.

  22. def toString(): String

    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped