com.alpine.plugin.core.spark.utils
Convert from Alpine's 'ColumnType' to the corresponding Spark Sql Typ.
Convert from Alpine's 'ColumnType' to the corresponding Spark Sql Typ. DateTime Behavior: Converts all DateTime columns to TimeStampType. If there is a format string will add that to the metadata. If there is no format string, will use the ISO format ("yyyy-mm-dd hh:mm:ss")
Converts from a Spark SQL Structfield to an Alpine-specific ColumnDef.
Converts from a Spark SQL Structfield to an Alpine-specific ColumnDef.
Converts from a Spark SQL schema to the Alpine 'TabularSchema' type.
Converts from a Spark SQL schema to the Alpine 'TabularSchema' type. The 'TabularSchema' object this method returns can be used to create any of the tabular Alpine IO types (HDFSTabular dataset, dataTable etc.)
Date format behavior: If the column def has not metadata stored at the DATE_METADATA_KEY constant, it wll convert DateType objects to ColumnType(DateTime, "yyyy-mm-dd") and TimeStampType objects to ColumnType(DateTime, "yyyy-mm-dd hh:mm:ss") otherwise will create a column type of ColumnType(DateTime, custom_date_format) where custom_date_format is whatever date format was specified by the column metadata.
-a Spark SQL DataFrame schema
the equivalent Alpine schema for that dataset
Convert the Alpine 'TabularSchema' with column names and types to the equivalent Spark SQL data frame header.
Convert the Alpine 'TabularSchema' with column names and types to the equivalent Spark SQL data frame header.
Date/Time behavior: The same as convertTabularSchemaToSparkSQLSchema(tabularSchema, false). Will NOT convert special date formats to String. Instead will render Alpine date formats as Spark SQL TimeStampType. The original date format will be stored as metadata in the StructFiled object for that column definition.
An Alpine 'TabularSchemaOutline' object with fixed column definitions containing a name and Alpine specific type.
Checks if the given file path already exists (and would cause a 'PathAlreadyExists' exception when we try to write to it) and deletes the directory to prevent existing results at that path if they do exist.
Checks if the given file path already exists (and would cause a 'PathAlreadyExists' exception when we try to write to it) and deletes the directory to prevent existing results at that path if they do exist.
- the full HDFS path
For use with hive.
For use with hive. Returns a Spark data frame given a hive table.
Returns a DataFrame from an Alpine HdfsTabularDataset.
Returns a DataFrame from an Alpine HdfsTabularDataset. The DataFrame's schema will correspond to the column header of the Alpine dataset. Uses the databricks csv parser from spark-csv with the following options: 1.withParseMode("DROPMALFORMED"): Catch parse errors such as number format exception caused by a string value in a numeric column and remove those rows rather than fail. 2.withTreatEmptyValuesAsNulls(true) -> the empty string will represent a null value in char columns as it does in alpine 3.If a CSV, The delimiter attributes specified by the CSV attributes object * Date format behavior: DateTime columns are parsed as dates and then converted to the TimeStampType according to the format specified by the Alpine type 'ColumnType' format argument. The original format is save in the schema as metadata for that column. It can be accessed with SparkSqlDateTimeUtils.getDatFormatInfo(structField) for any given column.
Alpine specific object. Usually input or output of operator.
Spark SQL DataFrame
Returns the dataframe corresponding to a given tabular dataset (can be Hive, or a HDFS path).
Returns the dataframe corresponding to a given tabular dataset (can be Hive, or a HDFS path).
The dataset containing a reference to the data on disk.
A DataFrame representation of the dataset.
Returns a DataFrame from an Alpine HdfsTabularDataset, created using a HiveContext (vs SQLContext in getDataFrame).
Returns a DataFrame from an Alpine HdfsTabularDataset, created using a HiveContext (vs SQLContext in getDataFrame). Using a Hive Context allows for window function calls on the DataFrame and Hive UDFs access. It does not require a previous Hive setup, but creating a Hive Context comes with large dependencies, hence this method should not be used if there is no need to leverage Hive UDFs.
(NOTE: loading the Hive dependencies might require to modify the "spark.driver.extraJavaOptions" by increasing MaxPermSize and enabling CMSClassUnloadingEnabled, UseConcMarkSweepGC. This parameter can be added in the "Advanced Spark Settings" dialog window of the operator).
Alpine specific object. Usually input or output of operator.
Spark SQL DataFrame that allows window function calls and Hive UDFs access.
JAVA TIME STAMP OBJECT--> STRING (via joda) Take in a dataFrame and map of the column names to the date formats we want to print.
JAVA TIME STAMP OBJECT--> STRING (via joda) Take in a dataFrame and map of the column names to the date formats we want to print. Convert to a dateFrames with the date columns as strings formatted correctly according to the map. Because the format is in joda time, we have to round trip via unix time. We use a udf to convert to a column with a Sql.TimeStamp type to a unix time, and then use Joda to format the unix time stamp according to the column format string provided in the map.
input data where date columns are represented as java TimeStamp Objects
columnName -> dateFormat to convert to
STRING -> JAVA TIMESTAMP OBJECT (based on unix time stamp) Take in a DataFrame and a map of the column names to the date formats represented in them and change the columns with string dates into JAVA TIME STAMP objects by parsing them according to the format defined in the map.
STRING -> JAVA TIMESTAMP OBJECT (based on unix time stamp) Take in a DataFrame and a map of the column names to the date formats represented in them and change the columns with string dates into JAVA TIME STAMP objects by parsing them according to the format defined in the map. Because Alpine uses Joda date formats, we use joda to parse the date strings into unix time stamps, and then convert unix time stamps to java sql objects. Preserves original naming of the columns. Columns which were originally DateTime columns will now be of TimeStampType rather than StringType.
the input dataframe where the date rows are as strings.
columnName -> dateFormat for parsing
"Illegal Date Format" if one of the date formats provided is not a valid Java SimpleDateFormat pattern. And "Could not parse dates correctly. " if the date format is valid, but doesn't correspond to the data that is actually in the column.
Write a DataFrame as an HDFSAvro dataset, and return the an instance of the Alpine HDFSAvroDataset type which contains the 'TabularSchema' definition (created by converting the DataFrame schema) and the path to the saved data.
More general version of saveAsCSV.
More general version of saveAsCSV. Write a DataFrame to HDFS as a Tabular Delimited file, and return an instance of the Alpine HDFSDelimitedTabularDataset type which contains the Alpine 'TabularSchema' definition (created by converting the DataFrame schema) and the path to the saved data. Also writes the ".alpine_metadata" to the result directory so that the user can drag and drop the result output and use it without configuring the dataset
where file will be written (this function will create a directory of part files)
- data to write
- an object which specifies how the file should be written
from parameters. Includes name and UUID Same as 'saveAsCSV' but also writes the ".alpine_metadata" to the result so that the user can drag and drop the result output and use it without configuring the dataset
More general version of saveAsTSV.
More general version of saveAsTSV. Write a DataFrame to HDFS as a Tabular Delimited file, and return an instance of the Alpine HDFSDelimitedTabularDataset type which contains the Alpine 'TabularSchema' definition (created by converting the DataFrame schema) and the path to the saved data.
where file will be written (this function will create a directory of part files)
- data to write
- an object which specifies how the file should be written
from parameters. Includes name and UUID
Write a DataFrame to HDFS as a Parquet file, and return an instance of the HDFSParquet IO base type which contains the Alpine 'TabularSchema' definition (created by converting the DataFrame schema) and the path to the saved data.
Write a DataFrame to HDFS as a Tab Delimited file, and return an instance of the Alpine HDFSDelimitedTabularDataset type which contains the Alpine 'TabularSchema' definition (created by converting the DataFrame schema) and the path to the saved data.
Write a DataFrame to HDFS as a Tab Delimited file, and return an instance of the Alpine HDFSDelimitedTabularDataset type which contains the Alpine 'TabularSchema' definition (created by converting the DataFrame schema) and the path to the saved data. Uses the default TSVAttributes object which specifies that the data be written as a Tab Delimited File. See TSVAAttributes for more information and use the saveAsCSV file to customize csv options such as null string and delimiters.
Also writes the ".alpine_metadata" to the result directory so that the user can drag and drop the result output and use it without configuring the dataset
Save a data frame to a path using the given storage format, and return a corresponding HdfsTabularDataset object that points to the path.
Save a data frame to a path using the given storage format, and return a corresponding HdfsTabularDataset object that points to the path.
The path to which we'll save the data frame.
The data frame that we want to save.
The format that we want to store in. Defaults to CSV
Whether to overwrite any existing file at the path.
Mandatory source operator information to be included in the output object. Note: Only set to None in testing. This information is required for Touchpoints and if this is an output to an operator with multiple inputs.
Mandatory addendum information to be included in the output object.
After saving the data frame, returns an HdfsTabularDataset object.
Calls above method but defaults to saving as a comma-seperated file with overwrite = true, and no addendum
Calls above method but defaults to saving as a comma-seperated file with overwrite = true, and no addendum
The path to which we'll save the data frame.
The data frame that we want to save.
Mandatory source operator information to be included in the output object. Note: Only set to None in testing. This information is required for Touchpoints and if this is an output to an operator with multiple inputs.
the HdfsTabularData object corresponding to the output
Will not properly handle data formats. Use toStructField
Converts from a Spark SQL data type to an Alpine-specific ColumnType
Converts from a Spark SQL data type to an Alpine-specific ColumnType
This doesn't properly handle date formats. Use convertColumnTypeToSparkSQLDataType instead
Save a data frame to a path using the given storage format, and return a corresponding HdfsTabularDataset object that points to the path.
Save a data frame to a path using the given storage format, and return a corresponding HdfsTabularDataset object that points to the path.
The path to which we'll save the data frame.
The data frame that we want to save.
The format that we want to store in.
Whether to overwrite any existing file at the path.
Mandatory source operator information to be included in the output object. Only set to None in testing, this is required for Touchpoints and if this is an output to an operator with multiple inputs.
Mandatory addendum information to be included in the output object.
After saving the data frame, returns an HdfsTabularDataset object.
Use signature with HdfsStorageFormatType rather than HdfsStorageFormat enum or saveDataFrameDefault
:: AlpineSdkApi ::