Create a simple version of the AutoTuner options.
Create a simple version of the AutoTuner options. This includes two basic constants used in the auto tuning process.
if the data is large we set the executors to the size of a yarn container. By default we set the driver to the same size as the executors. However, if your computation does not return much data to the driver you may not need so much driver memory. This scalar provides a way to scale the size of the driver. If you have a highly parallelizable algorithm that does not return much input to the driver (say KMeans) try setting this to less then one. If you want the driver memory to be larger even on small input data (perhaps for an algorithm that aggregates by a group and returns data to the driver try setting it to more than one.
In auto tuning we try to use enough executors so that the input data could be read in memory into the compute layer of the executors. We calculate the size of the input data in memory by multiplying the file size (which you can see by right clicking a dataset in alpine and viewing the "Hadoop File Properties" dialog) by a scalar.
The default if 4. i.e. we are assuming that you are using all of the input data and that it is perhaps a dataset of integers which take roughly four times as much space on memory than on disk. Adjust this value based on your estimation of the resources required. For example, if your operation immediately filters the data to a few integer columns a better multiplier might be (selectedColumns)/(totalColumns) * 4.