Quick start using Apache Spark⚡¶
Instead of using a container, you can run SPRUCE directly on Apache Spark either locally or on a cluster.
Prerequisites¶
SPRUCE accepts AWS CUR reports (Parquet), Azure cost details (CSV), or FOCUS
exports from either provider — see Generate usage reports for how to
produce them. This tutorial uses an AWS CUR, generated via
Data Exports and
stored on S3 as Parquet files; other inputs are selected with the -p and -f options shown below.
For this tutorial, we will assume that you copied the S3 files to your local file system. You can do this with the AWS CLI
To run SPRUCE locally, you need Apache Spark installed and added to the $PATH.
Finally, you need the JAR containing the code and resources for SPRUCE. You can copy it from the latest release or alternatively, build from source, which requires Apache Maven and Java 21 or above.
Run on Apache Spark¶
If you downloaded a released jar, make sure the path matches its location.
spark-submit --class com.digitalpebble.spruce.SparkJob --driver-memory 8g ./target/spruce-*.jar -i ./curs -o ./output
The -i parameter specifies the location of the directory containing the input reports.
The -o parameter specifies the location of enriched Parquet files generated in output.
The option -c allows to specify a JSON configuration file to override the default settings.
The option -p selects the cloud provider (AWS, AZURE); it defaults to AWS. The option -f
selects the format of the input report: NATIVE (the provider's own export — the default) or
FOCUS for a FOCUS export. FOCUS is currently supported for
AWS (FOCUS 1.2 exports) and Azure (FOCUS 1.0 exports). Note that Azure reports are read as CSV files, unlike the AWS ones which are in Parquet:
spark-submit --class com.digitalpebble.spruce.SparkJob --driver-memory 8g ./target/spruce-*.jar -i ./focus-report -o ./output -p AZURE -f FOCUS
The directory output contains an enriched copy of the input reports. See Explore the results to understand what the output contains.