Quick start: measure your cloud carbon footprint with Docker 🐳¶
Prerequisites¶
SPRUCE accepts AWS CUR reports (Parquet), Azure cost details (CSV), or FOCUS exports from either provider — see Generate usage reports for how to produce them. This tutorial uses an AWS CUR, generated via Data Exports and stored on S3 as Parquet files; commands for the other inputs are shown below.
For this tutorial, we will assume that you copied the S3 files to your local file system. You can do this with the AWS CLI
You will also need to have Docker installed.
With Docker¶
Pull the latest Docker image with
docker pull ghcr.io/digitalpebble/spruce
This retrieves a Docker image containing Apache Spark as well as the SPRUCE jar.
The command below processes the data locally by mounting the working directory as a volume. The input reports are assumed to be in a directory called curs.
The -i parameter specifies the location of the directory containing the input reports.
The -o parameter specifies the location of enriched Parquet files generated in output.
The option -c allows to specify a JSON configuration file to override the default settings.
The options -p (cloud provider) and -f (report format) are described in Quick start using Apache Spark.
The directory output contains an enriched copy of the input reports. See Explore the results to understand what the output contains.
Allocating more memory¶
The container runs SPRUCE with Spark in local mode, where a single JVM (the Spark driver) does all
the work. It defaults to 4 GB of heap, which is enough for the tutorial data but can be too little
for large reports — typically showing up as java.lang.OutOfMemoryError: Java heap space or as jobs
slowing down to a crawl in garbage collection.
Set the SPRUCE_DRIVER_MEMORY environment variable to give the driver a larger heap:
docker run --rm -v $(pwd):/workspace -w /workspace \
-e SPRUCE_DRIVER_MEMORY=16g \
ghcr.io/digitalpebble/spruce \
-i curs -o output
The value is passed to spark-submit --driver-memory, so it takes the usual Spark suffixes
(512m, 8g, ...).
Make sure the container is actually allowed to use that much: on Docker Desktop the VM has its own
memory limit, and if you constrain the container with --memory, keep the heap comfortably below it
so there is room for the JVM's off-heap memory.
SPRUCE_DRIVER_MEMORY only applies to the enrichment job. The report and dashboard commands do
not run Spark and ignore it.
Reporting with Docker¶
The same image contains the reporting tools, so you can generate a report or explore the enriched output interactively without installing Python.
Generate a static report (format inferred from the suffix: .md, .html or .pdf):
docker run -it --rm -v $(pwd):/workspace -w /workspace \
ghcr.io/digitalpebble/spruce \
report -i output -o report.html
Launch the interactive dashboard, then open http://localhost:8501:
docker run --rm -p 8501:8501 -v $(pwd):/workspace -w /workspace \
ghcr.io/digitalpebble/spruce \
dashboard -i output
The image version determines both the SPRUCE jar and the reporting scripts, so when
enriching and reporting are done at different times, use the same explicit tag
(e.g. ghcr.io/digitalpebble/spruce:1.1) for both steps rather than relying on latest.