Automated SQL Server Benchmarking with HammerDB and Docker: A Complete Testing Framework

I’m excited to announce the release of a new open-source project that fully automates HammerDB benchmarking for SQL Server using Docker. If you’ve ever needed to run TPC-C or TPC-H benchmarks multiple times, you know how time-consuming the manual setup can be. This project removes the hassle and gets you up and running a single command: ./loadtest.sh.

Updated for HammerDB 6.0

What’s new since I first wrote this:

• Every run writes a JSON report and HTML charts, not just console output.
• Response times are broken out per stored procedure, with p99 through p25.
• You can tag runs with a PROFILE_ID and diff two configurations directly.
• loadtest.sh runs through the whole lifecycle of a load test.

Why I Built This

In my work, I frequently benchmark SQL Server configurations, whether I’m comparing versions, testing new hardware, or validating performance tuning changes. Setting up HammerDB manually each time became a significant time bottleneck (see what I did there! ;). I needed an automated solution that would work consistently across different environments and reduce the time required to get test results.

Every setting is parameterized through the hammerdb.env file, no more digging through TCL scripts or using a GUI to change configurations!

Getting Started: The 5-Minute Setup

# Clone the repository
git clone https://github.com/nocentino/hammerdb.git
cd hammerdb

# Configure for your environment
cp hammerdb.env.example hammerdb.env
# Edit hammerdb.env with your SQL Server connection details.

# Run everything!
./loadtest.sh

What happens when you run ./loadtest.sh: 0. Starts up a SQL Server 2025 container.

1. Builds the TPC-C schema with your specified number of warehouses.
2. Runs the TPC-C OLTP workload.
3. Parses the results, writing a JSON report and charts.
4. Tears the container back down.

loadtest.sh reads its connection settings straight out of hammerdb.env, so it only creates and removes a container when SQL_SERVER_HOST points at localhost. Point it at a real server and it runs the benchmark and leaves that server alone.

TPC-H isn’t part of that script. Drive it through Docker Compose instead, which is also how you run any individual phase:

RUN_MODE=build BENCHMARK=tproch docker compose up --abort-on-container-exit
RUN_MODE=load  BENCHMARK=tproch docker compose up --abort-on-container-exit
docker compose run --rm --no-TTY -e RUN_MODE=parse -e BENCHMARK=tproch hammerdb

Building the schema is a one-time cost per size. Once it exists, loop on load and parse while you tune.

The Architecture

Here’s a breakdown of what’s under the hood:

hammerdb/
├── hammerdb.env                   # All your settings in one place, edit this based on the hardware you're testing.
├── hammerdb.env.example           # Template to copy to hammerdb.env.
├── docker-compose.yaml            # Orchestrates the containers.
├── dockerfile                     # HammerDB 6.0 plus mssql-tools18.
├── entrypoint.sh                  # Picks the right script from RUN_MODE and BENCHMARK.
├── loadtest.sh                    # Control script, TPC-C end to end.
├── compare.sh                     # Compares two configurations.
├── scripts/
│   ├── build_schema_tprocc.tcl    # Builds the TPC-C schema.
│   ├── build_schema_tproch.tcl    # Builds the TPC-H schema.
│   ├── load_test_tprocc.tcl       # Runs the TPC-C test.
│   ├── load_test_tproch.tcl       # Runs the TPC-H test.
│   ├── parse_output_tprocc.tcl    # Parses results for TPC-C.
│   ├── parse_output_tproch.tcl    # Parses results for TPC-H.
│   ├── compare_profiles.tcl       # Diffs two performance profiles.
│   └── hammerdb6_compat.tcl       # Works around a HammerDB 6.0 packaging bug.
└── output/                        # Results are stored here.

That’s it! This framework automates the entire process of building schemas, running tests, and collecting results. Once everything is set up, you can begin experimenting with more complex test scenarios to truly push your SQL Server to its limits. I have provided some examples and guidance in the GitHub repository. Just adjust the parameters based on your hardware and point the SQL_SERVER_HOST configuration hammerdb.env at your SQL Server an go!

Understanding the Results

The framework automatically extracts key metrics, including:

TPC-C Output:

• Transactions Per Minute (TPM)
• New Orders Per Minute (NOPM)
• Per-transaction response times, reported as p99, p95, p75, p50, and p25, plus min, average, max, standard deviation, and call counts for each stored procedure (NEWORD, PAYMENT, DELIVERY, SLEV, OSTAT)

TPC-H Output:

• Individual query execution times
• Total runtime for all 22 queries
• Query-specific metrics

Every run now leaves machine-readable output in output/ instead of just log text:

HammerDB’s own report goes further than the compact one. It records the benchmark configuration right next to the result, so you can always see what produced the number. It also works out an active window for the run, meaning the span between the first and last real transaction count. That lets you cut the rampup and the shutdown tail out of your charts instead of guessing where the run actually started.

Each chart is a single HTML file you can open straight from output/ in a browser. They pull the ECharts library from a CDN rather than embedding it, so they need internet access to render, worth knowing if your lab is isolated. TPC-H writes the same shape, plus the original plain text report.

That percentile detail is the part I use most. An average response time hides the tail, and the tail is usually what your users actually notice. Here’s a real run to show what I mean, 8 virtual users against 10 warehouses for 30 minutes, SQL Server 2025 CU8 in a container on a laptop:

TEST RESULT : System achieved 97341 NOPM from 226275 SQL Server TPM

And the response times that came with it, in milliseconds:

Look at NEWORD. The p99 is 8.5 ms, which is perfectly healthy. The max is 4.9 seconds! Something stalled hard a handful of times out of nearly three million calls, a checkpoint or an autogrowth probably, and the average wouldn’t have told you and neither would the p99. That gap between p99 and max is the first thing I look at now. It’s also why I wanted these percentiles written to a file instead of scrolling past me in a terminal.

Every per-run filename carries the job ID, so runs pile up rather than overwrite, and each chart shows exactly one run. The comparison output is the exception. It’s named for the profile pair, tprocc_profile_1_vs_2.html, so running the same comparison twice quietly replaces the last one. And nothing cleans up output/ or compacts hammer.DB on its own, so both of them grow run over run. Mine got to 6 GB before I noticed.

Comparing Two Configurations

This is the workflow I built the framework for in the first place. Tag each run with a PROFILE_ID, then diff the profiles:

# Baseline, with PROFILE_ID=1 set in its env file
HAMMERDB_ENV_FILE=hammerdb-2022.env RUN_MODE=load BENCHMARK=tprocc docker compose up --abort-on-container-exit

# The configuration under test, with PROFILE_ID=2
HAMMERDB_ENV_FILE=hammerdb-2025.env RUN_MODE=load BENCHMARK=tprocc docker compose up --abort-on-container-exit

# Compare them
docker compose run --rm --no-TTY -e RUN_MODE=compare -e BENCHMARK=tprocc \
  -e BASE_PROFILE_ID=1 -e COMP_PROFILE_ID=2 hammerdb

Which gives you a direct comparison rather than two result blobs to eyeball:

Profiles compared (unweighted): matched=2, avg_base=76948, avg_comp=69116

Please don’t quote those numbers at anyone. That run was SQL Server 2022 against 2025, but both were containers on one laptop, sharing the same CPU and disk, under x86 emulation on Apple silicon, and the editions didn’t even match. It tells you the comparison works. It tells you nothing about the versions.

What a profile id actually is

A PROFILE_ID is just a tag. When a run finishes, HammerDB writes that number into the job’s row in hammer.DB next to the result. Comparing two profiles means pulling every job tagged 1 and every job tagged 2, pairing them up by virtual user count, and diffing the averages. That’s all it is.

Which means a profile is a set of runs, not one run. That’s what matched is counting, pairs. Run both configurations at the same virtual user counts or there’s nothing to pair up, and a comparison built on a single run each is mostly noise.

It also means the tags stick around. They aren’t scoped to your session or your env file, they just sit in the repository, so anything you ran earlier carrying the same id is still part of that profile.

I found that out the obvious way. My hammerdb.env had PROFILE_ID=1 sitting in it, so a casual ./loadtest.sh from hours earlier had quietly joined my 2022 baseline. By the time I ran the real comparison, profile 1 held three runs instead of two, and two of them were at the same virtual user count. HammerDB picked one and ignored the other without a word.

The summary looked perfectly clean. It just wasn’t answering the question I thought I’d asked.

So keep PROFILE_ID=0 in your everyday hammerdb.env, which is what the example ships with. That keeps casual runs out of everyone’s way. Then give each comparison its own pair of ids rather than recycling 1 and 2 forever, and read the profile listing before you trust the summary line underneath it, because the listing is where a stray run shows itself.

compare.sh wraps the whole thing up:

VU_COUNTS="4 8 16 32" ./compare.sh hammerdb-2022.env hammerdb-2025.env

That runs the workload at each of those virtual user counts against both configurations and compares the curves, which is far more useful than a single data point.

Get the Code

The complete project is available on GitHub: https://github.com/nocentino/hammerdb. I’ve included extensive documentation and examples to help you get started quickly. The README contains additional details on customization options and advanced configurations.

The repository is currently built and tested against HammerDB 6.0 and SQL Server 2025 CU8, and there’s a GitHub Actions smoke test that builds the image and runs a full build, load, parse, and comparison cycle against a real SQL Server on every push and pull request, so the thing you clone actually works.

A special thanks to Andrew Pruski, my partner in SQL Server benchmarking, for his contributions to this project. I couldn’t have done it without you!

Related Posts

• AI-Driven Load Testing with HammerDB, Copilot, and Claude - Use AI assistants to orchestrate and interpret HammerDB load tests