Time: 2 hours. Difficulty: Intermediate. Start Building!
The goal of this workshop is to get you familiar with the powerful and versatile
tool NoSQLBench
. With that, you can perform
industry-grade, robust benchmarks
aimed at several (distributed) target systems, especially NoSQL databases.
Today you'll be benchmarking Astra DB, a database-as-a-service built on top of Apache Cassandra. Along the way, you will learn the basics of NoSQLBench.
In this repository you will find all material and references you need:
- NoSQLBench Discord
- NoSQLBench homepage
- Exercises
- Step-by-step guide
- DataStaxDevs Discord server to keep in touch with us
- Our Q&A forum (think StackOverflow for Cassandra and all things DataStax)
- Slide deck
- Before you start
- Create your Astra DB instance
- Launch Gitpod and setup NoSQLBench
- Run benchmarks
- Workloads
- Homework assignment
Heads up: these instructions are available in two forms: a short and to-the-point one (this one), with just the useful commands if you are watching us live; and a longer one, with lots of explanations and details, designed for those who follow this workshop at their own pace. Please choose what best suits you!
- What are the prerequisites?
This workshop is aimed at data architects, solution architects, developers, or anybody who wants to get serious about measuring the performance of their data-intensive system. You should know what a (distributed) database is, and have a general understanding of the challenges of communicating over a network.
- Do I need to install a database or anything on my machine?
No, no need to install anything. You will do everything in the browser. (That being said, the knowledge you gain today will probably be best put to use once you install NoSQLBench on some client machine to run tests.)
You can also choose to work on your machine instead of using Gitpod: there's no problem with that, just a few setup and operational changes to keep in mind. We will not provide live support in this case, though, assuming you know what you are doing.
- Is there anything to pay?
No. All materials, services and software used in this workshop is free.
- Do you cover NoSQLBench 4 or 5?
Ah, I see you are a connoisseur. We focus on the newly-release NoSQLBench 5, but we provide tips and remarks aimed at those still using nb4.
To complete the workshop and get a verified "NoSQLBench" badge, follow these instructions:
- Do the hands-on practice, either during the workshop or by following the instructions in this README;
- (optional) Complete the "Lab" assignment as detailed here;
- Fill the submission form here. Answer the theory questions and (optionally) provide a screenshot of the completed "Lab" part;
- give us a few days to process your submission: you should receive your well-earned badge in your email inbox!
First you must create a database: an instance of Astra DB, which you will then benchmark with NoSQLBench.
Don't worry, you will create it within the "Free Tier", which offers quite a generous free allowance in terms of monthly I/O (about 40M operations per month) and storage (80 GB).
You need to:
- create an Astra DB instance as explained here, with database name =
workshops
and keyspace name =nbkeyspace
; - (this will happen automatically with the previous one) generate and retrieve a DB Token as explained here. Important: use the role "DB Administrator" if manually creating the token.
- generate and download a Secure Connect Bundle as explained here;
Moreover, keep the Astra DB dashboard open: it will be useful later. In particular, locate the Health tab and the CQL Console.
Ctrl-click on the Gitpod button below to spawn your very own environment + IDE:
In a few minutes, a full IDE will be ready in your browser, with a file
explorer on the left, a file editor on the top, and a console (bash
) below it.
To download NoSQLBench, type or paste this command in your Gitpod console:
curl -L -O https://github.com/nosqlbench/nosqlbench/releases/latest/download/nb5
then make it executable and move it to a better place:
chmod +x nb5
sudo mv nb5 /usr/local/bin/
Ok, now check that the program starts: invoking
nb5 --version
should output the program version (something like 5.17.3
or higher).
This workshop is built for the newly-released NoSQLBench 5.
Locate, with the file explorer on your computer, the bundle file that
you downloaded earlier (it should be called
secure-connect-workshops.zip
)
and simply drag-and-drop it to the file navigator panel
("Explorer") on the left of the Gitpod view.
Once you drop it you will see it listed in the file explorer itself. As a check, you can issue the command
ls /workspace/workshop-nosqlbench/secure*zip -lh
so that you get the absolute path to your bundle file (and also verify that it is the correct size, about 12-13 KB).
Copy the provided template file to a new one and open it in the Gitpod file editor:
cp .env.sample .env
gp open .env
# (you can also simply locate the file
# in the Explorer and click on it)
Insert the "Client ID" and "Client Secret" of the DB Token you created earlier and, if necessary, adjust the other variables.
Now, source this file to make the definitions therein available to this shell:
. .env
To check that the file has been sourced, you can try with:
echo ${ASTRA_DB_KEYSPACE_NAME}
and make sure the output is not an empty line.
(Note that you will have to source the file in any new shell you plan to use).
Everything is set to start running the tool.
Try launching this very short "dry-run benchmark", that instead of actually
reaching the database simply prints a series of CQL statements to the console
(as specified by the driver=stdout
parameter):
nb5 cql-keyvalue2 astra \
driver=stdout \
rampup-cycles=10 \
main-cycles=10 \
keyspace=${ASTRA_DB_KEYSPACE_NAME}
You will see 21 (fully-formed, valid CQL) statements being printed:
one CREATE TABLE
, then ten INSERT
s
and then another ten between SELECT
s and further INSERT
s.
Note: we will use workload
cql-keyvalue2
throughout. This is functionally identical to thecql-keyvalue
workload but is expressed in the newer syntax foryaml
workloads, which comes handy when later dissecting its content. If you are working with NoSQLBench 4, remember to drop the trailing2
from the workload name in the following!
Now re-launch the above dry run and look for differences in the output:
nb5 cql-keyvalue2 astra \
driver=stdout \
rampup-cycles=10 \
main-cycles=10 \
keyspace=${ASTRA_DB_KEYSPACE_NAME}
is the output identical to the previous run down to the actual "random" values?
You can also peek at the logs
directory now: it is created automatically and
populated with some information from the benchmark at each execution of nb
.
It is now time to start hitting the database!
This time you will run with driver=cql
to actually reach the database:
for that to work, you will provide all connection parameters set up earlier.
The next run will ask NoSQLBench to perform a substantial amount of operations, in order to collect enough statistical support for the results.
Here is the full command to launch:
nb5 cql-keyvalue2 \
astra \
username=${ASTRA_DB_CLIENT_ID} \
password=${ASTRA_DB_CLIENT_SECRET} \
secureconnectbundle=${ASTRA_DB_BUNDLE_PATH} \
keyspace=${ASTRA_DB_KEYSPACE_NAME} \
cyclerate=50 \
driver=cql \
main-cycles=9000 \
rampup-cycles=9000 \
errors='OverloadedException:warn' \
--progress console:5s \
--log-histograms 'histogram_hdr_data.log:.*.main.result.*:20s' \
--log-histostats 'hdrstats.log:.*.main.result.*:20s'
Show me the command breakdown
Note that some of the parameters (e.g. keyspace
) are workload-specific.
command | meaning |
---|---|
cql-keyvalue2 |
workload |
astra |
scenario |
username |
authentication |
password |
authentication |
secureconnectbundle |
Astra DB connection parameters |
keyspace |
target keyspace |
cyclerate |
rate-limiting (cycles per second) |
driver=cql |
driver to use (CQL, for AstraDB/Cassandra) |
main-cycles |
how many operations in the "main" phase |
rampup-cycles |
how many operations in the "rampup" phase |
errors |
behaviour if errors occur during benchmarking |
--progress console |
frequency of console prints |
--log-histograms |
write data to HDR file (see later) |
--log-histostats |
write some basic stats to a file (see later) |
This way of invoking nb
, the "named scenario"
way, is not the only one: it is also possible to have a finer-grained control over what activities should
run with a full-fledged CLI scripting syntax.
Note: the syntax of the
errors
parameter has been improved in NoSQLBench 5 to allow for a finer control (with multiple directives, such aserrors='NoNodeAvailable.*:ignore;InvalidQueryException.*:counter;OverloadedException:warn'
). On version 4 you should revert to a simpler parameter, such aserrors=count
, instead of the above.
The benchmark should last about ten minutes, with the progress being printed on the console as it proceeds.
While this runs, have a look around.
Now it's time to find out what is actually being written to the database.
Choose your database in the Astra main dashboard and click on it; next, go to the "CQL Console" tab in the main panel. In a few seconds the console will open in your browser, already connected to your database and waiting for your input.
Start by telling the console that you will be using the nbkeyspace
keyspace:
USE nbkeyspace;
Check what tables have been created by NoSQLBench in this keyspace:
DESC TABLES;
You should see table keyvalue
listed as the sole output.
Look at a a few lines from this table:
SELECT * FROM keyvalue LIMIT 20;
Ok, mystery solved. It looks like the table contains simple key-value pairs, with two columns seemingly of numeric type. Check with:
DESC TABLE keyvalue;
Oh, looks like both the key and the value columns are of type TEXT
:
good for adapting this ready-made benchmark to other key/value stores.
Locate your database in the Astra main dashboard and click on it; next, go to the "Health" tab in the main panel. You will see what essentially is a Grafana dashboard, with a handful of plots being displayed within the tab - all related to how the database is performing in terms of reads and writes.
Check the operations per second from the "Requests Combined" plot; then have a look at the "Write Latency" and "Read Latency" plots and take note of some of the percentiles shown there.
Show me "sample values" one could read from the graph
Below is a real-life example of the values that could result from a cql-keyvalue2
benchmark session in the main phase:
Percentile | Write Latency | Read Latency |
---|---|---|
P50 | 709 µs | 935 µs |
P75 | 831 µs | 1.31 ms |
P90 | 904 µs | 1.53 ms |
P95 | 1.04 ms | 1.77 ms |
P99 | 2.45 ms | 15.6 ms |
When the benchmark has finished, open the latest *.summary
file and look
for cqlkeyvalue2_astra_main.result-success
.
Under that metric title, you will see something similar to:
cqlkeyvalue2_astra_main.result-success
count = 15000
mean rate = 50.00 calls/second
1-minute rate = 49.94 calls/second
5-minute rate = 50.29 calls/second
15-minute rate = 50.57 calls/second
Use this script to generate a graph of the data collected as "histostats":
./hdr_tool/histostats_quick_plotter.py \
hdrstats.log \
-m cqlkeyvalue2_astra_main.result-success
and then open, in the Gitpod editor, the hdrstats.png
image just created.
The version of the plotter script included in this repo is for educational purposes only: for general use, please head to the official release page.
The timings will be larger than those from the Astra health tab: indeed, these are "as seen on the client side" and include more network hops.
Use this script to generate plots from the detailed "HDR histogram data" generated during the benchmark:
./hdr_tool/hdr_tool.py \
histogram_hdr_data.log \
-b -c -s \
-p SampleData \
-m cqlkeyvalue2_astra_main.result-success
The version of the plotter script included in this repo is for educational purposes only: for general use, please head to the official release page.
Again, the timings are larger than those found on the Astra health tab (i.e. on server-side): these measurements are reported "as seen by the testing client".
Launch a new benchmark, this time having NoSQLBench start a dockerized Grafana/Prometheus stack for metrics (it will take a few more seconds to start):
nb5 cql-keyvalue2 \
astra \
username=${ASTRA_DB_CLIENT_ID} \
password=${ASTRA_DB_CLIENT_SECRET} \
secureconnectbundle=${ASTRA_DB_BUNDLE_PATH} \
keyspace=${ASTRA_DB_KEYSPACE_NAME} \
cyclerate=50 \
rampup-cycles=15000 \
main-cycles=15000 \
errors='OverloadedException:warn' \
--progress console:5s \
--docker-metrics
Reach the Grafana container in a new tab, with an URL that has 3000-
in front of your Gitpod URL (e.g.
https://3000-datastaxdevs-workshopnos-[...].gitpod.io
).
The default credentials to log in to Grafana are ... admin/admin
. Once you're
in, don't bother to reset your password (click "Skip"). You'll get to the Grafana
landing page. Find the "Dashboards" icon in the leftmost menu bar and pick the
"Manage" menu item: finally, click on the "NB4 Dashboard" item you should see
listed there. Congratulations, you are seeing the data coming from NoSQLBench.
You may find it convenient to set the update frequency to something like 10 seconds and the displayed time window to 5 minutes or so (upper-right controls).
The dashboard comprises several (interactive) plots, updated in real time.
To reach the Prometheus container, which handles the "raw" data behind Grafana,
open a modified URL (this time with 9090-
) in a new tab.
Click on the "world" icon next to the "Execute" button in the search bar:
in the dialog that appears you can look for specific metrics.
Try to look for result_success
and confirm, then click "Execute".
Tip: switch to the "Graph" view for a more immediate visualization. The graphs display "raw" data, hence are in units of nanoseconds.
To make sense of the (heterogeneous) results, some filtering is in order -- but we are not entering too much into the details of Prometheus here.
Just to pique your interest, try pasting these examples and click "Execute":
# filtering by metadata
{__name__="result_success", type="pctile", alias=~".*main.*"}
# aggregation
avg_over_time({__name__="result_success", type="pctile", alias=~".*main.*"}[10m])
# another aggregation, + filtering
max_over_time({__name__="result_success", type="pctile", alias=~".*main.*"}[10m])
This part is about how workloads are defined.
Tip: feel free to interrupt the previous benchmark, if it still runs, with Ctrl-C. You won't need it anymore.
Ask NoSQLBench to dump to a file the yaml
defining the workload
you just ran:
nb5 --copy cql-keyvalue2
(you can also get a comprehensive list of all available workloads with
nb5 --list-workloads
, by the way, and a more fine-grained output with
nb5 --list-scenarios
.)
A file cql-keyvalue2.yaml
is created in the working directory.
You can open it (clicking on it in the Gitpod explorer or by running
gp open cql-keyvalue2.yaml
).
Have a look at the file and try to identify its structure and the various phases the benchmark is organized into.
There are profound differences in the way the same workload is expressed in the NoSQLBench 4 yaml file and the NoSQLBench 5 format.
A good way to understand workload construction is to start from simple ones.
To run the following examples please go to the appropriate subdirectory:
cd workloads
Run the first example (and then look at the corresponding
simple-workload.yaml
) with:
nb5 run driver=stdout workload=simple-workload cycles=12
Look at how bindings connect the sequence of operations to "execute" (in this case, simply print on screen) with the data to be used in them.
Run the second example, which is an example of structuring a workload
in phases (and then open workload-with-phases.yaml
):
nb5 workload-with-phases default driver=stdout
Notable features of this workload are its multi-phase structure
(a nearly universal feature of actual benchmarks), the use
of the ratio
parameter, and the usage of template parameters in the
definition.
The "Lab" part of the homework, which requires you to finalize
a workload yaml
and make it work according to specifications,
is detailed on this page.
To submit your homework, please use this form.