Performance Engineering of Software Systems

Massachusetts Institute of Technology

Profs. Saman P. Amarasinghe and Charles E. Leiserson

September 8, 2016
6.172/6.871
Handout 2

Homework 1: Getting Started

Due: 11:59 p.m. on Tuesday, September 13, 2016
Last Updated: September 10, 2016
This homework introduces the environment and tools you will be using to complete your future project
assignments. You will experiment with a simple program for multiplying rectangular matrices. You should
use this assignment to familiarize yourself with the tools you will be using throughout the course.

Get a Microsoft Azure account

Follow the instructions in the Recitation 1: Basic Tools handout to get an Azure account and set
up your Azure VM.

Version control

We shall use the Git distributed version control system. The baseline code for this assignment is in
the repository /afs/athena.mit.edu/course/6/6.172/student-repos/fa16/homeworks/homework1/
username.git If you havent used Git before, please be aware that its a little bit different than
Subversion (svn), which you may have used in other classes.
To make a clone a local copy of the repository for your work on a machine without AFS,
type:
$ git clone ssh://[email protected]/afs/athena.mit.edu/\
course/6/6.172/student-repos/fa16/homeworks/homework1/username.git homework1

To make a clone on your Azure VM, on athena.dialup, or other machine with AFS, type:
$ git clone /afs/athena.mit.edu/course/6/6.172/student-repos/fa16/\
homeworks/homework1/username.git homework1

Edit the code in the project, and frequently, you should commit and push your changes back to
the repository:
$ git commit -am Your commit message
$ git push origin master

For more advanced usage of Git, please refer to your favorite search engine.

Handout 2 Homework 1: Getting Started

Building and running your code

You can build the code by going to the homework1/matrix_multiply directory and typing make.
The program will compile using Tapir, a cutting-edge derivative of Clang/LLVM. Notice that
we are only compiling with optimization level 1 (i.e., -O1). Modify your Makefile so that the
program is compiled using optimization level 3 (i.e., -O3).

Write-up 1: Now, what do you see when you type make clean; make?

You can then run the built binary by typing ./matrix_multiply. The program should print
out something and then crash with a segmentation fault.

Using a debugger

While running your program, if you encounter a segmentation fault, bus error, or assertion
failure, or if you just want to set a breakpoint, you can use the debugging tool GDB.
You can start a debugging session in GDB:
$ gdb --args ./matrix_multiply

This command should give you a GDB prompt, at which you should type run or r:
$ (gdb) run

Your program will crash, giving you back a prompt, where you can type backtrace or bt to get
a stack trace:
...
Program received signal SIGSEGV, Segmentation fault.
0x0000000000400de4 in matrix_multiply_run ()
(gdb) bt
#0 0x0000000000400de4 in matrix_multiply_run ()
#1 0x0000000000400bbe in main ()

This stack trace says that the program crashes in matrix_multiply_run, but doesnt tell any
other information about the error. In order to get more information, build a debug version of
the code. First, quit GDB by typing quit or q:
(gdb) q
A debugging session is active.
Inferior 1 [process 26817] will be killed.
Quit anyway? (y or n) y

Handout 2 Homework 1: Getting Started

Next, build a debug version of the code by typing make DEBUG=1:

$ make DEBUG=1
clang -g -DDEBUG -O0 -Wall -std=c99 -D_POSIX_C_SOURCE=200809L -c testbed.c -o testbed.o
clang -g -DDEBUG -O0 -Wall -std=c99 -D_POSIX_C_SOURCE=200809L -c matrix_multiply.c
-o matrix_multiply.o
clang -o matrix_multiply testbed.o matrix_multiply.o -lrt

The major differences from the optimized build are -g (add debug symbols to your program)
and -O0 (compile without any optimizations). Once you have created a debug build, you can
start a debugging session again:
$ gdb --args ./matrix_multiply
(gdb) r
...
Program received signal SIGSEGV, Segmentation fault.
0x00000000004011cf in matrix_multiply_run (A=0x603270, B=0x6031d0, C=0x603130)
at matrix_multiply.c:90
90
C->values[i][j] += A->values[i][k] * B->values[k][j];

Now, GDB can tell that a segmentation fault occurs at at matrix_multiply.c line 90. You can
ask GDB to print values using print or p:
(gdb) p A->values[i][k]
$1 = 7
(gdb) p B->values[k][j]
Cannot access memory at address 0x0
(gdb) p B->values[k]
$2 = (int *) 0x0
(gdb) p k
$3 = 4

This suggests that B->values[4] is 0x0, which means B doesnt have row 5. There is something
wrong with matrix dimensions.

Using assertions

The tbassert package is a useful tool for catching bugs before your program goes off into the
weeds. If you look at matrix_multiply.c, you should see some assertions in matrix_multiply_run
routine that check that the matrices have compatible dimensions. Uncomment these lines and a
line to include tbassert.h at the top of the file. Then, build and run the program again using
GDB. Make sure that you build using make DEBUG=1. You will get
(gdb) r
...
Running matrix_multiply_run()...
matrix_multiply.c:80 (int matrix_multiply_run(const matrix *, const matrix *, matrix *))
Assertion A->cols == B->rows failed: A->cols = 5, B->rows = 4
Program received signal SIGABRT, Aborted.
0x00007ffff7843c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6

Handout 2 Homework 1: Getting Started

Now, GDB tells that Assertion A->cols == B->rows failed, which is much better than
the former segmentation fault. The assertion provides a printf-like API that allows you to print
values in your own output, as above. However, even if you dont print values in your assertions,
the debug build still has the symbols for GDB, as above. Unlike the above, however, if you try to
print A->cols, you will fail. The reason is that GDB is not in the stack frame you want. You can
get the stack trace to see which frame you want (#3 in this case), and type frame 3 or f 3 to move
to frame #3. After that, you can print A->cols and B->cols.
(gdb) bt
#0 0x00007ffff7843c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff7847028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000040121b in matrix_multiply_run (A=0x603270, B=0x6031d0, C=0x603130)
at matrix_multiply.c:79
#3 0x0000000000400db2 in main (argc=1, argv=0x7fffffffdf58) at testbed.c:127
(gdb) f 3
#3 0x0000000000400db2 in main (argc=1, argv=0x7fffffffdf58) at testbed.c:127
127
matrix_multiply_run(A, B, C);
(gdb) p A->cols
$1 = 5
(gdb) p B->rows
$2 = 4

You should see the values 5 and 4, which indicates that we are multiplying matrices of incompatible dimensions.
You will also see an assertion failure with a line number for the failing assertion without
using GDB. Since the extra checks performed by assertions can be expensive, they are disabled
for optimized builds, which are the default in our Makefile. As a result, if you make the program
without DEBUG=1, you will not see an assertion failure.
You should consider sprinkling assertions throughout your code to check important invariants in your program, since they will make your life easier when debugging. In particular,
most nontrivial loops and recursive functions should have an assertion of the loop or recursion
invariant.
Fix testbed.c, which creates the matrices, rebuild your program, and verify that it now
works. You should see Elapsed execution time... after running
$ ./matrix_multiply

Commit and push your changes to the Git repository:

$ git commit -am Your commit message
$ git push origin master

Next, check the result of the multiplication. Run

$ ./matrix_multiply -p

The program will print out the result. The result seems to be wrong, however. You can check the
multiplication of zero matrices by running
$ ./matrix_multiply -pz

Handout 2 Homework 1: Getting Started

Using a memory checker

Some memory bugs do not crash the program, so GDB cannot tell you where the bug is. You can
use the memory checking tool Valgrind to track these bugs. If youre on your Azure VM, youll
need to run
$ sudo apt-get -y install libc6-dbg

to satisfy software dependencies (as a reminder, you have root access to your VMs). Now run
Valgrind using
$ valgrind ./matrix_multiply -p

You need the -p switch, since Valgrind only detects memory bugs that affect outputs. You should
also use a debug version to get a good result. This command should print out many lines. The
important ones are
==43644== Use of uninitialised value of size 8
==43644==
at 0x508899B: _itoa_word (_itoa.c:179)
==43644==
by 0x508C636: vfprintf (vfprintf.c:1660)
==43644==
by 0x50933D8: printf (printf.c:33)
==43644==
by 0x401137: print_matrix (matrix_multiply.c:68)
==43644==
by 0x400E0E: main (testbed.c:133)

This output indicates that the program used a value before initializing it. The stack trace
indicates that the bug occurs in testbed.c:133, which is where the program prints out matrix C.
Fix matrix_multiply.c to initialize values in matrices before using them. Rebuild your program, and verify that it outputs a correct answer. Again, commit and push your changes to the
Git repository.

Write-up 2: After you fix your program, run ./matrix_multiply -p. Paste the program
output showing that the matrix multiplication is working correctly.

Memory management

The C programming language requires you to free memory after you are done using it, or else
you will have a memory leak. Valgrind can track memory leaks in the program. Run the same
Valgrind command, and you will see these lines at the very end:
==2158== LEAK SUMMARY:
==2158==
definitely lost:
==2158==
indirectly lost:
==2158==
possibly lost:
==2158==
still reachable:
==2158==
suppressed:

48 bytes in 3 blocks
288 bytes in 15 blocks
0 bytes in 0 blocks
0 bytes in 0 blocks
0 bytes in 0 blocks

Handout 2 Homework 1: Getting Started

This output suggests that there are indeed memory leaks in the program. To get more information, you can build your program in debug mode and again run Valgrind, using the flag
--leak-check=full
$ valgrind --leak-check=full ./matrix_multiply -p

The trace shows that all leaks are from the creations of matrices A, B, and C.
Fix testbed.c by freeing these matrices after use. Rebuild your program, and verify that
Valgrind doesnt complain about anything. Commit and push your changes to the Git repository.

Write-up 3: Paste the output from Valgrind showing that there is no error in your program.

Checking code coverage

Bugs may exist in code that doesnt get executed in your tests. You may find it surprising when
someone testing your code (like a professor or a TA) uncovers a crash on a line that you never
exercised. Additionally, lines that are frequently executed are good candidates for optimization.
The Gcov tool provides a line-by-line execution count for your program.
To use Gcov, modify your Makefile and add the flags -fprofile-arcs and -ftest-coverage
to the CFLAGS and LDFLAGS variables. You will have to rebuild from scratch using make clean
followed by make DEBUG=1. Try running your code normally with ./matrix_multiply -p. Note
that a number of new .gcda and .gcno files were created during your execution.
Now use the gcov-tapir commandline utility on testbed.c:
$ gcov-tapir testbed.c

A new file, testbed.c.gcov was created that is identical to the original testbed.c, except that
it has the number of times each line was executed in the code. In that file, you will see:
...
1:
#####:
#####:
#####:
#####:
#####:
#####:
#####:
#####:
#####:
#####:
...

99:
100:
101:
102:
103:
104:
105:
106:
107:
108:
109:

if (use_zero_matrix) {
for (int i = 0; i < A->rows; i++) {
for (int j = 0; j < A->cols; j++) {
A->values[i][j] = 0;
}
}
for (int i = 0; i < B->rows; i++) {
for (int j = 0; j < B->cols; j++) {
B->values[i][j] = 0;
}
}

Handout 2 Homework 1: Getting Started

The hash-marks indicate lines that were never executed. In general, it is unusual to run a
code-coverage utility on a testbed, but a set of untested lines in your core code could lead to
unexpected results when executed by someone else.
Another handy use of Gcov is identifying which lines got executed the most frequently.
Code that gets run the most is often the most costly in terms of performance. Run gcov-tapir
matrix_multiply.c and look at the output:
10:
40:
160:
64:
64:
16:
4:

93: for (int i = 0; i < A->rows; i++) {

94:
for (int j = 0; j < B->cols; j++) {
95:
for (int k = 0; k < A->cols; k++) {
96:
C->values[i][j] += A->values[i][k] * B->values[k][j];
97:
}
98:
}
99: }

These are the loops in matrix_multiply_run. Clearly, this function is a good candidate for
optimization.
When you are done using Gcov, remove the flags you added to the Makefile because they add
costly overhead to the execution, and will negatively impact your actual performance numbers.
You should never run benchmarks on code that is instrumented with Gcov. Dont forget to
make clean to remove the instrumented object files.

Using AZRUN

You share the athena.dialup.mit.edu machines with all of MIT. Do not perform computationally
intensive tasks on these machines. Directly running and timing the execution on your own Azure
VMs may give inaccurate results; you will be running a small VM, and there might also be
measurement errors due to interference with your editor or other programs you are running. To
get an accurate timing measure on a dedicated machine, you can use the AZRUN utilities.
When you are ready to do a performance run you can use
$ azrun ./matrix_multiply

and your job will be queued. The command will not return until the job has been completed and
you get results unless you control-C twice (canceling the process). Once the job is complete, your
output should like something like below:
$ azrun ./matrix_multiply
Submitting Job: ./matrix_multiply
Waiting for job to finish...
==== Standard Error ====
Setup
Running matrix_multiply_run()...

==== Standard Output ====

Elapsed execution time: 0.000001 sec

Handout 2 Homework 1: Getting Started

10

Compiler optimizations

To get an idea of the extent to which compiler optimizations can affect the performance of your
program, increase the size of all matrices to 1000 1000. Rebuild your program in debug
mode and run it with AZRUN. Rebuild it again with optimizations (just type make), and run
it with AZRUN. Both versions should print timing information, and you should verify that the
optimized version is faster.

Write-up 4: Report the execution time of your programs compiled in debug mode with -O0
and in non-debug mode with -O3.

11

Performance enhancements

Now lets try one of the techniques from the first lecture. Right now, the inner loop produces a
sequential access pattern on A and skips through memory on B.
Lets rearrange the loops to produce a better access pattern. First, you should run the program
as is with optimizations to get a performance measurement. Next, swap the j and k loops, so
that the inner loop strides sequentially through the rows of the C and B matrices. Rerun the
program, and verify that you have produced a speedup. Commit and push your changes to the
Git repository.

Write-up 5: Report the execution time of your programs before and after the optimization.

12

C style guidelines

Code that adheres to a consistent style is easier to read and debug. Google provides a style guide
for C++ which you may find useful:
$ https://google.github.io/styleguide/cppguide.html

We have provided a Python script clint.py, which is designed to check a subset of Googles
style guidelines for C code. To run this script on all source files in your current directory use the
command:
$ python clint.py *

The code the staff provides has no style errors. We suggest, but do not require, that you
use this tool to clean up your source code. Part of your code-quality grade on projects is based

Handout 2 Homework 1: Getting Started

on the readability of your code. It can be difficult to maintain a consistent style when multiple
people are working on the same codebase. For this reason, you may find it useful to use the style
checkers provided by the 6.172 staff during your group projects to keep your code readable.

13

Submission

Submit the write-up as described in Write-ups 15 to the Stellar website. Then, if you havent
already, commit and push your final submission to the repository.
$ git commit -am Done!
$ git push origin master