Debugging concurrency programs in go

Debugging concurrent
programs in Go

Andrii Soldatenko
• Gopher, OSS
contributor
• father of 👧 👶
• debuggers fan
• Speaker at many
conferences
@a_soldatenko

Concurrency Programming
is Challenging!

Debugging
concurrent
programs is Hard!

@a_soldatenko
Debugging sequential
programs
dlv test -- -test.run TestFibonacciBig
(dlv) b main_test.go:6
Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/
debug_test.TestFibonacciBig() ./main_test.go:6
(dlv) c
> github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./
main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f)
1: package main
2:
3: import "testing"
4:
5: func TestFibonacciBig(t *testing.T) {
=> 6: var want int64 = 55
7: got := FibonacciBig(10)
8: if got.Int64() != want {
9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d,
want: %d", 10, got.Int64(), want)
10: }
11: }
(dlv)

@a_soldatenko
Debugging concurrent
programs
package main
import (
"fmt"
"time"
)
func say(s string) {
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
go say("world")
say("hello")
}
gowayfest git:(master) ✗ go run main.go
hello
world
world
hello
hello
world
world
hello
world
hello
gowayfest git:(master) ✗ go run main.go
world
hello
hello
world
hello
world
world
hello
hello
world

Debugging concurrency programs in go

What is a goroutine?
https://tpaschalis.github.io/goroutines-size/

How can I debug
concurrent Go program?

visualize goroutines🐞 ?
func main() {
c := coloredgoroutine.Colors(os.Stdout)
fmt.Fprintln(c, "Hi, I am go routine", goid.ID(), "from main routine")
count := 10
var wg sync.WaitGroup
wg.Add(count)
for i := 0; i < count; i++ {
i := i
go func() {
fmt.Fprintln(c, "Hi, I am go routine", goid.ID(), "from
loop i =", i)
wg.Done()
}()
}
wg.Wait()
}

https://github.com/xiegeo/coloredgoroutine

https://divan.dev/posts/go_concurrency_visualize/

Print scheduling events?
$ GODEBUG=schedtrace=5000 <binary>

@a_soldatenko
- delve
- gdb
using debuggers:

Advanced debugging techniques
of Go code

@a_soldatenko
How to set breakpoint
inside goroutine?package main
import (
"fmt"
)
func say(s string, r chan string) {
fmt.Println(s)
r <- s
}
func main() {
chan1 := make(chan string)
chan2 := make(chan string)
go say("world", chan1)
go say("hello", chan2)
res1 := <-chan1
res2 := <-chan2
fmt.Printf("Channel 1: %snChannel 2: %sn", res1, res2)
}

@a_soldatenko
> main.say() ./main.go:9 (hits goroutine(7):1 total:1) (PC:
0x10c46c9)
4: "fmt"
5: )
6:
7: func say(s string, r chan string) {
8: fmt.Println(s)
=> 9: r <- s
10: }
11:
12: func main() {
13: chan1 := make(chan string)
14: chan2 := make(chan string)
How to set breakpoint
inside goroutine?

@a_soldatenko
How to debug channel?
package main
func main() {
ch := make(chan int, 4)
ch <- 1
ch <- 2
ch <- 3
ch <- 4
close(ch)
}

@a_soldatenko
How to debug channel?(dlv) p ch
chan int {
qcount: 0,
dataqsiz: 4,
buf: *[4]int [0,0,0,0],
elemsize: 8,
closed: 0,
elemtype: *runtime._type {size: 8, ptrdata: 0, hash: 4149441018, tflag: tflagUncommon|tflagExtraStar|
tflagNamed|tflagRegularMemory (15), align: 8, fieldAlign: 8, kind: 2, equal: runtime.memequal64, gcdata: *1,
str: 663, ptrToThis: 22432},
sendx: 0,
recvx: 0,
recvq: waitq<int> {
first: *sudog<int> nil,
last: *sudog<int> nil,},
sendq: waitq<int> {
first: *sudog<int> nil,
last: *sudog<int> nil,},
lock: runtime.mutex {key: 0},}
(dlv) n
> main.main() ./main_ch.go:7 (PC: 0x105e8d9)
2:
3:
4: func main() {
5: ch := make(chan int, 4)
6: ch <- 1
=> 7: ch <- 2
8: ch <- 3
9: ch <- 4
10: close(ch)
11: }
(dlv) p ch

@a_soldatenko
dlv send to channel value
similar to set variable.

@a_soldatenko
GDB
t.me/golang_for_two

@a_soldatenko
Gdb and golang
go build -ldﬂags=-compressdwarf=false -
gcﬂags=all="-N -l" -o main main.go

@a_soldatenko
Gdb and goroutines

@a_soldatenko
Deadlocks happen and are
painful to debug.

@a_soldatenko
How to detect deadlocks
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/Users/andrii/workspace/src/github.com/andriisoldatenko/
gowayfest/main_deadlock.go:5 +0x50
exit status 2

@a_soldatenko
Real world examples
complicated scenario & tools
https://github.com/sasha-s/go-deadlock
https://github.com/cockroachdb/cockroach/issues/7972

@a_soldatenko
7 simple rules for debugging
concurrency applications
- Never assume a particular order of execution
- Implement concurrency at the highest level possible
- Don’t forget Go only detects when the program as a
whole freezes, not when a subset of goroutines get
stuck.
- STRACE

@a_soldatenko
7 simple rules for debugging
concurrency applications
- conditional breakpoints your best friend
- DEBUG=schedtrace=5000
- go-deadlock

@a_soldatenko
References 1
- https://github.com/golang/go/blob/release-
branch.go1.14/src/runtime/HACKING.md
- https://github.com/golang/go/wiki/LearnConcurrency
- https://rakyll.org/go-cloud/
- https://yourbasic.org/golang/detect-deadlock/

@a_soldatenko
References 2
- https://blog.minio.io/debugging-go-routine-leaks-
a1220142d32c
- https://golang.org/src/cmd/link/internal/ld/dwarf.go
- https://golang.org/src/runtime/runtime-gdb.py
- https://cseweb.ucsd.edu/~yiying/GoStudy-
ASPLOS19.pdf
- https://golang.org/doc/articles/race_detector.html

Telegram channel
https://t.me/golang_for_two

Debugging concurrency programs in go

Recommended

More Related Content

What's hot (20)

Similar to Debugging concurrency programs in go (20)

More from Andrii Soldatenko (12)

Recently uploaded (20)

Debugging concurrency programs in go