Networking and Go: An Engineer's Journey (Strangeloop 2019)

digitalocean.com
Networking and Go
An Engineer’s Journey

digitalocean.com
Hi!
I’m Sneha.
I’m a software engineer at DigitalOcean.

digitalocean.com
DigitalOcean
cloud-hosting company
12 data centers
1.15 million droplets
k8saas, dbaas, storage, VPC

digitalocean.com
2016:
Joined DigitalOcean
“What is Go?”

digitalocean.com
Designed as a systems-language, used
in a lot of verticals.

digitalocean.com
Go is a statically-typed, memory-safe,
compiled programming language.

digitalocean.com
Excellent concurrency support

digitalocean.com
Goroutines
lightweight processes multiplexed
onto threads

digitalocean.com
SYNC
package with concurrency primitives
such as mutexes, thread-safe maps,
actor goroutine groups

digitalocean.com
CHANNELS
a conduit through which you can
send and receive data with a typed
operator

digitalocean.com
Syntactically simple

digitalocean.com
Enforces a coding style

digitalocean.com
2016:
Joined DigitalOcean
“What is go?”
2017:
Developer tooling
k8s, Prometheus

digitalocean.com
Microservices are a small, loosely-coupled
and independently-deployable collection
of services.

digitalocean.com
Let’s containerize these services.

digitalocean.com
Let’s manage and schedule these
containers.

digitalocean.com
DOCC: an abstraction layer on top of
kubernetes (k8s).

digitalocean.com
Okay, we need some observability.

digitalocean.com
Tracing, logging, and metrics.

digitalocean.com
Pandora: a functionally sharded,
data-center wide Prometheus setup.

digitalocean.com
2016:
Joined DigitalOcean
“What is go?”
2017:
Developer tooling
k8s, Prometheus
2018:
Networking Team
DHCP, Gateways

digitalocean.com
What is the OSI model?

digitalocean.com
What is a socket?

digitalocean.com
DHCP, ARP, NDP Protocols

digitalocean.com
2016:
Joined DigitalOcean
“What is go?”
2017:
Developer tooling
k8s, Prometheus
2018:
Networking Team
DHCP, Gateways
2019:
MORE
LB, port scanner

digitalocean.com
How and when to use
Go to build networking
primitives.

digitalocean.com
The plan for today

digitalocean.com
▣ why use Go
⬚ the networking stack
⬚ networking primitives
LB: Layer 7 vs. 4
Port scanner: Layer 4
DHCP server

digitalocean.com
Why use Go for networking services?

digitalocean.com
Concurrency
EASY TO FUZZ-TEST
SERVER-SIDE
GREAT FOR CLI
TOOLS
EASY TO WRITE
REST/RPC SERVICES

digitalocean.com
But when might Go NOT be the answer?

digitalocean.com
High-performance packet-processing

digitalocean.com
Why did we use Go?

digitalocean.com
DigitalOcean has a monorepo and
shared packages.

digitalocean.com
hvflowd
hvaddrd
OvS
br0
hvflowctl
RNS
AddFlows
OpenFlow
SetParameters
bond0
ethX
ethX addr0
bolt
DHCPv4 NDP
gRPC
DHCPv6
tapX dropletX
hvaddrd traffic
hvaddrctl
AddFlows
internet traffic
Hypervisor
main
Using gRPC and CLI tools

digitalocean.com
Performance was already good enough.

digitalocean.com
▣ why use Go
▣ the networking stack
⬚ networking primitives
LB: Layer 7 vs. 4
DHCP server

digitalocean.com
OSI: Open-systems Interconnect Model

digitalocean.comSource: https://www.researchgate.net/ﬁgure/OSI-model-seven-layer-protocol-stack-28_ﬁg6_322568288

digitalocean.com
Source: https://searchnetworking.techtarget.com/deﬁnition/TCP-IP

digitalocean.com
The Layers: Data Encapsulation

digitalocean.com
Source: https://www.aboutdebian.com/network.htm

digitalocean.com
socket: internal endpoint for sending or
receiving data on a computer network

digitalocean.com
How do we read a segment?

digitalocean.com
TCP (stream socket): provides a
connection-oriented, sequenced ﬂow of data

digitalocean.com
UDP (datagram socket): provides a
connectionless ﬂow of data

digitalocean.com
How do we read a packet?
How do we read a frame?

digitalocean.com
raw socket: a socket that allows sending and
receiving of data without protocol-speciﬁc
transport layer formatting

digitalocean.com
raw socket: IP layer (3)
socket(AF_INET,SOCK_RAW,protocol)
packet socket: link layer (2)
socket(AF_PACKET,SOCK_RAW,protocol)

digitalocean.com
bpf (berkeley packet ﬁlter) interface:
raw link-layer interface on most
unix-based systems

digitalocean.com
bpf ﬁlters also used with raw sockets
or tcpdump

digitalocean.com
Networking Protocols

digitalocean.com
Source: https://www.quora.com/What-are-some-recommendation-to-fully-understand-networking-protocols

digitalocean.com
“The best way to learn is simply
by doing.”
--- Me

digitalocean.com
▣ when to use Go
▣ the networking stack
▣ networking primitives
LB: Layer 7 vs. 4
DHCP server

digitalocean.com
Layer 7: Loadbalancer

digitalocean.com
Loadbalancer (LB)
A device distributing traﬃc
across a number of backends.
network deﬁnition
Source: https://gbhackers.com/load-balancer-reverse-proxy/

digitalocean.com
Building a layer-7 LB
1. Use HTTP protocol.
2. Accept client-side connections.
3. Pass client-side request to one of the
backends.
4. Return server-response back to the client.

digitalocean.com
HTTP:
HyperText Transfer Protocol
An application layer (7) protocol:
● request-response based
● stateless
● Has diﬀerent methods (GET,
PUT, etc)
● Uses TCP or UDP sockets
under-the-hood
network deﬁnition
Source:
https://www.ntu.edu.sg/home/ehchua/programming/webprogramming/HTTP_Basic
s.html

digitalocean.comGO FEATURE
net/http

digitalocean.com
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
...
})
log.Fatal(http.ListenAndServe(":8080", nil))
1. ListenAndServe

digitalocean.com
n := rand.Intn(len(backends))
r.URL.Host = backends[n]
r.URL.Scheme = "https"
req, err := http.NewRequest(r.Method, r.URL.String(), r.Body)
...
res, err := client.Do(req)
...
}
2. Read and send request to backend

digitalocean.com
...
w.WriteHeader(res.StatusCode)
_, err = io.Copy(w, res.Body)
if err != nil {
log.Printf("error writing response to client: %v", err)
}
...
}
3. Return response to client

digitalocean.com
Layer 4: Loadbalancer

digitalocean.com
Building a layer-4 LB:
1. Use TCP protocol (via a streaming socket).
2. Accept client-side connections.
3. Open backend connection.
4. Use one goroutine to shuttle packets from the
client to the backend.
5. Use another goroutine to shuttle packets from
the backend to the client.

digitalocean.com
TCP: Transmission
Control Protocol
Connection-oriented protocol
with 3 phases:
● Connection establishment
(TCP handshake)
● Data transfer
● Connection termination
network deﬁnition
Source:
https://www.researchgate.net/figure/Three-way-Handshake-in-TCP-Connection-
Establishment-Process_fig7_313951935

net

github.com/oklog/run

digitalocean.com
listener, err := net.Listen("tcp","127.0.0.1:9090")
if err != nil {
log.Fatal(err)
}
for {
conn, err := listener.Accept()
if err != nil {
log.Printf("error accepting client conn: %v", err)
continue
}
go handleConn(conn)
}
1. Open and bind client TCP socket

digitalocean.com
func handleConn(clientConn net.Conn) {
n := rand.Intn(len(backends))
backendConn, err := net.Dial("tcp", backends[n])
if err != nil {
log.Printf("error %s: %v", backends[n], err)
return
}
...
}
2. Open and connect backend TCP socket

digitalocean.com
var g run.Group
{
g.Add(func() error {
return copy(clientConn, backendConn)
}, func(error) {
clientConn.Close()
backendConn.Close()
})
}
{
g.Add(func() error {
return copy(backendConn, clientConn)
}, ...)
}
err = g.Run()
3. Conﬁgure rungroups

digitalocean.com
func copy(from net.Conn, to net.Conn) error {
for {
readBytes := make([]byte, 1024)
n, err := from.Read(readBytes)
if err != nil {
...
}
if _, err = to.Write(readBytes[:n]); err != nil {
...
}
}
}
4. Read and copy

digitalocean.com
Loadbalancer IRL:
● Exact mechanism of handling connections and/or requests
depends on the layer.
● Layer-4 loadbalancers should be more eﬃcient:
○ Won’t necessarily create a new backend connection per
client conn.
○ Use NAT and munging incoming packets.
● Better loadbalancing algorithms.
● Client stickiness.

digitalocean.com
Layer 4: Port Scanner

digitalocean.com
Nmap
Linux utility that scans hosts and
attempts to determine open
UDP/TCP ports
network deﬁnition
Source: https://commons.wikimedia.org/wiki/File:Screenshot_Nmap.png

sync

digitalocean.com
Building a TCP port-scanner
1. Select range of ports to scan.
2. Try to open and connect a TCP socket to a
remote address (and port)..
3. Print results.

digitalocean.com
addr := fmt.Sprintf("%s:%d", hostname, port)
conn, err := net.DialTimeout("tcp", addr, timeout)
if err != nil {
fmt.Printf("port %d closed: %vn", port, err)
} else {
fmt.Printf("port %d openn", port)
conn.Close()
}
1. Open and connect TCP socket

digitalocean.com
conSema := make(chan struct{}, 10)
var wg sync.WaitGroup
for i := 1; i <= 65535; i++ {
wg.Add(1)
go func(port int) {
conSema <- struct{}{}
...
wg.Done()
<-conSema
}(i)
}
wg.Wait()
2. Use waitgroup and channels

digitalocean.com
Nmap IRL:
● Checks UDP and TCP ports on local or remote hosts.
● Host discovery
● OS detection
● Auditing security of a ﬁrewall

digitalocean.com
Layer 2: DHCP Server

digitalocean.com
DHCP is technically layer-7.

digitalocean.com
DHCP Protocol
● Dynamic Host Configuration
Protocol is used by routers to
allocate IP addresses to
network interfaces
● DHCPv6 uses NDP and
DHCPv4 uses ARP
network deﬁnition
Source: https://study-ccna.com/dhcp-dns/

digitalocean.com
Need link layer MAC addresses.

digitalocean.com
Building a DHCP server:
1. Open and bind a raw socket to an interface.
2. Read data from socket into bytes buﬀer.
3. Unmarshal into DHCP message and retrieve sender
hardware address.
4. Switch between handlers based on message type.
5. Validate DHCP request message and craft response.
6. Unicast response back to sender.

github.com/mdlayher/raw

filename := "eth-packet-socket"
typ := unix.SOCK_RAW
if cfg.LinuxSockDGRAM {
filename = "packet-socket"
typ = unix.SOCK_DGRAM
}
sock, err := unix.Socket(unix.AF_PACKET, typ, 0)
...
f := os.NewFile(uintptr(sock), filename)
sc, err := f.SyscallConn()
...
n, addr, err = unix.Recvfrom(int(fd), p, flags)
github.com/mdlayher/raw/raw_linux.go

// Try to find an available BPF device
for i := 0; i <= 10; i++ {
bpfPath := fmt.Sprintf( "/dev/bpf%d", i)
f, err = os.OpenFile(bpfPath, os.O_RDWR, 0666)
if err == nil {
break
}
if perr, ok := err.(*os.PathError); ok {
if perr.Err.(syscall.Errno) == syscall.EBUSY
{
continue
}
}
return nil, err
}
github.com/mdlayher/raw/raw_bsd.go

syscall
os
golang.org/x/sys/unix

digitalocean.com
ifi, err := net.InterfaceByName(iface)
if err != nil {
return nil, err
}
pc, err := raw.ListenPacket(ifi, uint16(ethernet.EtherTypeIPv4),
&raw.Config{
// Don't set any timeouts to avoid syscall busy
// loops, since this server will run forever anyway.
NoTimeouts: true,
})
1. Open and bind raw socket

digitalocean.com
b := make([]byte, 1500)
for {
n, from, err := s.pc.ReadFrom(b)
if err != nil {
...
continue
}
buf := make([]byte, n)
copy(buf, b)
workC <- request{
buf: buf,
from: from,
}
}
2. Read data into buﬀer

digitalocean.com
var workWG sync.WaitGroup
workWG.Add(Workers)
workC := make(chan request, Workers)
for i := 0; i < Workers; i++ {
go func() {
defer workWG.Done()
for r := range workC {
s.serve(r.buf, r.from)
}
}()
}
3. Worker handles request

digitalocean.com
func (h *handler) serveDHCPv4(ctx context.Context, req
*dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr)
{
...
if !from.EthernetSource.Equal(reqMAC) {
...
return h.shouldNAK(req.Type, to, broadcast)
}
...
}
4. Validate request

digitalocean.com
func (h *handler) serveDHCPv4(ctx context.Context, req
*dhcp4.Packet, from *dhcp4conn.Addr) (*dhcp4.Packet, net.Addr) {
...
res, err := h.buildResponse(ctx, params, req, from)
if err != nil {
topError(span, err, "failed to build response")
return h.shouldNAK(req.Type, to, broadcast)
}
res.Broadcast = broadcast
return res, to
...
}
5. Build and send response

digitalocean.com
DHCP Summary:
1. DHCP: dynamic host conﬁguration protocol
used to dynamically assign IP address.
2. Use raw sockets.
3. Jump down to link-layer to do source MAC
validation.

digitalocean.com
Go can be good for building networking
primitives.

digitalocean.com
Layer 7:
net/http package

digitalocean.com
Layer 4:
net package

digitalocean.com
Layer 3/2:
raw package
os,syscall
golang.org/x/sys/unix

digitalocean.com
sync package
github.com/oklog/run package
goroutines
channels

THANKS :)
@mdlayher
@juliusvolz
@nicboul

digitalocean.com
Ohai!
@snehainguva
sneha.inguva@gmail.com

digitalocean.com
Sources
● Port Scanner: https://github.com/si74/portscanner
● Layer 7 loadbalancer: https://github.com/si74/layer7lb
● Layer 4 loadbalancer: https://github.com/si74/tcpproxy
● Raw package: https://github.com/mdlayher/raw
● Godocs: https://godoc.org/
● Golang: https://godoc.org/

Networking and Go: An Engineer's Journey (Strangeloop 2019)

Recommended

More Related Content

What's hot (20)

Similar to Networking and Go: An Engineer's Journey (Strangeloop 2019) (20)

More from Sneha Inguva (8)

Recently uploaded (20)

Networking and Go: An Engineer's Journey (Strangeloop 2019)