Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
0 likes•1,004 views
This document discusses using traffic control to test microservices applications running in containers. It describes using tools like Kubernetes, Weave Scope, and netem to configure network emulation scenarios like bandwidth limiting, latency injection and packet dropping. A demo application is shown running in Kubernetes pods with Weave Scope visualizing network traffic and a traffic control plugin modifying network behavior for testing purposes. Advanced filtering options using eBPF and custom plugins are proposed to define more complex traffic classes and collect test statistics.
Testing applications with traffic control in containers / Alban Crequy (Kinvolk)
- 1. In containers Alban Crequy Testing applications with traffic control HighLoad++ - November 2016https://goo.gl/knaz6m
- 2. Alban Crequy ∘ Worked on traffic control for multimedia applications in cars (tcmmd) ∘ Worked on the container run-time rkt ∘ Working on Weave Scope https://github.com/alban
- 3. Berlin-based software company building foundational Linux technologies Some examples of what we worked on... OSTreegit for operating system binaries
- 5. Find out more about us… Blog: https://kinvolk.io/blog Github: https://github.com/kinvolk Twitter: https://twitter.com/kinvolkio Email: [email protected]
- 6. ∘ What is traffic control and how does it work on Linux ∘ How it can be used for testing a microservices application ∘ Demo ∘ With Minikube, Kubernetes, Weave Scope, Weave Socks Plan
- 7. What is traffic control? How does it work on Linux?
- 8. Traffic control, why? web server client client client THE INTERNET ∘ fair distribution of bandwidth ∘ reserve bandwidth to specific applications ∘ avoid bufferbloat
- 9. ∘ Network scheduling algorithm ∘ which packet to emit next? ∘ when? ∘ Configurable at run-time: ∘ /sbin/tc ∘ Netlink ∘ Default on new network interfaces: sysctl net.core.default_qdisc Queuing disciplines (qdisc) eth0 THE INTERNETqdisc
- 10. Stochastic Fairness Queueing (sfq) eth0 THE INTERNET FIFO n FIFO 1 FIFO 0 ... round robin
- 11. Traffic control for testing?
- 12. Developers & Internet Developers usually have good Internet connection - Low latency - High bandwidth - Not much packet loss or packet corruption Difficult to see problems
- 13. Network emulator (netem) eth0 THE INTERNETnetem bandwidth latency packet loss corrupt ...
- 14. Testing with containers container 1 container 2 eth0eth0 Testing framework configure “netem” qdiscs: bandwidth, latency, packet drop...
- 18. Kubernetes
- 19. Pods Group of container(s) running together, in the same context Kubernetes node 1 Kubernetes node 2 pod container container pod container pod container pod container Each pod has - Its own network namespace - Its own IP address
- 20. Replica sets - Control the number of “Pods” replicas running - Deploy pods Kubernetes node 1 Kubernetes node 2 pod catalogue pod front-end pod orders In this example: - 2x front-end - 1x catalogue - 1x orders pod front-end
- 21. Replica set “catalogue” Services Directing the traffic to “Pods” - Cluster IP - DNAT catalogue1 catalogue2 catalogue3 Service “catalogue” front-end
- 22. Testing with traffic control in Kubernetes Kubernetes node 1 pod pod Kubernetes node 2 pod pod tc tc controls ∘ Latency ∘ Bandwidth ∘ Packet drop ∘ configure network simulator ∘ play scenarios
- 23. Weave Scope
- 24. Weave Scope
- 25. Testing with Weave Scope Kubernetes node 1 tc Scope Probe pod pod pod pod Kubernetes Node 2 tc Scope Probe pod pod pod pod Scope App
- 26. <Demo> Reproduce the demo yourself: https://github.com/kinvolk/demo
- 47. </Demo> Reproduce the demo yourself: https://github.com/kinvolk/demo
- 48. Testing framework for web apps Selenium
- 49. Plugins in Scope tc plugin Scope Probe Scope App report reportcontrol control - Unix socket in /var/run/scope/plugins/ - Protocols: report and control - Write your own plugins for your testing needs
- 51. Testing more complex scenarios (my “wishlist”)
- 52. Add latency on a specific connection front-end Firefox catalogue ordersorders-db payment latency=100ms
- 53. How to define classes of traffic eth0 netem interface latency=100ms dest_ip=10.0.4.* dest_ip=10.0.5.* other
- 54. u32: filter on content eth0 HTB HTB HTBHTB HTB netemnetem netem interface root qdisc (type = HTB) root class (type = HTB) leaf qdiscs (type = netem) leaf classes (type = HTB) filters (type=u32) otherip=10.0.5.*ip=10.0.4.* latency=10ms
- 55. Filtering with cBPF/eBPF eth0 BPF netemnetem kernel userspace BPF_JMP... BPF_LD... BPF_RET... if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation clang... -march=bpf upload in the kernel: - bpf() - Netlink x86_64 code JIT compilation
- 56. eBPF maps eth0 BPF netemnetem kernel userspace x86_64 code eBPF map tc ∘ Build statistics ∘ Make them available to the testing framework
- 57. - Another Weave Scope plugin could handle more filters: u32 or eBPF Advanced tc plugin Another plugin Scope Probe Scope App control control eth0 u32, eBPF, ...
- 58. The End Try the demos yourself: https://github.com/kinvolk/demo Weave Scope’s traffic control plugin: https://github.com/weaveworks-plugins/scope-traffic-control The slides: https://goo.gl/knaz6m
- 59. Questions?