Building Observable Applications w/ Node.js -- BayNode Meetup, March 2014
8 likes4,690 views
The document outlines a presentation by Yunong Xiao on observable applications using Node.js, discussing issues like increased errors and latency, and presenting tools such as vasync and bunyan for improved observability and error management. It emphasizes the integration of these tools with restify for better logging and performance in production environments, enabling effective monitoring and debugging. The presentation also highlights various unix tools that can be used to process and analyze JSON logs generated by these frameworks.
Building Observable Applications w/ Node.js -- BayNode Meetup, March 2014
- 1. Observable Applications w/ Node.js Yunong Xiao, Senior Node.js Engineer, UI Platform, Netflix @yunongx, yunong@netflix.com March 2015 BayNode The Bay Area Node.js Meetup
- 2. Node.js @ Netflix • Website www.netflix.com • Discovery • Playback • Acquisition • Account Management • Internal Services • TV/Mobile/Devices
- 4. Increased Errors Increased Latency Memory Leak
- 5. What Now? “It is a capital mistake to theorize before one has DATA. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” Sherlock Holmes -A Scandal in Bohemia
- 12. vasync • https://github.com/davepacheco/node-vasync • Similar to caolan/async. • Added observability.
- 13. async f() state failed f() successful f() pending f() finished f() # of errors
- 14. vasync • Make API requests. • Persist state to file system. • Query database. • Update caches. Async workflow management is error prone
- 17. How Do I See this in Prod? • Logs • Core Dumps • REPL • HTTP API
- 18. node-bunyan • https://github.com/trentm/node-bunyan • Streaming JSON logging library for JS. (node and browser)
- 19. Streaming JSON • One JSON object per line. e.g. Twitter’s streaming API. • Perfect for machine processing. i.e. works with Unix tools such as grep(1), cut(1), … • Illegible for Humans
- 20. Streaming JSON
- 21. bunyan(1) CLI
- 22. Features • Lightweight API • Log levels: trace, debug, info, warn, error, fatal • Extensible Streams interface. • Custom object rendering with serializers. • DTrace support.
- 23. Demo
- 24. DTrace • Dynamic tracing framework. • Available on Mac OS X (dev env). `man dtrace` • Node.js DTrace USDT provider: https:// github.com/chrisa/node-dtrace-provider • Pure JS probes.
- 25. bunyan -p Demo
- 26. Production Tracing • Production environment is Ubuntu Linux on EC2. • Future work: get probe access. Possibilities: • SystemTap • LTTng (Linux Tracing Toolkit Next Gen) • perf_events markers (currently a proposed patch)
- 27. restify • http://restifyjs.com • Production tested REST framework. • Observability. • Metrics. • First class Bunyan integration. • DTrace.
- 28. Vanilla App
- 29. Vanilla App
- 30. Logging
- 31. Request Capture Stream • Tradeoffs, Perf vs Verbosity. • Captures all log statements including trace in memory. • Dump all logs only on error.
- 32. Audit Logs
- 33. Audit Logsstatus code request ID (UUID) URL request headers response headers individual handler timers req latency
- 36. Logging • Native Bunyan integration. • Request capture stream. • Audit logs with each req. • Scoped child loggers with each req.
- 37. restify + Bunyan • Streaming JSON. • Processing using Unix tools is easy. • Helpful tools: • Unix: cut(1), wc(1), grep(1), awk(1), perl(1), … • JSON: https://github.com/trentm/json (npm install -g json) • daggr: https://github.com/joyent/daggr (npm install -g daggr)
- 38. Examples • Find all logs for a specific request. • Count the # of non-200 responses. • Show all requests that took longer than 200ms.
- 39. Advanced Example Request latency distribution by URL
- 40. DTrace
- 41. Demo
- 42. distributed processing • elasticsearch • spark • hive bunyanrestify audit logs req id req headers req latency handler latencies URL res codeErrors res headers Processing w/ Unix tools • grep(1) • awk(1) • cut(1) • wc(1) • sort(1) • json(1) • bunyan(1) • … req scoped logs application specific context req capture stream dumps all logs on error DTrace* req latency handler latencies * Where available restify metrics
- 43. Increased Errors Increased Latency
- 44. restify Interested? Call for contributors. restifyjs.com
- 45. The Killer Combo • vasync: https://github.com/davepacheco/node-vasync • Observable async operations. • bunyan: https://github.com/trentm/node-bunyan • Streaming JSON logs. • restify: restifyjs.com • Observable REST applications. • Unix Tools • Stream of text model. Many tools to manipulate JSON logs.
- 46. Data Data Data
- 47. Thanks • Questions? • yunong@netflix.com • @yunongx • http://restifyjs.com March 2015 BayNode The Bay Area Node.js Meetup