Monitoring and Debugging your Live Applications

Editor's Notes

• #4: When we encounter a problem with our live applications (on the web or elsewhere), we often have no opportunity to follow the normal debugging process through to its conclusion. The app needs to get back up, stat! And trying to piece together logs afterwards is hard.
• #5: I believe in logging virtually everything - if it helps during development it’ll probably help during debugging. Certainly beats “turning up” or adding logging, restarting the app, and waiting for the problem to happen again.
• #6: who logs like this?
• #7: What about this? Standard error is good, right? Separates our logging from any real output...
• #8: the problem with these is you end up commenting everything just to make it shut the hell up. And then have to uncomment to start talking again. http://www.flickr.com/photos/larimdame/2575986601
• #9: whoa? an actual logging framework? Yes, Python has a powerful builtin logging framework with the logging module, we should use it.
• #10: Logging separates the source of the messages from the output. So you can turn up and down the level in configuration (live even), or point logs somewhere else, without ever going back into the code itself.
• #11: Python’s logging module gets a real bad rap about the place... http://www.flickr.com/photos/theeerin/1108095143
• #12: Lots of people think it’s based too closely on log4j. How many ways are there to specify log.debug? It’s actually a pretty clean design that works fast. http://www.flickr.com/photos/lumaxart/2136948489
• #13: Others say its too hard. Seriously, 3x lines of code to get file logging going for an entire app is not “too hard”
• #14: The first key component of logging is a Logger, which generates log messages. Loggers are named by the developer, usually based on where the code is. The names form a hierarchy using the dots. Then we have the debugging methods at different levels. For the exception() one it’ll just pick up on the current exception and log a traceback all by itself. And all the logging methods accept % formatting via their arguments, so we don’t have to build strings just to chuck them away.
• #15: Handlers do something with log messages. They might write it to the console, to a file, send to a web server, post an email, stick it in Syslog or the NT event log, or a multitude of other things. They can filter out which messages they apply to, so we can log everything to a file as well as email just the critical errors.
• #16: The last one is a formatter. What are our log messages going to look like? The formatter has access to the code location where the log message came from, as well as the process, thread, the time, and so on.
• #17: So lets start with an example. We have our “machine” application, which wants to log different things to various places.
• #18: So in our code, we don’t need much. Somewhere we call fileConfig to set things up, pointing to our ini-style config. After that we just get logger objects by name, then call methods on them.
• #19: so, we want to send everything to a file by default. This bit of config will do that. We set up a format for our log lines. Since the loggers form a hierarchy, the root one is at the top and everything floats up there eventually. And we tell it to use a FileHandler pointing to our log.
• #20: Next, startup/shutdown messages to syslog. Anything logged by a logger named machine.startstop.something will end up here. We use a SysLogHandler to get it into syslog for us.
• #21: We want to squash the debug & info messages for three libraries: noisy, chatty, loud. We do this by setting propagate=0, which prevents the messages from bubbling up to the root logger. And we have no handler, so they don’t go anywhere else.
• #22: Then we want to send core errors (machine.core.anything) by email. We use level=error to filter out just the errors. And then we use a SMTPHandler to log via email.
• #23: And that’s prettymuch logging. There are some other tools I’m fond of, since you’re logging everything... Syslog-ng can help pull logs from multiple machines into one place. ack & less are the awesome duo of finding anything in text. And Splunk is a great tool for matching and tracing lots of events across multiple servers.
• #25: (DEMO 4-1.py)
• #26: The next thing in our debugging arsenal is a remote console. Don’t you wish you could just look inside your app and find out what it’s thinking, rather than hypothesising? Remote consoles certainly beats trial and error, or reconstructing chains of events from log files. We have the technology to just ask it :) http://www.flickr.com/photos/pasukaru76/3959355664/
• #27: Who’s heard of Twisted? The twisted library does lots of magic things, but one of the cool ones is a telnet or ssh server that gives access into a running python environment. You can read and change variables, call methods, whatever you want. (DEMO 2-1.py)
• #28: And the code is pretty simple. Basically we create a dictionary object we use as a namespace for where the user ends up. We specify an IP and a port (and encryption keys if you want an ssh shell), and a file with usernames and passwords. that’s it. At least, if you have a Twisted app.
• #29: Twisted is an asynchronous framework, it’s single threaded with an event loop, and doesn’t work in the same way most Python code does. But we can run the Twisted part (which does our manhole) in a separate thread, leaving our normal code alone. The only downside? we have to handle concurrency if we’re changing things :) (DEMO 2-2.py)
• #30: The 3rd idea i want to talk about is robots. We can review our app’s progress with logs. We can telnet right inside it for hardcore inspection. But we can also get it talking to us via instant messaging. http://saiogaman.deviantart.com/art/Danbo-Wallpaper-107237965
• #31: everybody runs IM of some sort, its already open on the desktop, it just works. Our apps can say to us, in real time: “hey, Rob, I’m not happy”. Plus, the marketing guy down the hall might like to get pinged if someone orders 100 copies of our awesome software. Bots can do this.
• #32: Among its arsenal of networking awesomeness, twisted also does XMPP (also known as Jabber). It’ll also do IRC and some other stuff, but we’ll focus on XMPP for now. Basically it’s an asynchronous protocol for passing bits of XML around. No polling required. There’s a library called Wokkel which makes XMPP even easier than Words does. (DEMO 3-1.py)
• #33: This is the talk-ey part of the code for that example. It gets a message object in, then composes and sends a reply. Do whatever you want in these methods. And your app can call .send() itself anytime to send messages.
• #34: And this is basically what I’d like you to take away. Arm yourself with some great tools and debugging and keeping an eye on your apps will be a lot easier. Trial and error sucks, move on!