Measuring web performance. Velocity EU 2011

Editor's Notes

• #2: Good Afternoon. My name is Stephen Thair and I am a freelance webops manager and performance specialist based in London, UK. I am also the organiser for the London Web Performance Meetup community. My topic today is “measuring web performance” and before we drill down into the specifics of measuring web performance I have one piece of bad news… <click>
• #3: And not just wrong because of esoteric stuff like the observer effect, or even the accuracy of our measuring tools… it’s wrong because of one major reason… <click>
• #4: And that reason is the human brain… The human brain does not have a metronomic clock in tick tocking away to a regular beat like the clocks we use to “measure” web performance… <click>
• #5: The key here is “subjective” and “variable” – there is a lot of stuff that the “numbers” won’t and CAN’T tell you about how the user perceives the performance of your website… Subjective – because YOUR experience is not MY experience!!!And when we say that performance is variable what do we mean… Well, we’ve all heard about time “slowing down” under the effects of adrenaline in emergencies… so perhaps if we are visiting a website that particularly gets the adrenaline flowing (ahem) our perception of time might “slow down” and what is in reality a “fast” website might appear slow. Conversely, there is a psychological state called “flow” where we can “lose track of time” because we are involved in a task, perhaps playing an online game, where suddenly we find an hour or two has gone past without us being aware of it. But our perception of performance is variable on other ways, too <click> Different for different sites – for different users (Age, Gender, emotional state (“Is the train about to leave, I’m running late”?), culture, level of experience) – at different stages in the user journey (e.g Navigation browser vs Search vs checkout) Different devices? – Mobile vs wireless vs wired?
• #6: Actual = what your “numbers” say it is…Expected = what your user wanted it to be… for your website… at this moment in time… which is to say expectations are not fixed & immutable!Perceived = How long the user “thought it took” with their subjective and variable perception of time…Remembered = What they told their friends down the pub about your crap & slow (or awesome&fast) website! Stoyan’s Talk at Velocity “The Psychology of Performance” – highly recommended… So… <click> http://velocityconf.com/velocity2010/public/schedule/detail/13019"Satisfaction = perception minus expectation" - David Maister
• #7: So we have talked about the <click> “Subjective” nature of web performance but our challenge as developers, testers and WebOps is to devise ways to make the subjective…. <click> Objective… and measure it! So how can we do that? Well, science has been struggling with this problem of “subjective” and “objective” for centuries and have developed different techniques to apply to each… <click>
• #8: To look at subjective data we use <click> Qualitative Techniques which are commonly used in the Social Sciences…. <click> case studies, focus groups, interviews <click> etc. If some of these sound familiar that’s because Many of these are the kinds of tools that people from the User Experience world use in their UE labs… And it’s worth making the point that you can start “measuring performance” very early on in the software development lifecycle, even with paper-based or simple static HTML click models by seeing how long people take to choose, decide, navigate… or even simply how many “clicks” they take to achieve a given task (less clicks = “faster performance”). And that’s all I am going to say on the qualitative side of things because as Web professionals we generally prefer to look at “objective” measures… <click>
• #9: And objectively normally means Quantitatively – means we can use NUMBERS…. And bring our statistical tools to bear… And there are 7 techniques of “HOW” to measure website performance But before we dive into the “HOW” we need to talk about some other things <click>
• #10: And those other things are what Rudyard Kipling called his “honest serving-men” – the what, why, when etc.So firstly, in terms of “what” do you want to measure do you care about <click> “objects”
• #11: About “objects”… or pages… or the entire user journey? Even if we are talking about pages we have multiple metrics to chose from… <click>
• #12: At a page and object level you have multiple metrics you can choose from… but I generally find that there are 4 I really care about…
• #13: So what are they <click>TTFB – how fast is my back-end responding? RenderStart – when does the user start to get visual feedback from the page – remember, it’s about Perception… but it has got to be meaningful… e.g. not just a CSS background changeDOMContentLoaded – How soon can my developers start hooking up their fancy Javascript stuff to the DOM?Page (onLoad) – when have all the elements on the page been loaded (and I can start all my deferred resource loading via Ajax!). One “new” metrics you might have heard about it “above the fold time”… <click>
• #14: AFT is basically designed to be a “render complete” timing, or at least a “render of the static stuff in the viewable area of the page”AFT is a nice idea… but it’s implementation is troublesome at present… 4 mins to calculate at present… But for most sites AFT = PLT… and but according to Pat from Webpagetest he has seen it range from ½ PLT to 2x PLT… Personally, I really like using the screen capture videos for this and look at it in comparison to previous versions, competitors etc…
• #15: This sort of video comparison that you can create with webpagetest.org… but when you rely on human judgement you are back into the subjective, again…So what other metrics might we be interested in? <click>
• #16: We start with the raw metrics… and then move up into counts (which we normally show as histograms) into the statistical measures and finally into artificial summary metrics like Apdex (which I will explain in a second) All of this data can be sliced and diced in your datawarehouse… but keep in mind that you can easily run into gigabytes and terabytes of data for a high-volume website in a month… so plan carefully! Ok, so back to Apdex - what is apdex <click> ApdexCalculated “summary” metricsStatisticalMetricsMeanModeMedianStdDevCounts/HistogramsHistogramsRaw MetricsConnection TimeRender Start TimePage Load Time“Above the Fold” time etc
• #17: Apdex is simple – split the page load time for every visit to your website into 3 bucketsSatisfiedToleratedFrustratedAnd you get a score from 0 – 1 that represents an “overall” measure of your site’s performance, across all URL’s, across all the visits during the time period. So why do we need a “single number” metric like Apdex? <click>
• #18: Because web performance is multi-dimensional…. Multiple Metrics For Multiple URLS From Different (measurement)Locations Using Different Tools Across the (software)Lifecycle Over TimeAnd it gives you a great number to stick on the plasma screen in the Ops area and a nice number to stick on your weekly report to your boss…But beyond just these metrics on how long a page took to load there is something else we need to record… <click> and that is “context”…
• #19: “Context” is the metadata about the “numbers” we have collected. They are the key to EXPLAINING why the performance number recorded is “good or bad”…
• #20: <click> “context” is the metadata about the measurement you made… what browser, from what geographical location, over what type of network etc etc. Without context your performance data is meaningless…
• #21: Context helps us answer this question!!! We can see that the mode of the page load time is about 0.9sec but what about this cluster out at around 2.7 and 2.9 secs? Maybe they are from a group of customers in one location, or using an older browser etc… But back to our 6 honest serving men and let’s look at who and when… <click>
• #22: We want to measure performance across the lifecycle (SDLC) and different teams will need to use different techniques to get the different metrics they need… We’ll talk more about this as we go through each technique…
• #23: Where you chose to measure the your web performance depends on your objectives… what exactly are you trying to measure, at what stage in the lifecycle, synthetic or real-user? The further we move away from the origin server the more network latency begins to dominate… and the more contextual factors come into play… and hence <click> the signal-to-noise ratio increases…But why have I drawn a distinction between “real users” and “synthetic agents” like monitoring or performance test agents? <click>
• #24: Well, because there is quite a debate raging out there at present on the future of web performance measurement… Synthetic = the active monitoring from Site Confidence / Keynote / Gomez / Pingdom we all know and love…Real-User = measuring the performance of real user visitors to your website using tools like Atomic Labs Pion, Triometric, Coradiant, Tealeaf etc.
• #25: A lot of people have strong opinions about whether we should be measuring “real-user” performance, or whether we should be synthetically making requests/transaction to test our website, regardless of whether those “synthetic requests” come from real-browsers or browser emulating agents. My view is that people who say either/or are missing the SCIENCE behind two different techniques… and we can look to the scientific method to help us conceptualise the difference between the two.
• #26: And science talks about two different quantitative techniques for gaining knowledge about the world, or in this case, web performance… and that is the “Observational Study” versus “Experiment” <click>
• #27: Both seek to detect a relationship – “what is making this page load slowly?”Create the difference… keeping everything else the same… controlling the experimental factors (as much as possible)For example, what happens when I measure with a different browser… but keeping everything else the same?
• #28: Observational Studies = Real-User monitoringWe can only measure what the OCCURS NATURALLY in the sample population. If no one visits that URL for a while, how will we know that it’s broken or slow? So what do I mean by “confound variables”? Wll, what I mean is <click> Context!
• #29: In “real-user” performance measurement the USER’s define the context… all the of the variables that might affect the number that you measure. So… how can I get some control back and reduce the number of confounding variables? Run an Experiment! <click>
• #30: <click> Experiment = Synthetic testing where we request the page we measure… <click> and hence we get to Design our “experiment” <click> We chose what to measure, from what location, over a fixed bandwidth, using a known agent/browser, with a known frequency (which means a stable sample size which is important for statistic when comparing means etc from a different URLs) <click> as we seek to control the confounding variables (as much as we can) to we get <click> a lower signal to noise ratio and hopefully get better at understanding the “root cause”… <click> I said “Hopefully”…  <click> So which one is “better” RUM or Synthetic, Observational Study or Experiment <click>
• #31: So which one is better? <click> It depends on what you are trying to achieve… what’s your role… what’s your goal. Personally, if I am going to be woken up a 3am with an alert saying there is a problem with my website I’d like to have a higher degree to confidence in the alert than just because some ISP is having problems and giving their users a slow connection… I want to be alert about problems I can DO SOMETHING ABOUT and as an Ops Mgr I will “design my experiments” accordingly… But what about a “hybrid model” <click> where we move from RUM to Synthetic?... <click>
• #32: So how would Pat’s idea work? <click> The RUM to detect changes out there “in the real world”… <click> then pass the URL to test via an API <click> to try and narrow down the signal/noise in…(note we might be calling an entire SET of regression tests here… But the goal is <click> to Move from Observation <click> by controlling the variables <click> to a well defined ExperimentBut don’t forget you can also go the other way… to make sure that your “experiment” even vaguely reflects “reality” by cross-checking your synthetic results with what’s out there in the real world… which is exactly what any scientist does when they create an experimental model… they make sure that it correlates with reality!Ok, so we’ve covered off the who,what, when, where etc, lets get back to the “HOW”… <click>
• #33: Which is not to say that we can’t measure subjective things… qualitatively…
• #34: There are basically 7 techniques used to measure web performance:Each one has it’s pros and cons… easy of use, what it can measure, cost etc <click> x 7 So which technique is best? Depends on what you want to measure, where etc… comparing them all together we get <click>
• #35: So let’s look at each one in turn and how it works (in a very simplified way!)…
• #36: For example, the following JavaScript shows a naive attempt to measure the time it takes to fully load a page:<html><head><script type="text/javascript">var start = new Date().getTime();function onLoad() {var now = new Date().getTime();var latency = now - start; alert("page loading time: " + latency);}</script></head><body onload="onLoad()"><!- Main page body goes from here. --></body></html>
• #37: You can do custom page instrumentationby wrapping critical sections of the page in start/stop timersBut it relies on Javascript and Cookies… which might be disabled or not available (especially in Mobile). Only accurate for 2nd page.
• #38: THE BROWSER is doing most of the timing for us… Brilliant!!! No more OnBeforeUnLoad event! It all occurs “after” the page has loaded…No more cookies… lots more metrics <click>
• #39: Many more metrics in the Navigation-Timing spec… at a PAGE level, at least…
• #40: Biggest pro is that the TIMING is mostly done by the browser… so it’s less intrusive and more accurate with a much better set of metrics… CON – browser support…A bit more about “SiteSpeed” <click>
• #41: Free Navigation-Timing Based real-user performance monitoring…Also uses timings from the Google Toolbar… which leads us nicely into the next technique <click>
• #42: Filtered to remove all measurements > 60 and samples > 2. Scale on the left is 3.5 second intervals
• #43: Here is your histogram, turned on it’s side. 0-1 (23%), 1-3 (45%), 3-7 (22.
• #45: Excellent metrics including object level metrics… so you can get that nice waterfall diagram we know and love!
• #48: Basically you are sticking a recording mechanism, a proxy debugger like Charles or Fiddler, between you and the origin web server… and that proxy will record all your requests and the timings associated with them…
• #49: YOU ARE NO LONGER IN THE CLIENT… so no RenderStart, no OnLoad Event, and hence the concept of a page gets “fuzzy”… particularly with AJAXy pages… How does the proxy affect your traffic – probably the biggest potential issue is how the proxy server connects to the origin server. There is no guarantee that it’s going to use the same number of connections or re-use them in the same way that your browser will… From 2005 EricLaw - http://insidehttp.blogspot.com/2005/06/using-fiddler-for-performance.htmlIn Fiddler 0.9 and below, Fiddler never reuses sockets for anything, which may dramatically affect the performance of your site. Fiddler 0.9.9 (the latest beta) offers server-socket reuse, so the connection from Fiddler to the server is reused. Note that the socket between your browser and Fiddler is not reused, but since this is a socket->socket connection on the same machine, there's not a significant performance hit for abandoning this socket.So, Fiddler isn't suitable for timing. But this doesn't impact your ability to check compression, conditional requests, Expires headers, bytes-transferred, etc. Other than the actual timings, the browser does not behave much differently with Fiddler than without (and chances are good that your visitors are using some type of proxy). The browser will often send Proxy-Connection: Keep-Alive; this isn't sent without a proxy.IE will send Pragma: no-cache if the user hits F5 or clicks the refresh button; without a proxy, you have to hit CTRL+F5 to send the No-Cache value.The fact that a client-socket is abandoned can lead to extra authentication roundtrips when using the NTLM connection-based authentication protocol.
• #51: Write a mod or filter… that can see every request… start/stop timers… send them to a collector…
• #52: Web Server mods/ISAPI filters are how most of the APM solutions work. AppDynamics, Dynatrace, New Relic are all in this space, and some of them have implemented the javascript timing as well… Great for measuring the performance of your web tier and backend… not that useful in measuring page level performance unless you go the hybrid approach.
• #53: SPAN port then sniff the traffic…. Re-assemble the packets then the requests then the “page” then record the data…
• #54: The network sniffing approach is really the only true “passive” technology out there i.e. one that doesn’t have any “observer effect” on the measurement. Pion, Coradiant, Tealeaf, TriometricThere is a Not cloud friendly since EC2 doesn’t allow promiscuous mode appliances…
• #55: So in summary…. <Click x 6>And before we go, a quick plug for my user group… <click>
• #56: A great WPO case study next week from “The Times” newspaper… and then in December we have a special Xmas event hosted by Betfair!