FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

Editor's Notes

• #2: we haven’t figured this out yet so do not take this as gospel or assume I know what I’m talking about we are still experimenting like many of you
• #3: liferay is an open source java-based portal platform for wcm, collab, social networking prior to liferay I was at Sun for a number of years, and oracle for a number of weeks, and since 2010 the liferay community manager, fostering innovation on and around the platform
• #4: since this is a metrics meeting, let’s start with some metrics this is a snapshot just to give you an idea of the scale we are working with some numbers big, some small. what’s more intersting is how they evolve and why.
• #5: body count: looks pretty good, right? one measure of community Is its size – we’re at 115k nevermind the little dip, that’s just cause we turned on recaptcha, so what if almost half those registrations were spambots, we’re at 115k! that looks pretty good – what else?
• #6: our forums get the most traffic, so here’s another graph showing forum posts nevermind the dip in there due to turning on akismet spam filtering, we have 300k posts!!
• #7: and the marketing department’s go-to numbers – social media metrics. followers, reach, passion, sentiment, circles these numbers get bandied about like a badge of honor to tout how great a job we in the community are doing! this tells us that we are doing great.
• #8: facebook likes, twitter followers, downloads, page views.. they can and are routinely doctored a former marketing person at liferay once saw one of our competitors had 25k facebook likes and we were at around 7k. So we bought some ads on facebook and guess what? our numbers shot up to 20k! – clearly we had an untapped market in karachi and a remote village in russia. all it did was dilute our future facebook posts.
• #9: so we basically threw money away chasing these vanity metrics at liferay, we’re big believers in value for your money – we have money but spend it like it’s our own personal so considering the previous metrics, which ones are the most important to track? to answer that question requires stepping back and thinking about who you are as an organization and what you want.
• #10: we are an open source project firstly, so we rely on open source contributions from outside the company what’s going on here? show this to an exec, and their mood changes. but is it really bad news? how do you know? and what are you going to do?
• #11: What’s going on? What this tells me at least is that there are forces acting on our community and project that these vanity metrics aren’t revealing them
• #12: step 1: who are you, really? founders wanted to serve our community and so tried to do that but their skills lied in software, not farming or construction so they built a software company, and are using that money to achieve the ultimate goal “for life”. you can put metrics around each level of life goals, and so this is where we come to our community metrics – building up our community to build the company, make money, and serve. many of us are here to measure and quantify our communities to presumably make them better – regardless of how you got here, and so step 2 is..
• #13: we, and I suspect many of you do not have infinite resources to measure every conceivable measurable thing in your community nor could you accurately know what to do in each case, so you really need to focus on a small number of specific goals before you spend the $$$ in liferay we thought about this and came up with 3 main goals
• #14: so for the liferay community, we wanted to give more value to participating in our community, because more value means more participation, more innovations, and more contributions, . So then the question is, what do our community members value? quick tale about video. ZERO submissions. later video contest prize was showing the video at a dev conference – result: 15 submissions in 4 weeks
• #15: it turns out that our community isn’t motivated by money, prizes, and gadgets at all, but rather from more “meta” motivations like autonomy (e.g. becoming a moderator, ability to close bugs, etc), mastery of skill, and purpose. Dan Pink, an author on motivations, did an interesting youtube video on this very same subject, and I think it it describes motivation for most technically-centered communities like open source communities. He basically found that for simple, mechanical, repetitive tasks, people were motivated by monetary or material rewards, but for tasks requiring more cognitive skills, there was an entierly different, non-monetary motivation. People want to master their skill, be given the freedom to accomplish it in their own way, and to understand why they are doing it. In our community, this translated almost directly to the answers given to a motivation survey we did about 18 months ago.
• #16: to understand how better to serve our community, we need to know what they want. Luckily, the “money” answer was low on the list, which aligns with the previous question. So they don’t want money, they want special powers and privileges. this costs nothing, and it’s really great news that we use all the time.
• #17: we want to be able to know with more certainty whether we made the right or wrong decision to spend corporate resources on specific initiatives. For example, when we introduced the marketplace, what effect did that have on our business? what about on the community? We have several initiatives being kicked off this year – what if we made a mistake? it wuold be nice to know. What about events? if we spend 50k on an awesome booth at oscon, do we get more developers contributing ideas or apps?
• #18: and finally, goal 3 - it’s sort of similar to goal 2, but with an important difference in terms of what the metrics that align with the goals can tell us. it doesn’t tell us if we made the right decisions in the past , it tells us instead what we should concentrate on in the future. if there is a strong correlation between, say, average time to fix community-reported issues and quality, and we believe in that correlation, that tells us to concentrate on improving that time. understanding the relationships can help us fine tune what we spend resources on in the future. but you have to be careful here:
• #19: right? according to this graph, if we import more lemons to the USA, we can decrease death on our highways! so we are looking out for false positives like this. number of releases or downloads might correlate with more bug reports, for examople. so if we go file a bunch of bug reports, maybe we can make our downloads go up!
• #20: metrics are pretty useless if you’re not going to do anything with the data, so what are you going to do? and assume you won’t always be right. deciding what to measure: hard – see e.g. MLB and Oak A’s and Moneyball. what to do after: programs to change: careful of bad incentives in those programs – MLB for example, suppose you have a nice metric showing community contributor retention – and you see you’re not retaining contributors – what to do? there are many things you could do, but it’s important to know what to do rather than waiting around for it to happen and then deciding. you may argue that if you have a plan for increasing x, why not just do it now and increase x? limited resources. pick your battles/don’t boil the ocean. ok enough theory, what are we doing @ liferay?
• #21: good to have, but most don’t mean much on their own – vanity metrics – so some may wish to even keep them private, not because of secrecy, but because of the tendency to use them on their own. but sometimes it’s better to use on their own than not at all – for those that ask and that matter most of these are fed into the next level of metrics, so fight the urge to use them on their own – there are often relationships in there that are important to understand.
• #22: good to have, but most don’t mean much on their own – vanity metrics – so some may wish to even keep them private, not because of secrecy, but because of the tendency to use them on their own. but sometimes it’s better to use on their own than not at all – for those that ask and that matter most of these are fed into the next level of metrics, so fight the urge to use them on their own – there are often relationships in there that are important to understand.
• #23: our second level metrics are closely related to our goals, like time from contribution to codebase, or bug report->fixed, or the # of ignored contributions. It is these that we have defined actions for, for example if a lot of members begin showing up in a region or city, we’ll target those for events, community outreach, localiztion efforts or time from forum q to a, if that goes up, we try to recruit new moderators which give a sense of belonging and responsibility so more interesting, but also more expensive to track in some cases, so try to evolve this list over time
• #24: accuracy in terms of raw ability to report reality does it help us reach our metrics goals? do you have a plan? are there relationships between them or other external, hard-to-measure factors?
• #27: our forums are also the most active area of the community in terms of raw participation numbers
• #29: e.g. jun 2013 – the 3rd quartile went up, but that wasn’t much cause for alarm as the median was about the same – so we were able to incorporate roughly half of the contributions within the same amount of time even though there were a few outliers. we have found that this metric means a lot to our community – it’s super frustrating to contribute and have it go nowhere, or worse, get completely re-written without an explanation, and you don’t get credit or anything. In this case the spike was due to a pending release – as both liferay and non-liferay contribution performance went way down. but we see one or the other go up independently, we have a plan of action -
• #30: like I said in the beginning, we are still trying to find our way
• #31: 3rd-order metrics can identify even individuals that transition, and send them email or follow-up and keep them engaged