QA as responsibility of Whole Team

Editor's Notes

• #3: Before I will speak about the way how I see QA Process on Agile projects I would like to spend few minutes defining what the Quality itself is. Now thinking in the scope of Software development but it is nice to go beyond such borders. There are a lot of definitions what Quality is and I really recommend everybody to think about it carefully. Choose one, ore better build one which suits you. It is quite important because this will be your target, something you aim for. I like to define Quality using concept of four levels. First level is “Deliver according to Requirements” – that is really not enough but sad to say this is very common understanding of testing = verify that we delivered according the specification. I hear this very often from Product Mangers and others roles on that level of hierarchy. If we aim for that we are monkeys, sorry but this is true. Next level is to “Deliver according to Correct requirements” – now we in the time when monkeys start to think. What I am doing? Does it make sense? And very soon we are on next level when we Deliver value. This is natural step forward because there is always a point (sooner or later) when people asks the question “Why am I doing it?”. Now we analyze requirements seeking for the purpose. Everybody is probably familiar with the concept of User Story which has always three main parts:1. As somebody2. I want something3. To achieve some goalI want something is the actually the less important part. The most important is the goal, when we need to know the person to be able to deliver desired solution. I said it should have four levels so here is the last one “Deliver value according to correct requirements when all stakeholders are satisfied and happy.” In other worlds for me Quality is Satisfaction of all Stakeholders. You can easily apply this definition to all aspects of our life.
• #4: People always asks me if I really do everything I present when speaking about ideal QA process. Of course not. This is my vision, target, something I aim for. But it is very important to know how you would like to do things, to know the direction to be able to find out the way how to be on the way.
• #5: So how do I see it? Lets simulate one hypothetical ideal project now to demonstrate what is the job of QA and how everybody needs to participate and work together to achieve satisfaction we aim for. It usually starts even before the project when project is sold. This is the phase when things can be really messed up and when they are you have basically no chance to change it later.
• #6: All projects are based on some contracts. Agile contracts is probably topic for another speech so I would like to state only several thing here quickly. Our SOWs in Vendavo defines a lot of things and they are also written on legal English. I do not why, but this language is always very hard to understand. Usually our customers have no experience with Agile or their experience is not good. They basically do not understand the process at the time of contract signed off and we do not push to achieve that. Than they get feeling like Agile is easy. That is big mistake. Never ever say that Agile is easy for customer. It is not true and my experience is that is challenge, specially for these big corporations we work with. Make sure that all stakeholders really understand process and their responsibilities. Find out where you are as soon as possible. Do not think you will solve this later.
• #7: Our SOW is quite fine, not perfect but there is everything what should be there. But from whatever reason nobody cares about the most important chapters too much. What does it mean to work in Sprints? What is definition of done? What do we expect from customer? Who cares? The most important chapter for them is Scope! Yes, we have a scope in SOW but it supposed to be kind of wish list, however customers very often understand it as promise. And boom, we have a big problem. Another interesting fact is that despite the fact that SOQ is legal document it is very hard to push to follow it later during the project. For example there is an statement that customers needs to verified all user stories during the next sprint after the one they were delivered in. This is really challenging and they often struggle to do it. But nobody really cares.
• #8: Now we have SOW signed by all important parties and project could start. We always start with something called Foundation Sprint which is basically period of time before real implementation starts. It is very good idea with clear acceptance criteria like All stakeholders are identified and understood – it is a good practice to identify different stakeholders on customer side, because I guarantee you that there are different groups on the customer side who have different goals, expectations, etc. Also do not forget that you are important stakeholder too and you have your needs and you also would like to be satisfied. Nobody would like to spend 8 hours each day do something he does not like. In this phase we also again review and agree on the process we will follow and who does what and when. Product owner should share basic idea and most critical requirements with the team to help them understand what this project is about Another very important commitment for Product Owner is to prepare enough user stories to start next Sprint – ideally scope for more sprints to mitigate risk him been bottleneck Exit criteria from QA point of view is that Test Strategy is prepared, reviewed and agreed. Looks nice? Yes, but…
• #9: Unfortunately we are not so strict on these exit criteria. Here are the areas we struggle with and typical problems. All stakeholders? So who will test on customer side? What are you talking about? Testing is your piece of cake. We have nobody to test. We have only demo right? Agile? Is it really this? That is not how we understood SOW, we do not like it. High level idea is clear = thanks for at least on green point here because this usually works fine. Enough user stories… Not really… Most of the user stories are often really only about one sentence without any acceptance criteria. You really cannot start implementation based on it. And regarding test strategy? What do you need from us? That is your responsibility.
• #10: Whole team needs to be involved in Foundation Sprint when some activities are shared, some are special. So what is the job of Testers (QAs) in this phase? We are often called to be the one who likes finding problems and criticize. I think this is misunderstanding. I prefer to say we like to provide feedback. On everything of course  Sometimes it is not welcomed. Even I think that sometimes fault is a bit on our side when we really would like to help but we do not choose right way how to communicate it. There are three main topics to be done by QA during Foundation Sprint. We should help with definition and negotiation of Process. If the project is sold in proper way than that is easy. Yes, I saw it once, or twice. In real live this a time when you start digging in it and you ask for personal commitments. My opinion is that customers are more scared by their new role than not willing to do it. But of course without support they tend to protect themselves and do things in the way they are used to. I recommend you to do some workshop with them to explain what you expect from them on real examples. This will help you to set the trust and you can agree on required level of support during each sprint. I will speak about it in more details later. Second tricky part is about the way you capture requirements. In Vendavo we customize enterprise product for different customers and there are quite good standard User Stories for standard product which is always the starting point. However each team and customer are different and you need to consider it. Having team of only seniors means you do not need to specify a lot of details and everybody understands it very well. If you have a team with juniors than more details are required. I found out it works well to assign senior QA to junior Dev team. Not because of expectation having more bugs but because he can teach them a lot of and he can mitigate these risks. On other hand assigning junior QA to senior Dev team is often a problem because he has no respect. You also need to think about customer and his expectations about requirements. Last topic is the Test Strategy. It is living document regarding all test activities during the project, who, why and how will do it. It is good to have a template of Test Strategy to capture the main company approach but be aware to avoid copying it without thinking. Rest of the speech is basically description of the agreement which is the most important part of test strategy.
• #11: Lets assume that we successfully met all Exit Criteria of Foundation Sprint and we can start implementation phase.
• #12: What is typical structure of Agile Team? It always looks somehow like this. There is a customer, some consultant (Product Owner) who prepares requirements and we of course have some Developers and QAs (testers). And it is always in circle because Agile loves circles. Iteration (Sprint) is the short period of time to complete all activities and deliver some real value to customer. However there is one big risk and frequent problem. Even if you draw it as circle, it can still be easily run in sequence and than we have small waterfalls. What is the small waterfall? It is when you practically divide you lets say 3 weeks sprint in W1 = analysis, W2 = implementation, W3 = testing.That is very bad idea. I always like to say “There is no test phase in Agile”
• #13: First two activities which are very closely connected with quality are Grooming and Sizing. These are here to ensure proper business understanding of whole team. Grooming should be regular activity when at least weekly Product Owner goes trough the new or changed user stories the share business understanding with the team. On one hand this helps team to understand what they will do soon but more importantly why will they do it. On other hand team provides feedback to Product Owner which avoids awkward situation when new sprint should start but there are no clear user stories. Product Owner also needs to know opinion of the team from implementation and testing point of view. This is the first step when we aim to prevent defects instead of finding them later. Goal of the grooming is to achieve team agreement about requirements. It looks easy but it is not. These are the main difficulties we have and I believe we are not alone. Thirst is that Grooming is not happening regularly. There can be different reasons for that like occupied product owner, tight estimates, etc. Another reason why grooming is not happening can be simply the fact there is nothing to be groomed. This problem is frequent when staffing is not done properly. We usually have teams of 2 Devs, 1 QA and 1 Product Owner. That works fine. But sometimes it is staffed in the way like 5 Devs, 3 QAs and only one Product Owner. And that is a problem. Because preparation of new requirements is not the only one job PO needs to do. He also needs to support the team by answering questions, clarifications and he should do some validation too. I said it is always about people and that is true. I saw many times that somebody simply does not care. I do not know why but this is often problem of developers who do not want to be involved in this new way of working. They were used to code based on detailed requirements without any thinking. We are back to monkeys. If you have only one in you team you can probably explain to him why it is needed, what is the goal, etc. Having more of them will never work. Last most frequent problem which is sometimes closely connected with previous is voiceless team. If you have a groomings when only Product Owner speaks there can be only two explanations. First is that he is genius and everything is clear. Second, more often, is that he has no attention from the team and than these sessions has no value. Something in the middle is the situation when only one person is challenging requirements – it is usually QA who asks a lot of questions while developers have none. They simply think that it is too soon to think about it and they are waiting to be closer the river to build the bridge. It can be quite late than. ______________________ Second whole team activity is sizing. When you are satisfied with outputs from grooming it is a time to size the stories. Size is a relative measure of user story and it is basically team agreement about complexity. There can be different opinions provided by different people. You need to discuss the correctly and agree on the size. It often happens that devs and QAs have very different opinion about the size and it is a source of very good discussions which again help you to agree and to prevent defects. Again this sounds very easy and I think it is also presented as trivial think to do. I always hear something like “just play a poker”…. That is very misleading. I found the hardest think is to agree what is the middle size example. Another very frequent issue is when you think that story points equals hours. Especially managers always would like to know effort in hours. But if they are the same than there is no point having both, is it? Of course there is some relation between story points and hours but it is definitely not linear. But that is out of this speech. ________________________ So whole team understand user stories, we agreed on sizes. Now it is time to plan the sprint. Sprint planning is the session when you break down user stories to task when each task has owner who will provide estimate. There should always be at least one dev and one QA task. I recommend you to have more for bigger user stories. During planning you should agree who will test what (I will explain it soon) There are always some buts and planning is not exception to that rule. Estimating is very hard thing to do. It is kind of science and that is why people often hack it. Yes it is estimate but you should achieve some level of accuracy. And this target is the source of two frequent problems. First is underestimating – everybody been there Second is guessing and adding buffer – that is consequence of underestimating Over the time you will (you should) learn how to estimate and what formulas works for you. I always use something like (optimistic + realistic + pessimistic) / 3. Estimating takes a bit more time using it but it pushes you to consider possibilities. Testing is an activity, it is not a role. And this is something what people really struggle to understand. As I said our staffing is usually in ratio 2 Devs on 1 QA. It means that devs needs to do a lot of testing which is natural and good. Testers are not safety net for developers, they should be responsible for own code and should be able to prove basic scenarios work. I will explain you how in a second
• #14: So we planned the Sprint and we can start. Do you remember I was talking about avoiding small waterfalls and preventing issues? So here is next opportunity. I call it QA touch point and it is the session between developer who will develop the user story and QA who will test it. This session is done before development start and it has quite simple agenda. QA will present how he will test it and these two sides will agree who will test what. In other words it is important to define what is the scope of functional testing to be done be Developer before he can say he is finished. Yes I do expect developers to test at least basic flows themselves to speed it up. It does not matter if they do it manually or if they prepare automated script for that. Point is to avoid situation when there are obvious defects which slows down everything. Also we agree about cooperation during the development. Sometimes we agree on regular demos, code reviews, etc.
• #15: One of the most important part of QA job is supporting others. Sometimes I make a joke that QA means Questions and Answers but it is not far from the truth. QAs should have sufficient knowledge to be able to answer most of the questions regarding requirements and business point of view. There is always very close cooperation between Devs and QAs during implementation. This also mitigates one big risk and it is a possibility that Product Owner became the bottleneck. He usually has a lot of work with preparation of next sprint. We also do pair programming with Devs sometimes, when I think that Senior QA need to understand code and see problems there. _____________ We still have a lot of our work to be done when Test Cases are part of it. Topic of Test Cases is probably another candidate for separate speech but I would like to mention few things here. Test Cases needs to have right level of details when I always say that Steps are optional. If you understand the requirement you do no need detailed steps to test it. I can imagine only one good reason to have them and it is when you would like to outsource you testing. Important is what test cases are and what they are not. The main purpose of them is identification of flows and its effective way how to test it. We can say that they break down big piece to smaller parts which are dependent each other. Another purpose of test cases is that they are perfect measure of progress and status if used well. Purpose of test cases are often misunderstood in many different way. I would like to speak only about two cases which makes me crazy sometimes. First is that they are prove that testing was done. It is a lie. In fact they are prove that we spent time creating them instead of testing. Second common motivation is that customers requires them form us because they would like to use them later for education or starting point for their own ones for UAT, etc. That is not a good motivation. They should use User Stories and its acceptance criteria for it. Moreover such starting point is very often also ending one. ______________________ Of course there is no success in Agile without automation. It is not necessarily QA task and it depends on the project. Implementing without at least some automated smoke test is suicide.
• #16: So lets continue in our Sprint. Developer finished his activities and they are ready to hand over it to QA. And this is another opportunity for conversation. Yes we speak a lot of… Now it is time when Developer should show what and how he implemented and also quickly describe how he tested it. QA can provide direct feedback and this really speeds things up. You can avoid never ending ping pong of issues in tracking system and it will also reduce a time which you need later to test it. Because you know what is working and you have some level of confidence. Yes we should trust each other. This also the time when I very often asks Devs to show me the code. Lets imagine you have some order which can be in 20 different states. This takes ages to test in UI but it is a minute in code simply asking the question “Please show me how you implemented it.”
• #17: We are satisfied with Hand over so now it is time to test it. Yes we still test but you can see that testing itself is only a part of our daily piece of cake. We execute our test cases now, do some exploratory testing and other cool staff, we report defects of course, but there are not many of them because we prevented them following previously described practices.
• #18: Next activity which I recommend you to do is to have regular internal demo for whole team. I think that QAs are quite good candidates to be presenting here when the biggest benefit is that team can provide feedback. Others can see where we are, devs can comment on others work, QAs getting bigger picture and most importantly Product Owner can validate it. I do not know how it works in your teams but PO validation is always done late in our case and this will help to push it
• #19: Of course there should be demo to the customer by the end of each sprint, when you should actively seek for the feedback. Take as much as possible from this opportunity. Some times customers can accept User Story during the demo. Often this is not realistic and I will speak about it in a minute.
• #20: You have to do retrospective after each sprint to discuss what was good and what went wrong. You should plan an action for each identified item and follow these actions and plans as soon as possible otherwise your retrospectives will be only crying sessions.
• #21: Now we finished the circle and it was really the circle where problem of small waterfall was mitigated by communication and frequent hand overs.
• #22: But where is the customer? We talked about him only during demo. But is it enough? Definitely not. So?
• #23: Should we involve him to all activities we talked about? That will be not effective.
• #24: Important is when customer knows he needs to be prepared whenever is needed. And it is really challenging because you need strong people on customer side to support you when they are decision makers and knows the business well.
• #25: So all previous activities were mainly internal. It means it is in your hand. But customer involvement is critical key of success and I spoke about it a bit when was talking about SOW. Customers needs to test regularly. And this is something new for them. It is normal that people are scared from new things, things they do not know, they are not skilled in, etc. That is why need to support customer during their testing. We deliver a code after each sprint, they see it in the demo but they can accept only basic user stories and they provide high level feedback on others. Yes this looks fine, no this is not what I expected. But saying “it looks fine” is not enough, we should aim for regular acceptance. Ideal is when customer has own test team who is in close cooperation with your one and they are basically on Sprint behind. They will test what you develop in previous sprint. They are not often text experts but it is not a problem if there is a will to do the testing. Your QAs should agree on the required level of support when you basically teach them what does it mean to test, how to prepare correct test cases, data, report defects, etc. Also they should think about future UAT in advance to prepare it properly. Customer acceptance should be mainly from business point of view, that is why it is better if you can involve some real business users and teach them to test, than to having some testers who has no clue about business. Important is to only guide them when they are responsible for these activities. You should not do it instead of it because then it has no value at all. You need them to think.
• #26: This is my idea how QA process can work on Agile project. I was thinking how to visualize my job, specially to my wife, she is not IT person. So I created this image where you can see distribution of my work. You cannot succeed if your QA is only Testing. On other hand yes I still test, of course I do, it is very important, I like it, it is fun. But there are many other things to be done and all of these are shared activities. QA is not a role, QA is an activity. I hope you enjoyed it and now it is a time for Questions.