Agile analysis development
- 1. Agile Analysis Pipeline Andy Brown New Pipeline Development
- 2. Who Are We?
- 3. Who Are We? One of the world's largest DNA Sequencing Centres Second largest compute centre after CERN in Europe
- 4. What Do We Do? Human, Mouse, Zebrafish and Pathogen Genome Projects Post sequencing analysis, annotation and maintenance (It's never truly finished!)
- 5. Who Am I? Tracking systems and analysis pipeline for Next Generation Sequencing Technologies Perl, Web Technologies, Moose
- 6. Next Generation Sequencing? Massively Parallel DNA Sequencing Producing Millions of Reads per run ~38 instruments ~5Tb of data a day Managing quick turnaround on Staging of 320Tb data a month
- 7. Analysis Convert Images to Bases Obtain quality values Recalibrate quality Separate up DNA sequences from different projects Do this in parallel Be able to extend this
- 8. Analysis Current analysis running script was unable to cope with changing demands
- 9. What Did I Have?
- 11. A Brief
- 12. Run Completes Bustard Adaptor Removal Split by Tag CIF Qseq, Sig2 Split by Tag Calibrate Scores Index: rejectsIndex: rejects Index: + tags Split by Tag Split by Tag Split by Tag Create Cal Table Cal Table Control Refs Calibrate Scores Consent Align Index: + tags Cal-Qseq Consent Align K-mer Error Correction Cal-Qseq Index: + consent K-mer Error Correction K-mer Error Correction K-mer Error Correction K-mer Error Correction K-mer Error Correction Index: + consent Index: + rejects K-mer Error Correction K-mer Error Correction K-mer Error Correction Create Fastq K-mer Error Correction K-mer Error Correction K-mer Error Correction Align to Ref Index: + rejects Fastq K-mer Error Correction K-mer Error Correction Create SRF Control Refs Sample Refs Next Page! BAM Initial Product Creation Initial Product Creation Gray boxes may be pass-through
- 13. Control Refs SRF Sig2 Index fastq BAM Run Summary (Summary.htm stuff) IVC Plots Q20 Counts Fastqcheck Insert Size Histogram Error rates and QQ-Plots Heatmaps SNP Finder ... And Anything Else You Can Think Of Human QC Fuse Archive QC and Archival
- 16. Working in a Agile Manner Current manner – still close to Cascade, some idea of iterations I wanted more agility – defined iterations Got close
- 17. First Iteration - It1 Chop down the brief into stories Spoke with creator of the brief, my boss & team about what was needed Pluggable, Automatic, Auto QC
- 18. It1: First bit of Coding Read old code – anything I can steal – yes! Write some 'in principle' tests to get an idea of the way to go. Write some code for those tests.
- 19. It1: Prototype Launch next LaunchSelforFinish LSF DEPENDENCIES
- 20. It1: Fail Test Principle – Worked Reality – Too Unwieldy
- 21. It1: Evaluation Too much wrapping Too much could go wrong with lots of parts Out the Window!
- 22. Second Iteration - It2 So, I'm Agile. I don't see this as a set back. Opportunity to try a different approach. I sketch it out.
- 23. Flag Waver Function b Function c Function d Function eFunction a Object to Launch Ca Object to Launch Cb Object to Launch Cc Object to Launch Cd Object to Launch Ce Component a Component b Component c Component d Component e
- 24. It2: Second lot of Coding Again, start off with in principle tests Write some code to pass those tests Select a bit of real world to apply it to
- 25. It2: Pass This real world bit works All jobs are launched as expected Replace the old section with this bit It still works :) A perfect replacement
- 26. It2: Evaluation Success :) The Flag Waver model - functions that know what to do, but no knowledge of other functions This should make it pluggable
- 27. It2: Evaluation Bulky data getting generated multiple times over – Needs more DRYness
- 28. It3: Some new requests It would be easier to code if we didn't have users of the applications! The first new request comes in for some automated QC Just launch them at the correct time
- 29. It3: Scrum So, I scrum. The objective: Work out priorities for this iteration. There are many 'stories', I decide on the following.
- 30. It3: Scrum Write something to make data construction and passing more DRY Write another replacement pipeline section Try to incorporate 1 QC into previous pipeline section
- 31. It3: Tests I write some tests to assess launching the analysis pipeline I write some tests to incorporate a QC launch into the post analysis pipeline I run the tests, which fail
- 32. It3: Code I decide first to add the QC launch My boss wants to start getting the data I get a quick view of how pluggable the system actually is It is good :)
- 33. It3: Code The analysis guys want their pipeline to start showing up Good reason - a new version of the scripts have appeared, and they don't want to patch the old This takes the rest of the iteration
- 34. It3: Release The most important release so far Completely replace old code with new Took about 2 days, with bug fixing
- 35. It3: Evaluation Bugs on Release - tests don't always prove everything! No time to DRY out the code Successful product into production Old code has gone to 'silicon heaven'
- 36. It4: Scrum I again scrum So far, iterations have been quite quick In order for some time to pass for the pipeline, I decide to do refactoring this time
- 37. It4: Scrum Utilising more Inheritance (using Moose Roles) Create external role to translate attributes without building hashes each time
- 38. It4: In Brief After 2 weeks » a nicely refactored pipeline » external role to DRY out data (released to CPAN) » time to have monitored how the pipeline was running Release and go
- 39. The next few iterations Iterations continue, releasing every 2-3 weeks :) Until it all broke :(
- 40. The Broken Pipeline Iteration Up until now, the pipeline had been behaving itself. New analysis code came from our supplier, our R&D team would test, then I would throw the switch and release.
- 41. The Broken Pipeline Iteration However, they changed something we didn't find in testing. Runs with multiplexed lanes broke, as they have an extra 'barcode' read
- 42. The Broken Pipeline Iteration Luckily, here is where being agile really helped. Whilst I had just 'scrummed' to decide my priorities, I just dropped them New Priority – Fix the Pipeline
- 43. The Broken Pipeline Iteration Pluggable, so could a function or two be moved to help? Yes! 1 function move would halve the problem. Run on example – expected outcome
- 44. The Broken Pipeline Iteration Now to fix the 3 read / 2 read problem Again, write tests, test, code, test, run on example, write tests for bugs, test, code, test, run on example .... End of this iteration, able to release a fully fixed pipeline
- 45. The Broken Pipeline Iteration Evaluation: Being Agile, both in project management and design, helped here. How?
- 46. The Broken Pipeline Iteration Design: Plugin design of the pipeline - half the problem was solved just by moving something. The other part just by writing a new module. It just worked!
- 47. The Broken Pipeline Iteration Project Management: Changing an iterations priorities so that the urgently required fix could be done... ...barely disrupting the flow of work on feature requests
- 48. What has happened since? Development has settled into a 2-3 week release cycle Team knows development position Made it easier for them to cover me
- 49. What else happened since?
- 54. Acknowledgements David Jackson Guoying Qi John O'Brien Marina Gourtovaia Sri Deevi Tom Skelly Irina Abnizova Steve Leonard Tony Cox You
Editor's Notes
- #2: Intro myself Here to talk about the agile development process of the analysis pipeline I developed
- #3: We are the Welcome Trust Sanger Institute. Here is a picture of our campius which is south of Cambridge in Hinxton
- #4: We are one of the worlds largest DNA sequencing centres. Until fairly recently, the largest, but we have been overtaken in the last few years by some centres in America, and then the Chnese have blown all the competition away. We also have the largest compute centre in Europe after CERN. Biology has very much moved into the informatics domain, and unlike many other disciplines such as Physics, which have had the time to develop their compute infrastructure over the 15+yrs it takes to design the rest of the experiments, Biology has jumped such that we are sometimes lucky to get 1.5 motnhs.
- #5: Originally set up to sequence 1 third of the Human Genome, we also worked on the Mouse Genome. We have also sequenced other organisms and Pathgens ourselves. We are also involved in the Post sequencing analysis, annotation and sequence maintenance. Contrary to popular belief, the sequences are never truly finished.
- #6: I'm a software developer in a group responsible for producing the tracking systems and running the primary analysis pipeline for the Next Generation Sequencing Instruments. I mostly develop using Perl, Web Technologies and Moose
- #7: What is Next Generation Sequencing. The Human Genome cost millions of dollars, and took around 15 years to complete. Very costly, as it used a sequencing techinique which just did a few strands of DNA at a time. NGS is Massively Parallel Sequencing of strads, sequenicng millions at a time. However, with approx 38 instruments, producing around 5Tb of data a day, that is a lot forus to deal with. We have a quick turnaround of 320Tb of data a month.
- #8: Here is an outline of the primary amalysis requirements, which needs to be done within 2 weeks of the run completing on an instrument.
- #9: Our current analysis pipeline running script was unable to cope with the changing demands.
- #11: Time – I had a little bit of time to look at how to approach this, and the best way to structure it
- #12: A suggestion of what the whole pipeline would be needed to do
- #15: Vision, Ideas and Enthusiasm
- #16: Desire to develop in a more agile way, even if the rest of my teams focus wasn't quite along with that
- #17: We had always had visions of working in an agile manner, but in reality it was still quite close to cascade. We had tried to apply some idea of iterations, but it mostly meant I've done a feature, if no-one has objections, I'll release it. I wanted more than that, I wanted Agility ad the idea of defined iterations. I got close with this.
- #18: For my first iteration, I decided to look at the brief, chop it down into manageable chunks or stories, and decide look at what we wanted, then prototype it I spoke with the creater of the brief (who had been running the previous script up until now), my boss and got some ideas from my team. We wanted something that was pluggable, automatic, and would be able to produce some QC.
- #19: So, in my first bout of coding, I started by reading the old, and now convoluted, script. Mostly looking for anything I could steal for a prototype. I wrote some tests of how I wanted it to work. I then worte my prototype
- #20: I had heard of someone elses pipeline which involved a daemon running constantly, polling for finished jobs beofre launching the next. We had already decicded that this was not that feasible for us, since it would involve too much overhead of a daemon per run, approx 5 new daemons per day. I hit upon an idea of a script which would Know the order to launch jobs Know which job had just been launched – via a command line parameter Launch the next job Launch itself, with the state, and a dependency on the previous job, to feed through to launch the next, or just finish
- #21: In principle, it worked. However, in reality it was an epic fail, It got too unwieldywith more that parts, and that was before we wanted to parallise jobs
- #22: Also it had too much wrapping code, dealing with processing what had gone before it. We decided that just too much could go wrong with it. So we threw it out of the window.
- #23: Part of being agile is not to worry. I had only spent a week on this, so don't see it as a set back. It was a prototype, and a first attempt. I learmt from it. This surprised my boss, since I'm normally quite particlular about my work, and I don't like things going wrong. Well, unless I deliberately plan it that way. I spent 3 weeks trying to prove that for us, message queues miight be unreliable, and finally did. I took this as a chance to try a different approach. I sketched it out.
- #24: This approach got termed the flag waver. A central function would launch in turn smaller functions which would call out to objects. Those objects would know how to launch other programs from the pipeline, and would only feedback the minimal amount of information required to launch the next one. Using lsf, this would only need to be the job ids of the launched programs, which would then be used as job requirements for the next launched job. This fitted nicely with the idea of it being pluggable, as the different functions should be loosely coupled. The order might be important a before b, but it might not matter if c got put in between them.
- #25: With this idea, I got back on coding. I write some in principle test, and some code to pass those tests. I then try it on some real world to see it it still is ok, as before.
- #26: This time the real world works! All my jobs launch and go through an example as expected. I replace the old section I had taken (some post analysis analysis) and actually install it. WooHoo, it is a perfect replacement.
- #27: Evaluating again – success! I now have a model which is loosely coupled enough to make it pluggable – I hope. I'm also happy enough with this as a prototype to move it into a production scale framework
- #28: However, I notice something about the way the functions are getting called. Some data is in the form of bulky hashrefs, and geneated multiple times over
- #51: My boss and other team members have read some of my books
- #52: We scrum daily
- #53: Sprints and feature requests are tracked using RT
- #54: Our productivity has increased