Conquering Chaos: Helix & DevOps

Editor's Notes

• #2: Marc Towersap, Sr Consultant at ReleaseTEAM. He has 15 years experience in Software Configuration Management, version control, builds, and deploys across both Windows and Unix, across a broad range of companies, defense, telecommunications, dot-com’s, medical, and financial industries. Mark Eliason, Consultant at ReleaseTEAM. He has 15 years experience in Software Configuration Management, version control, requirements management, change & issue management, workflow management, installer development, etc. across software development, services and manufacturing industries.
• #3: Our consulting services vary from a 1 week environment or tool analysis to ongoing staff augmentations. We are typically brought in to address an ongoing concern, like performance problems, tool migrations, upgrades, customization or special integrations. We often are invited back to tackle additional work. ReleaseTEAM has vast experience in multiple toolsets, integrations, languages and scripting
• #4: No matter the size of your development group, we are here to help you define a plan and stick with it for significant enhancements to your SDLC and DevOps environments. Here’s a sampling of the many clients we’ve helped, clients of all sizes.
• #6: Those of us who have been working in the configuration management and technology world for awhile know that we’ve all been “Doing DevOps” for a long time. But it’s only within the past decade that DevOps has found an actual name for itself. In a relatively short amount of time, formalized DevOps has grown a lot and doesn’t show any signs of disappearing.
• #7: Configuration Management and DevOps have suffered from similar problems since the terms were coined. When people talk about either, it's not uncommon to think about them in the context of just one or two areas (which can be huge in and of themselves). When you start down the path of conquering chaos through DevOps, you have to think about the entire spectrum.... It’s not just about any one thing, really.
• #8: All that said, to really get DevOps right…..
• #9: Bam!
• #10: Bam!
• #11: Bam!
• #12: Bam!
• #13: What’s the point of all this? Why do you need to care about these things? It all adds up! And failure might be more common than you think…
• #14: The takeaway here? Read it and weep. Literally.
• #15: That’s right. Billions…. Whether you define it as failure or chaos, one of the other takeaways here is that aside from the fact that chaos has a real and tangible cost -- IT, software development and product development is hard. It's not easy from a technology perspective, it's not easy from an engineering perspective and it's not easy from a people perspective.
• #16: As a former hero, I say, being heroic isn’t all that cracked up. I recall giving up a birthday party given by my family, as I was stuck at the office. I recall long calls on vacation, burning up cell phone batteries, missing out on things because I was tied to a pay phone paying international rates diagnosing deploy failures even though I left detailed instructions (people wanted their hands held)
• #17: We love Perforce and we love Helix, but Helix is not going to magically solve everything for you!   It's up to you to define what it is you're trying to accomplish. And you can't just jump in and start engineering a solution. Doing that will likely lead to chaos... in the worst cases, to failure.   DevOps and Helix aren't magic bullets. But given how much is at stake, it's clear that you need to spend some time thinking about things before you jump in. Time is often a luxury a lot of organizations don't think they can afford. But the time you put in up front will pay off in the end. Conquering chaos isn’t the same as going on a witch hunt. Seeking blame is a recipe for failure… it makes people defensive and and encourages them to hide or obfuscate information that could be key rather than help. Don’t derail your own efforts!
• #18: I had a client that had a complex deployment consisting of many many pieces, one of which was a ugly manual deployment step requiring over 100 mouse clicks, manually filling in html forms. Rather than wasting time trying to just automate that (it does need to be automated), but start that while tackling other pieces that are easier, but not just deploy html page.
• #19: Shielding teammates from firefighting is key, because it seems there are always firefights steal away valuable focus and time. Implementing a good DevOps solution should help reduce the need for firefighting. Don’t wait to the end to integrate!! I’ve seen branches permanently deviate where 100’s of developers worked on separate branches for months then tried to merge one to the other. Surprise – the merge failed, causing weeks of extra effort and a missed project date. In other cases, developers stayed isolated in their own development branches until deployment night, which became a comedy of errors. Bad performance, bugs, hacks (where fixes never made it back to the version control tool and had to be fixed again in the next deploy), all-day and all-night night efforts and a sizeable probability of aborting the deploy… again, don’t wait to the end to integrate!
• #20: So don’t hardcode in specific project names into any scripts, make it flexible enough so it can accommodate adoption into other projects without ruining the existing project. Try to genericize as much as possible, so it’s not brittle that it breaks when adding other projects.
• #21: Sometimes you have to branch because of differing release schedules. Maybe one codebase is released on a much more frequent basis compared to others. If you’re dealing with frequent (weekly if not daily) releases, you have a candidate for mainline/trunk-based branching. This is particularly true the thing you’re releasing isn’t dependent on the slower moving parts. Keep branching as simple as possible, but no simpler than that! (Yeah, real clear! But also true!)
• #22: We’ve assisted our clients with branch and stream patterns to accommodate all the things we mentioned above. If you’re working with someone who’s a little indecisive or erratic (e.g., “let’s put 5 features into this release… they also need to be merged into the next release… oh wait, I changed my mind, I only want 3 of those features now… pull out 2 of them out of this release, one of those also out of the next release…and bring forward a feature from the next release into this release…”), Helix’s flexibility will help you adapt and conquer!
• #23: The focus of product development obviously isn't about the tools or processes you use or how well you can write an engineering spec, a design doc, a tech manual or a defect description. But... you can't have good engineering if you don't pay attention to all of these things. When we're talking about "good engineering," we're not just talking about a developer's ability to write the most elegant piece of code on the planet. Processes can be bad. Process can be too heavy. And if an organization's processes are too rigid, they can do some serious damage. Team satisfaction, creativity, product innovation and quality - they can all take a hit. Get your process right. Keep it as light as you can get away with, but remember - when there's too little attention paid to process, people, technology and how they all need to fit together, you end up with chaos. Not all silos are inherently bad, mind you. But you have to be careful where the lines are drawn. Silos can lead to an "us against them" mentality in the same organization, and can also be a nightmare to manage from a technology & infrastructure standpoint. A lot of companies try to silo, particularly with deployment – Dev is Dev and Ops is Ops. Combining the two into DevOps actually scares some people! People get worried about job security, getting automated out of a job, etc. Moving from a silo mentality into the DevOps world can be scary. But it’s also an opportunity! It’s a chance to learn new skills, to test deployments safely and to become an even more valuable asset to your organization. If you can automate deployment, it too is job security. Traditionally, deployment is thought of something that happens when? At the end of the day. At night. On the weekends. The dream is to not have to piece things together after everyone else has gone home while you’re trying to hit some critical milestone. The dream is to find and fix issues early and to deploy new code quickly and easily as its built. Break down rigid Dev and Ops silos. It’s not Dev & Ops, it’s DevOps!
• #24: So now we get to the exciting part – what can Helix do for you? Whatever business or technology drivers you come up with, Helix can be one of the pillars of your DevOps infrastructure. What makes Helix such a valuable DevOps tool? Think back to all the things we mentioned earlier, the things you need to think about and do to really get DevOps right. Helix has you covered in every one of these categories.
• #25: Helix Swarm. It helps you collaborate, it helps you communicate, it can help you integrate with tools like JIRA and Jenkins (which you’ll see in the following slides), and possibly best of all – all of this power is included in your existing Helix license!
• #26: Just like DevOps in general, there are things you have to do in order for systems to talk to one another. Helix Swarm and P4 are no exception to the rule. But once you have things set up (which in this case includes installing & configuring Swarm and adding a few triggers to your P4 depots), it becomes painfully easy to initiate code reviews from a submitted or shelved changelist. The functionality is there for you in the P4V context menus or from the command line. And again - it’s important to review code sooner than later. If you code review after the production deployment, you could find that you missed an opportunity to solve a major performance issue now hammering production with slowness, excessive database calls, piled up threads, etc. Finding and fixing poorly written code sooner than later is just as important as integrating sooner!
• #27: Helix Swarm also gives you an easy to set up and lightweight integration between P4 submits and JIRA issues. Enable the integration, add JIRA ticket numbers to your submit messages, and Helix Swarm takes care of the rest. The nice thing with this integration is that it gives you the essential links you need to move between systems with ease. If you want to get fancier with your integrations, you can add the Perforce Defect Tracking Gateway into the mix. That’ll give you the ability to customize a lot more of the detail you get in JIRA, and it’ll also let you tie P4 Jobs into the equation. Even better yet – if you want to, you can use both integrations!
• #28: Helix Swarm also makes it pretty easy to pull in pass/fail information from your continuous integration system. There’s a bit of a chicken and egg scenario in this example… you do need to have a Jenkins server running already, of course. And whether or not you’re using Swarm, you need to set up a build project that maps to Helix in a predictable way. But once you have those pieces in place. It doesn’t take a lot more to update that Jenkins project so it can talk to Swarm. When the connections are there, you can map a Helix Swarm project to a project in Jenkins. Any submits to that review project will fire off a build automatically (whether or not you have Jenkins configured to poll the SCM system), and the results of that build will be indicated right there for you in the review header. Voila!
• #29: One of the hallmarks of Helix Swarm is its ability to enhance communication and collaboration within a team. You don’t want a system like this to replace in-person conversations. But at the same time, communication systems that completely rely on in-person interaction (or email) aren’t going to do the job for you either. . Think of Helix Swarm as an addition to your communication process. It can help you jump start conversations and get some of the easy stuff out of the way, letting you focus on the things that are going to take more time. Swarm is also a huge asset when it comes to linking conversations back to the actual changes being reviewed. Good luck capturing that type of detail in an email conversation  In the example you see here, we’re obviously not dealing with a code review. But also don’t think of Swarm as being limited to code reviews. It’s really something you can use to review just about anything you can version in Helix.
• #30: Ideally, automate everything associated with builds and deploys. If you can automate it, you can make it fast and cheap and easy to build/test/deploy/re-test and catch bugs from not only source code, but deployments and infrastructure changes too. Sooner than later.
• #31: Again, it’s up to you to make use of these tools… but you have options! Don’t create tools because you can, though. Create them because you need them.
• #32: Bam! Napoleon Bonaparte. Conqueror and creator of chaos….
• #33: Even without any fancy tools, there are things you can and should take a look at from time to time. Arthur C. Nielsen (the grandfather of market research) once said “The price of light is less than the cost of darkness.” There’s information in Helix right there at your fingertips, and some metrics really don’t have to be that complicated! Even the most basic measurements can open the door to information that is far better to be known than unknown... how many times have you been caught the chaos caused by problems that are easily avoidable, like running out of licenses at an inconvenient time or running out of disk space at an even more inconvenient time? How many of you are monitoring whether or not your checkpoints and backups are running like a well-oiled machine? Don’t think of metrics as evil (even though improper use of metrics is evil…but that’s an entirely separate topic). Start somewhere, pay attention, and grow from there.
• #34: Graduating to the next level in Helix metrics – we give you Helix Insights! Again, this is a tool that’s available to you as part of your Helix license. It is a separate installation, and there are some things you need to configure properly. But once you have the system up, you upload a checkpoint, generate some initial statistics, and you’re off! This tool won’t tell you absolutely everything out of the box. But with relative ease, you can visualize some of the information that’s just sitting there in your Helix metadata. Information regarding project stability and activity – Insights makes it visible and easy to understand. Insights even gives you the ability to see some server-level statistics powered by a kind of “watchdog” service running in parallel to Insights. Don’t feel as if you absolutely need to evaluate every project under the sun. Don’t feel as if you need to view every slice of data either. But don’t look at nothing. It’s also important to mention p42db here. If you want to extract your Helix metadata out into SQL and run your own queries and build your own reports – have a look at p42db. Also check out Tableau Software’s presentation – Tableau makes incredible analytics software, and they use their own technology to visualize and monitor the KPIs that matter to them.
• #35: The interesting thing about metrics – numbers can help you understand what’s happened in the past. They can also give you a glimpse into the possible future. The study and statistics mentioned here include data regarding a lot of different types of security issues and data breaches. But the thing to really pay attention to is the last bullet – out of all the issues documented in Verizon’s 2015 Data Breach Investigations report, a statistically significant (or at least interesting) number of incidents stemmed from insider misuse of information. Security violations don’t always happen to someone else, and threats don’t always come from the outside….
• #36: As if all the things Helix does for you wasn’t enough – there are even more reasons to think of Helix as as core part of your DevOps infrastructure!