![Building configurable applications for the web Questions? [email_address] Tom Melendez, Yahoo!](https://image.slidesharecdn.com/ipc09tommelendez-1-091118014342-phpapp01/85/Building-configurable-applications-for-the-web-38-320.jpg)
Ad
More Related Content
What's hot (19)
Similar to Building configurable applications for the web (20)
Ad
Recently uploaded (20)
Ad
Building configurable applications for the web
- 1. Building configurable applications for the web Tom Melendez, Yahoo!
- 2. Building configurable applications for the web About Me Using PHP since 1998 (3.0.5) Lead Infrastructure Engineer, Yahoo! News Native New Yorker (so, I’m skeptical of everything) Tom Melendez, Yahoo!
- 3. Building configurable applications for the web What does “configurable” mean? What are we really talking about? Am I in the right place? Tom Melendez, Yahoo!
- 4. Building configurable applications for the web Intended audience Developers (who don’t want to annoy their Operations folks) Ops folks (who don’t want to constantly harass their Developers) QA folks (who want to make sure Dev and Ops have their sh*t together) Managers and Execs (who want to tell their bosses, “Yeah, we can do that”) and… Tom Melendez, Yahoo!
- 5. Building configurable applications for the web Anyone who doesn’t want to get paged in the middle of the night. Tom Melendez, Yahoo!
- 6. Building configurable applications for the web We’re talking about application and system configuration What's on the box? What state is the box in? What is it allowed to do? Not personalized user configuration of your web app. Tom Melendez, Yahoo!
- 7. Building configurable applications for the web Why highly configurable systems? The Need for: Multiple Deployment Platforms Multiple Experiences Different Architectures Multiple installations for different purposes Different data and/or presentation of that data Unique states of some instances of the application Real-time tuning Backwards compatibility; Deployment and Rollback Situations Tom Melendez, Yahoo!
- 8. Building configurable applications for the web Why now? A bug in production costs 200 times more to fix than what it would have if caught prior. * Something similar could be said about adding in more flexibility and configuration, because really, it’s a bug. Plus, when your app gets Oprahified or Slashdotted you’ll have plenty of levers to pull and knobs to turn. * Sebastian and Stefan's talk on PHP and CI on Monday at IPC 2009 Tom Melendez, Yahoo!
- 9. Building configurable applications for the web I’m a one-man band: do I really need all this stuff? photo courtesy of origamidon@ on Flickr Tom Melendez, Yahoo!
- 10. Building configurable applications for the web YES. (Besides, the alternative is that we sit here in silence.) Tom Melendez, Yahoo!
- 11. Building configurable applications for the web These are goals, not implementation requirements! The Need for: Multiple Deployment Platforms Multiple Experiences Different Architectures Multiple installations for different purposes Different data and/or presentation of that data Unique states of some instances of the application Real-time tuning Backwards compatibility/Deployment and Rollback Procedures Tom Melendez, Yahoo!
- 12. Building configurable applications for the web Our ultimate goal is allow the system to be as configurable as possible without changing code and without doing too much work. Tom Melendez, Yahoo!
- 13. Building configurable applications for the web I do have some requirements though. photo courtesy of duncan@ on Flickr Tom Melendez, Yahoo!
- 14. Building configurable applications for the web Automated deployment: Host, Application and Configuration All of this needs to be in SCM Going forward without the above is bad Tom Melendez, Yahoo!
- 15. Building configurable applications for the web Other nice to haves would be: Automated Builds Monitoring Profiling Tom Melendez, Yahoo! Despite the logos above, even a poor man’s shell script will do. We want to build our software without effort, know that the app is up, and make sure performance didn’t drop dramatically.
- 16. Building configurable applications for the web So today, things that we’ll touch on: The Cloud (well, not really, but you’ll be cloud-deployable) CI Caching Performance Proxies Defensive coding/Debuggable code Tom Melendez, Yahoo!
- 17. Building configurable applications for the web Now lets make our app configurable. Question: What should be made configurable? Answer: Everything. Tom Melendez, Yahoo!
- 18. Building configurable applications for the web Seriously. Look at your app and prioritize how important it is for the business to have that feature/widget right where it is showing the data it is currently showing. Examples that come to mind: Tom Melendez, Yahoo! Form Fields Payment screens AJAX updates Account Information Summary Data Data about your contacts Previous transactions Recommendations Photos/Multimedia Ads
- 19. Building configurable applications for the web Lets look at some examples: Twitter Facebook Y! News Tom Melendez, Yahoo!
- 20. Building configurable applications for the web I would argue that the previous exercise will probably make you come up with a better solution. It also sets expectations with your stakeholders as to what points you would degrade your service and at which points you would fail. Tom Melendez, Yahoo!
- 21. Building configurable applications for the web Now that we have our priorities for app features, we begin to make them configurable. Take the lowest priority items first If increased load occurs, it is a no-brainer to turn these off Since they are low priority, they are likely to be changed for something else that could be a higher priority You’ll want to apply some high value/low cost logic to the above. Tom Melendez, Yahoo!
- 22. Building configurable applications for the web You will also use the priority list you created as the basis for your DEFCON levels. Example: Level 5 – normal operation Level 4 – planned high traffic event Level 3 – unplanned high traffic Level 2 – component failure Level 1 – site failure Tom Melendez, Yahoo! http://en.wikipedia.org/wiki/Defcon
- 23. Building configurable applications for the web What about a given feature should be configurable? It depends, but generally: Hostnames/Ports Database names Usernames/Passwords Cache times Base file locations Timeouts Application defaults/Flags Display settings Minimums and Maximums Tom Melendez, Yahoo!
- 24. Building configurable applications for the web Is there a strategy to figure this out? Well, ask yourself: Could it ever change, and be changed, without affecting anything else? (u/p, db names) Would someone else ever need to change it? (hostnames, different platforms, diff envs) Do I really want to get out of bed, edit some code, build and deploy just for this change? Tom Melendez, Yahoo!
- 25. Building configurable applications for the web Further strategy (and a little detour). Please write debuggable code Or else Tom Melendez, Yahoo!
- 26. Building configurable applications for the web Writing debuggable code We need extended information available to us for troubleshooting/diagnostics Any service calls need terse logging available to show: the request made to me The URL that I requested What parameters/headers/cookies I sent the HTTP code I got back The response I got back Ideally need to do this while box is in use, but definitely without changing code Tom Melendez, Yahoo!
- 27. Building configurable applications for the web It is 3am . I don’t know why something isn’t working. I don’t want to spend an hour trying to figure it out. I should be able to flip a switch and get this information in the log. Better yet, someone else should be able to do it for me. Tom Melendez, Yahoo!
- 28. Building configurable applications for the web Now that your code is debuggable, strive for Runbooks Step by step decision tree on how to handle a situation and how to escalate if necessary When you’ve reached this stage you now have given someone else control over the maintenance of your application. Congrats! http://en.wikipedia.org/wiki/Runbook Tom Melendez, Yahoo!
- 29. Building configurable applications for the web Writing defensive code Expected objects need to be validated is_object($foo) is not enough instanceof FooClass may not be enough You should ensure that what you have is a valid object that you can use Tom Melendez, Yahoo! if (is_object($xml_resp) ) { //site blows up $xml_resp->xpath(“/something/that/does_not/exist”); } if (is_object($xml_resp) && $xml_resp instanceof SimpleXMLElement) { //better, but you still don’t know if it is valid as you see it. Could be an error message in XML format $xml_resp->xpath(“/something/that/does_not/exist”); }
- 30. Building configurable applications for the web Degrading Service A mechanism to request data without killing your connections Your code depends on something which might not be available Or, you’re just a smart cookie and know that at any time, anything might not be available Can you still serve traffic? Are you eating up connections by connecting to something that probably won’t be there? Should you be serving traffic? Review your priority list as to when you should fail Tom Melendez, Yahoo!
- 31. Building configurable applications for the web Degrading Service Assuming that you shouldn’t fail, the service should be degraded At some point, you should stop requesting that service, you’re just eating up connections. How it works How many times should I try to fetch and have it fail? How long should I wait until I try again to fetch? Tom Melendez, Yahoo!
- 32. Building configurable applications for the web Managing your cache File cache, local cache, distributed cache, proxy cache, CDN Do you have/use tools to manage your caches (CRUD operations) ? Can’t tell you how many issues I’ve seen to bad caching, whether it be expiration or poision All Devs and Ops need to know how to use these tools (Runbooks, remember?) Tom Melendez, Yahoo!
- 33. Building configurable applications for the web Storing and reading your configuration Single PHP file, auto prepended? Apache SetEnv (requires restart)? Database? APC? Beware of apc cache slam Tom Melendez, Yahoo!
- 34. Building configurable applications for the web Configuration via Proxy: Introducing Apache Traffic Server “ Traffic Server is a high-performance web proxy cache that improves network efficiency and performance by caching frequently-accessed information at the edge of the network.” http://incubator.apache.org/trafficserver/docs/admin/ How do we know which INTL to show? We use an Apache TS remap rule to proxy the user to the front end passing in the INTL in the URL Super-high performing: Unofficial “test” had it handling 90k rps (!) on 12 boxes in 3 colos Enabling caching on TS is our DEFCON 1. We’ve never needed that. Allows for other cool things like bucket testing and URL rewriting. Tom Melendez, Yahoo!
- 35. Building configurable applications for the web Configuration via Proxy: configured with YAML Tom Melendez, Yahoo! vars: story_fe_ep: <internal dns name> templates: news.template paths: ... /s/: endpoint: %{story_fe_ep}/news/story/v3/en-US/s ... hosts: news.yahoo.com template: news.template
- 36. Building configurable applications for the web Wishlist: If I could have it my way Tom Melendez, Yahoo! Dev writes tests and code, commits Commit build runs on build server, passes OK Packages are deployed to Integration periodically and BAT and Smoke Tests are run Packages are deployed to a QA env, Selenium and regression tests are run In parallel, packages are deployed to another environment and perf tests are run All looks good, we automatically deploy to a bucket See good performance, high clicks, user satisfaction in the metrics, we initiate the process to go live.
- 37. Building configurable applications for the web Summary Give others the options, tools and info necessary to administer the app An ounce of prevention is worth a pound of cure Don’t get paged! Tom Melendez, Yahoo! photo courtesy of red5standingby@ on Flickr
- 38. Building configurable applications for the web Questions? [email_address] Tom Melendez, Yahoo!
Editor's Notes
- #3: As infrastructure, I’m responsible for building backend components as well as telling everyone else why their components don’t work. We’ll talk more about that during the presentation
- #4: It means a lot to me. I’m using the term configurable “loosely”
- #6: No big ideologies or frameworks here. If you want to go on vacation and/or work on cool stuff, give others the power to control your app so they don’t need to come to you.
- #8: Unique states used for debugging or bucket testing for example. We’ll definitely be sloped toward site availability in this talk but these are all important
- #10: Or maybe you’re a small company or startup or a manager who has engineers that are overworked and deadlines that are too tight.
- #13: Build info into your app and your process to let others manage it effectively. This is not a selfish endeavor; we all want to spend more time doing the fun things in programming like coding and design. Not triage.
- #15: You won’t get the bang for the buck if you don’t have this. You want have the power to quickly and reliably deploy fixes, configuration changes and even roll back. In the same vain as what Sebastian said yesterday, “You want to make deployments a non-event”. This is hard, as in order for them to truly be routine, you need extensive automated testing and such in place. But, the first thing you need is the mechanism to actually build the host.
- #16: We want to have a baseline as to how much you can handle. something like siege, ab, http_load
- #20: Twitter – don’t forget the API
- #21: Getting this on the table early is important. You’re doing this with wireframes, right? Once everyone knows at what points these things happen, discussions can begin as to what to do at those points and how to mitigate them
- #22: High value low cost: if something is a priority 3 but really expensive resource-wise to produce you might want to take that early on, especially if you are expecting increased traffic
- #23: Go over Defcon levels. Defcon levels also have business procedures as well (Notify XYZ of situation, etc.). Your priority list serves as the basis as to what levels and knobs to engage. Example Level 4 for News: Emmy awards, MJ tribute. Level 3 is where we plan on degradation of service based on expections, Level 2 is where we implement degradation of service
- #24: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #25: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #26: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #28: Lets also talk about defensive code a bit
- #29: Lets also talk about defensive code a bit
- #30: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #31: Flesh this out, talk about a db or web service that isn’t there. Yes, you can fetch it from the cache but you shouldn’t try to connect to it if it likely won’t be there
- #32: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #33: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #34: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #36: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #37: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #38: Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
- #39: As infrastructure, I’m responsible for building backend components as well as telling everyone else why their components don’t work. We’ll talk more about that during the presentation