ProdSec: A Technical Approach
1 likeโข662 views
The document discusses the fundamentals of product security engineering, emphasizing risk management and the integration of security practices in software development. It outlines various roles within security teams, strategies for reducing attack surfaces, and the importance of automation and tooling in achieving security goals. Additionally, it highlights the significance of fostering a security-aware culture within organizations and continuous training for developers to minimize vulnerabilities.
![Bug Hunting
โข Evaluating assumptions in the productโs design and implementation
โข No one should ever send a large buffer, no need to check length before copy
โข Local users are trusted anyways, we donโt need to protect data on filesystem
โข Only registered accounts can upload files and we shouldnโt restrict them
โข Users know they cannot insert scripts here, nothing to worry about
โข We trust clients not to upload bad data, we donโt need to check it beforehand
โข We want users to be able to run whatever they want, weโll block the bad stuff
โข The client should validate data before parsing it, thatโs not the serverโs job
โข The service is just open to internal users, so authentication is unnecessary
โข [โฆ.]](https://image.slidesharecdn.com/prodsec-atechnicalapproach-180416023101/85/ProdSec-A-Technical-Approach-15-320.jpg)
![Having perspective
When someone tells you your code has bugs, you can either respond
1) โWhy are they picking on us!?โ
or
2) โI wish we were more prepared, but this is good data to use and improve[โฆ]โ
Which one is more productive? J](https://image.slidesharecdn.com/prodsec-atechnicalapproach-180416023101/85/ProdSec-A-Technical-Approach-18-320.jpg)
ProdSec: A Technical Approach
- 1. ProdSec A technical approach Jeremy Brown, April 2018
- 2. Bio โข Been around for a while in the industry โข Bug hunter โข Hardening products and systems โข Tools and teams that make stuff ./ โข Security Lead @ NVIDIA โข All things ProdSec, breaking/fixing and providing solutions โข Previously Microsoft (windows), Amazon (cloud) and independent consulting
- 3. Agenda I. Intro II. Fundamentals III. Tooling & Automation IV. Campaigns V. Conclusion
- 4. Fundamentals
- 5. NVIDIA is more than just a gaming company now
- 6. Fundamentals โข Security is managing risk โข Identify โข Approach โข Mitigate โข R&D to understand and mitigate future risks โข Defense is trained by offense
- 7. * Server-only perspective, still tons of caveats
- 8. Foundations โข Vertical โข Security persons or teams in each product group โข Horizontal โข Central team that collaborates across product groups โข Hybrid โข Most companies with significant security presence fit somewhere in here
- 9. Roles โข Security devs โข Build security features or tooling, automate to scale โข Security engs โข Work with product team(s) or company-wide initiatives on reducing risk โข SecOps โข Often handle IT security, incidents, network hardening, holistic pen-testing
- 10. Roles โข Pen-testers โข Focus on internal and external testing, even red teaming and service security โข Exploiters / Researchers โข Pushing bugs to maximum potential, assessing blast radius, determining risk โข PSIRT โข Managing mostly externally reported bugs, product team fixes, public comms All have various shades, levels and specialties in between
- 11. What are we trying to do โข Ship less bugs โข Avoid the avoidable โข Limit the blast radius โข Make attacking more difficult in the first place โข If compromise occurs, harden platform / network to mitigate advancement โข Maintain balance โข Security in lieu of usability/perf generally doesnโt ship (or sell) Relevant: https://cansecwest.com/slides/2015/Project%20Zero%20-%20making%200day%20hard%20-%20Ben%20Hawkes.pdf
- 12. Org tools at your disposal โข Culture / Policy โข โWe should do itโ or โAccording to the SDLโ โข Alignment โข โX says we need to do thisโ โข Escalation โข โYour people talk to my peopleโ Of course, always be nice J
- 13. Models โข Training โข Teach them how to fish, but risk losing sight of the rewards long term โข Hands-on โข Work with them every step of the way, but this is hard to scale โข On-boarding process, partnering, general visibility into org โข Opt-in only โข Must use these libraries and check these boxes, but requires culture Hybrid models are common
- 14. MVSB โข Minimally Viable Security Bar โข The basics a dev team has to do without risking their ship getting blocked โข Eg. design review, static analysis scan, pen-test โข Negotiable, yet dedicated % of product roadmap for security activities โข Allows security to be a requirement instead of โnice to have, but no timeโ
- 15. Bug Hunting โข Evaluating assumptions in the productโs design and implementation โข No one should ever send a large buffer, no need to check length before copy โข Local users are trusted anyways, we donโt need to protect data on filesystem โข Only registered accounts can upload files and we shouldnโt restrict them โข Users know they cannot insert scripts here, nothing to worry about โข We trust clients not to upload bad data, we donโt need to check it beforehand โข We want users to be able to run whatever they want, weโll block the bad stuff โข The client should validate data before parsing it, thatโs not the serverโs job โข The service is just open to internal users, so authentication is unnecessary โข [โฆ.]
- 16. Security inside โข Access to source code โข Whitebox all the things โข Reuse data for longer term initiatives to improve code quality โข Coordinated new releases with product team Bugs are filed and queued for fix.
- 17. Security outside โข Only access to OSS, not closed products โข But as they say, if you can read assemblyโฆ โข One bug may affect many different products or only certain versions โข Often negotiated disclosure All bugs are special!
- 18. Having perspective When someone tells you your code has bugs, you can either respond 1) โWhy are they picking on us!?โ or 2) โI wish we were more prepared, but this is good data to use and improve[โฆ]โ Which one is more productive? J
- 19. Having perspective โข Vendors โข Once you release the code, package or service, itโs out there โข Assume anything someone can do with it, they will โข Come to terms with โข You wrote the code, therefore you also wrote the buggy code โข Someone writing an exploit is just exercising the bug that was already there โข Better for you if they report it and itโs fixed than if they donโt
- 20. Having perspective โข Reporters โข It costs money to fix bugs, so each fix must justify the cost โข Key reason why bug bars / risk rating / exploitability index exist โข Come to terms with โข Your bug is unlikely to stop all development just to fix it โข The codebase may be old (internally) and devs working on latest & greatest โข Many things depend on the size of the company and resources available Relevant: https://2017.zeronights.org/wp-content/uploads/materials/ZN17_AndreyLabunets_BootstrappingSecureByDesign.pdf
- 21. Mindset โข Think โvulnerable until proven otherwiseโ โข Instead of asking for proof thereโs bugs or attack surface, assume yes โข Identify and poke each one to see if thereโs merit โข Document & address if there are issues, else document why not vulnerable โข Reengineer systems or transition to new ones that make patching easier if thatโs a pain point
- 22. Mindset โข Map the notion of not doing security to a risky business โข Whatโs going to save us if they break out of this isolation? โข Has this third party software been reviewed? โข Should this parser be in kernel land? โUnsandboxed ImageMagick is an unacceptable liability for any kind of businessโ Reference: https://www.imageflow.io / @micksmix0
- 23. Interacting with developers โข Itโs not enough to point and say โthis is brokenโ โข Each problem you describe should come paired with a solution โข Tons of benefits โข They trust you actually know what the problem is โข They treat you as a partner instead of just a critic (always be constructive) โข Enables reuse of the solutions in the future โข Automate the bug/class away Relevant: https://lcamtuf.blogspot.com/2018/02/getting-product-security-engineering.html
- 24. Strategies โข Integrate into the dev process โข Insert people (or robots) into code repos and product checkpoints โข Make security easier for product team โข Automation and tooling โข Donโt make security optional โข Code gating โข Secure-by-defaults โข Hold folks accountable Relevant: https://lcamtuf.blogspot.com/2018/02/getting-product-security-engineering.html
- 25. Strategies โข De-value where possible โข Do we need to store any user data here? โข Why is access to this service so powerful? โข Could we not keep anything sensitive going between hosts in this network? โข Attackers donโt spend time on worthless targets โข Increase cost, decrease value gained if compromised
- 26. Strategies โข Finish line โข Threat model correct? Y/N โข Triaged static analysis results? Y/N โข Tests? Y/N โข Pen-test? Y/N โข For each Y, attach proof โข For each N, you may not pass (without exception)
- 27. Being Effective โข Easy to say, but surprisingly more difficult to achieve โข Make mistakes non-repeatable where possible โข Automate tasks to make it easier for devs & secengs to find/fix bugs โข Thereโs only so many security folks, so one must scale themselves
- 28. Being Effective โข More security and hardening generally means less bugs โข Less incidents, less pages, less randomizing patches โข Mix this thoroughly within your company culture โข Get data where possible โข Thereโs a difference between doing stuff thatโs fun vs measurably productive โข Focus on making them the same thing
- 30. Static analysis โข Automate code reviews where you can โข A very finely tuned SA tool > team of code reviewers โข Spend your time writing or skimming more than reading โข Donโt just run it one time โข Make it run every time
- 31. Static Analysis โข IDE plugins that mark bad code โข using an annotator or linting โข Input โข printf(line); โข Output โข printf(line); Relevant: https://www.cs.utah.edu/~tdenning/files/papers/baset-ide-plugins.pdf https://www.slideshare.net/cypressdatadefense/continuous-integration-live-static-analysis-with-puma-scan https://github.com/SublimeLinter/SublimeLinter-annotations
- 32. Static Analysis โข Many telemetry opportunities โข Capture these events for metrics on common hits, focus targeted training around these for opportunities say share safer coding alternatives โข Input โข printf(line); โข Output โข printf(line); Relevant: https://www.cs.utah.edu/~tdenning/files/papers/baset-ide-plugins.pdf https://www.slideshare.net/cypressdatadefense/continuous-integration-live-static-analysis-with-puma-scan https://github.com/SublimeLinter/SublimeLinter-annotations
- 33. Static Analysis โข Banning dangerous functions via headers โข Throw errors during compile โข Eg. gcc poison and __attribute__ ((deprecated)) โข Or parse build logs for warnings โข Start campaigns for stomping out ignored bugs โข โTreat warnings as errorsโ approach where practical Reference: https://leafsr.wordpress.com/2013/12/02/gcc-poison/
- 34. Static Analysis โข Gating code via infrastructure โข Client side commit hooks โข Server side scanning for undesirable coding practices โข Eg. credentials in source code is an easy catch โข Another telemetry opportunity โข Gather statistics on which patterns keep getting attempted โข Focus training and bug bashes accordingly Reference: https://github.com/awslabs/git-secrets
- 35. Static Analysis โข Source/Binary Diffing โข How do you know the compiler is producing code as intended? โข Statements that are security-related could be optimized out โข Check if critical checks are missing in the release binary โข โNever underestimate RE in your threat modelโ Relevant work: http://www.eng.utah.edu/~cs5785/slides-f10/Dangerous+Optimizations.pdf
- 36. Static Analysis โข Scanning code upon build โข Hook platforms into code repos โข Scan on-demand, every build or bug bash before a new release
- 37. Static Analysis โข Subscription of code changes โข Addition of interesting patterns โข Modification of critical files or components โข Notification via email, auto-added to CRs, Slack pings โข Also use the code index to gain additionally visibility into projects
- 38. Static Analysis โข Mitigating hardware attacks โข Insert redundancy, other anti-glitching strategies in critical code sections at compile time or pre-commit โข Pros and cons with both transform points, still needs quick manual review
- 39. Static Analysis โข Variant finding โข Input: bad code โข Output: more bad code โข For PSIRT bugs, use reported bug data to find & fix any similar issues โข In the same product or other product lines โข Various algorithms to approach similarity searches
- 41. Static Analysis โข Machine learning โข If you have bug data, why not put it to work? โข Hope to talk about this one next time ;-)
- 42. Product DNA โข What components are native or imported from elsewhere โข Using OSS saves time / !re-inventing the wheel โข But also adds to your attack surface and requires maintenance โข You want to know.. โข If youโre running an old version โข If youโre using an deprecated package โข If youโre no longer relying on code thatโs still accessible
- 44. Dynamic analysis โข Thereโs nothing like running the code โข Understand how it actually works, validate/invalidate assumptions โข Eg. Sanitizers โข Free bugs from just opting-in and running the application Reference: https://github.com/google/sanitizers
- 45. Dynamic analysis โข Re-use unit tests to create fuzzing tests โข Input: tests/* -> mutation engine โข Output: tests/security/* โFreeโ security tests!
- 47. Fuzzing Lab โข If youโre not fuzzing your software, someone else will โข Much easier to find it โfirstโ and solve it in-house โข Generic platform with plugin-based fuzzer system โข Foundations first, then add specific capabilities โข AFL, Libfuzzer, custom mutators, etc โข Make it really ./ to fuzz Relevant work: https://github.com/google/oss-fuzz
- 49. Dynamic Analysis โข Sub-system diffing โข What changes when the app is installed or service turned on? โข Are new DLLs being loaded? โข Network ports, pipes, registry, ACLs, etc Related: https://packetstormsecurity.com/files/140055/Whale-Win32-Attack-Surface-Toolkit.html
- 50. Dynamic Analysis โข Auto-isolation โข Each new app is thoroughly exercised in an emulator โข Based upon behavior, sandbox config is generated โข App is restricted to necessary calls and โknown goodโ โข Rinse and repeat for each application โข Can also use this for offense โข Capture in more detail what the app is doing and poke assumptions
- 51. Campaigns
- 52. Attack Surface Reduction โข Target product lines that ship large codebases โข Do we need to ship every native API? โข Do we need to include all these applications by default? โข Should we remove unused or deprecated components? โข If we turn this off in the config file, does anything fail? โข Also use code coverage data to drive removals โข More code == more attack surface == more bugs
- 53. Automating Information โข Donโt spend time repeating yourself โข Build a knowledge base โข Add reusable content and solutions โข Keep it as a running wiki for everyone to update
- 54. Bug Bashes โข Getting devs + security team together for a day or evening โข Code review/fuzz attack surfaces โข Shake out bugs quickly
- 55. Door Knocking โข Continuously scanning the network for misconfigurations โข Default passwords, weak or no auth, open shares โข FTP, Telnet, Network shares, Web, etc โข Notify server owners with guidance on how to improve โข Document and follow-up
- 56. Hardening APIs โข Make APIs safer to use by default โข ParseStruct() -> HardenedParseStruct() โข Just wraps API to do some extra checks before passing it on โข Code thatโs prone to bugs requires explicit exception or override โข Look for any projects using non-Hardened versions and switch them out Relevant project: http://www.gwtproject.org/doc/latest/DevGuideSecuritySafeHtml.html
- 57. Pen-testing โข Have a process and stick to it โข Try to be frictionless with the intake โข Test the code thatโs shipping, not the previous version โข Use surveys post-test for feedback and continuous improvement โข Vend it out when you need to โข But build up internal capabilities for deeper dives
- 58. Red Teaming โข Blackbox approach for pen-testing โข Trading coverage for simulating real world attacks โข Different perspectives can make external attack surface more clear โข โHeat checksโ can be healthy โข Prioritize and donโt be destructive โข But not on Fridays J
- 59. Training โข Expensive, butโฆ โข The less you know, the more mistakes youโll make โข Can sprinkle this into roadmaps, explicitly or implicitly
- 60. Wide-net hunting โข Using a index to search for particular bug patterns across code bases โข Patterns can come from externally reported bugs (re-using free data) or generically what certain bug classes look like
- 61. Mentoring โข Work on the next generation โข Fundamentals first โข Then how to be effective โข Develop a specialization โข So many distractions out there, focus and be productive
- 62. Conclusion
- 63. Culture โข Do stuff that matters โข Just because itโs cool doesnโt mean it will be effective โข Have an open mind โข Maybe the way weโve been doing it isnโt the best way to do it โข Do the little things โข Thereโs many thankless, little to zero visibility tasks that make a big difference
- 64. Culture โข Externally reported bugs are randomizing by nature โข Get what data you can out of them and use it โข Not productive to โwish awayโ bugs โข There really is no โnot our bugsโ explanation โข Adding external, third party code to your platform can extend attack surface โข It doesnโt not make your platform any less vulnerable โข Evaluate and set expectations with vendors before you buy, or donโt Relevant: http://addxorrol.blogspot.com/2015/12/a-decisionmakers-guide-to-buying.html
- 65. Culture โข Train folks to make the best decisions overall โข The ecosystem includes many different companies, researchers and consumers โข Sometimes it takes time to steer the ship, so think big โข Cross-company > cross-org > intra-team โข Constructive criticism is healthy โข Itโs ok to make mistakes, but be teachable and accept feedback โข Do post-mortems to ensure the same thing doesnโt happen twice
- 66. In Closing โข We discussed many tools and techniques to help you ship less bugs โข Balancing security with shipping a product will have tradeoffs โข Document stuff even if you canโt work on it today โข Prioritize and conquer โข Work on what you think is valuable โข problem:solution > problem