Trusting the Unknown
Download as PPTX, PDF•0 likes•18 views
The document discusses security concerns regarding extensions in Azure DevOps pipelines, highlighting vulnerabilities and the need for cautious use of tasks sourced from public or private marketplaces. It provides guidance for extension authors, administrators, and users, emphasizing best practices for maintaining and updating extensions. Additionally, it outlines Microsoft's efforts to enhance security and offers resources for further information and tools related to Azure DevOps extensions.
![Guidance for deprecating an extension
• Add a warning to all versions of your tasks
• Timebomb the task to fail after today+X
• Mark all tasks deprecated:true in the task.json
• Add [DEPRECATED] to your task friendly name in the task.json
• Add [DEPRECATED] to your extension name in the vss-extension.json
• Publish this last version
• Unpublish the extension
• Archive the GitHub repository](https://image.slidesharecdn.com/trustingtheunknown-230201190550-c2eaab59/85/Trusting-the-Unknown-32-320.jpg)
![Guidance for deprecating task version
• Add both the old and the new version of the task to your extension
• Update the old version’s friendlyName and add [DEPRECATED] in
task.json
• Add a warning / error to the implementation to warn you users](https://image.slidesharecdn.com/trustingtheunknown-230201190550-c2eaab59/85/Trusting-the-Unknown-33-320.jpg)
Trusting the Unknown
- 1. Trusting the Unknown - Ensuring Safe & Secure Extension Use in CI/CD Workflows
- 2. Of extensions contain some vulnerability
- 3. Of extensions owned by Microsoft
- 4. With a ⭐ rating
- 5. Of extensions updated in the last 3 months
- 6. Of Node based Azure Pipelines tasks
- 7. Of PowerShell Azure Pipeline Tasks…
- 8. That’s unfair But it’s also a bit true…
- 10. Jesse Houwing Chief Trainer & Tinkerer xpirit.com/jesse jessehouwing jessehouwing jessehouwing hachyderm.io/jessehouwing +31 6 41 81 333 8
- 13. Extensions in the context of Azure Pipelines Manifests Marketplace pritties Tasks And potentially tabs and context menu actions…
- 14. So what is a task? Loads of dependencies At least azure-pipelines-task-lib Frozen at time of publish Another Manifest The actual custom code More pretties
- 15. Or… using PowerShell A lot fewer dependencies at least VstsTaskSdk Another Manifest The actual custom code More pretties
- 16. An extension can contain more than 1 tasks Task A Task A v1
- 17. Where do these tasks come from? • They’re “built-in” tasks • Automatically updated by Microsoft in Azure DevOps • Manually updated through Microsoft Update packs in Azure DevOps Server • Manually updated by administrator through sideloading (see below) • They’re from an extension from the public marketplace • Installed through marketplace.visualstudio.com • Automatically updated in Azure DevOps and Azure DevOps Server • They’re from an extension from the private marketplace • Manually uploaded to Azure DevOps Server • They’re sideloaded directly • Manually uploaded to Azure DevOps or Azure DevOps Server
- 18. 2 problems with task updates By default, the agent: 1. Pins the major version to the one you selected 2. Updates to the latest minor version available
- 19. Where do these tasks come from?
- 20. So how do we know we can trust these tasks? You don’t.
- 21. How do you know? Well… I downloaded 700GB of vsix files from the Azure DevOps marketplace Extracted them and scanned them
- 22. So how do we know we can trust them? Trusted publisher? Verification? Rating? Number of issues and pull-requests? Metadata provided by the author? Recently updated? Looks nice?
- 23. Is it really that bad? Maybe not…
- 24. All of these potential security issues Do not mean there is a known exploit chain
- 25. But it’s not pretty!
- 26. It’s not just the updates you should worry about
- 27. So what’s the risk we’re running • Some of these extensions turn off TLS security • Some might overwrite arbitrary files • Some might be used to send data to the outside world • Some install things directly from npmjs. • Others download 100’s of MB, extract it and execute without verification • Often on a host with admin permissions There is a lot of potential
- 29. Why is PowerShell so much better? It might not be but… • Many of the dependencies are automatically updated through Windows Update • PowerShell relies on the .NET framework for most functionality • There are a lot fewer dependencies to worry about
- 30. What is Microsoft doing? • They’ve introduced a new Node 16 handler for Azure Pipelines • They’re deprecating Node 6 and Node 10 for Azure Pipelines tasks. • They’re updating all the built-in tasks for Azure DevOps. • They’re updating all the built-in tasks for Azure DevOps Server 2022? • They’ve added the ability to turn off Node 6 on the Pipelines Agent. • They’re adding warnings to every workflow that runs uses Node 6 (and later 10).
- 31. What am I doing / have I done • I actively update all my extensions regularly • I automatically update my dependencies using Snyk, Dependabot and RenovateBot • I scan my code for vulnerabilities using Snyk Code and GitHub advanced Security • I maintain the Azure DevOps Extension Tasks to implement CI / CD for Azure DevOps extensions • I recently contributed to RenovateBot to add support for the Azure Pipeline Marketplace. Now doing the same for dependabot-core.
- 32. Guidance for deprecating an extension • Add a warning to all versions of your tasks • Timebomb the task to fail after today+X • Mark all tasks deprecated:true in the task.json • Add [DEPRECATED] to your task friendly name in the task.json • Add [DEPRECATED] to your extension name in the vss-extension.json • Publish this last version • Unpublish the extension • Archive the GitHub repository
- 33. Guidance for deprecating task version • Add both the old and the new version of the task to your extension • Update the old version’s friendlyName and add [DEPRECATED] in task.json • Add a warning / error to the implementation to warn you users
- 34. What should Extension authors do? • Employ secure password practices and use 2FA • Deprecate extensions you’re no longer maintaining • Update your existing extensions to Node 16 and latest dependencies • Enable DependaBot to automatically keep your dependencies up-to- date • Enable GitHub Advanced Security to detect issues in *your* code • Implement CI / CD to ship new versions with ease • Add repo metadata to their extension manifests
- 35. What should Azure DevOps admins do • Disable insecure extensions • Implement RenovateBot to automatically suggest updates to installed Azure Pipelines tasks • Disable Node 6 support on the Azure Pipelines Agent • Do not install every extension in the marketplace without thought • Provide Hosted Agents or ephemeral Scale- set agents
- 36. What should Azure DevOps admins do • Run Endpoint Security on your Azure Pipelines Agents • For DevOps Server • Upgrade to Azure DevOps Server 2022 and stay current • Migrate to Azure DevOps Service • Manually upgrade “built-in” tasks • Be careful with internal marketplace
- 37. Overwrite a built-in task npm install -g tfx-cli tfx build tasks upload --task-zip-path Task.guid-version.zip --service-url https://yourtfs.com/tfs/DefaultCollection . ./script/install-task.ps1 -CollectionUrl https://yourtfs.com/tfs/DefaultCollection -TaskZip Task.guid-version.zip Delete all versions of a task npm install -g tfx-cli tfx build tasks delete --task-id ad884ca2-732e-4b85-b2d3-ed71bcbd2788 --service-url https://yourtfs.com/tfs/DefaultCollection
- 39. What should Azure Pipelines authors do? • Consider whether you actually need an extension. • Keep your pipelines current. Update your major task versions and resolve any issues. • Use Azure Pipelines YAML • Pin the exact version of your tasks • Use RenovateBot to automatically keep your pipelines up to date • Use Hosted Agents / Ephemeral Scale-set agents whenever possible
- 40. What should the community do? • Sponsor the authors of extensions you rely on. • Submit pull requests to improve extensions. • Submit pull requests to upgrade extensions to Node 16. • Leave reviews on the marketplace and update them.
- 41. Useful resources
- 42. Useful Resources • Azure DevOps Marketplace Scan https://github.com/jessehouwing/azure-devops-marketplace-scan https://jessehouwing.net/security-state-of-the-azure-devops- marketplace/ • Hardening guidelines for Azure Pipelines https://learn.microsoft.com/en- us/azure/devops/organizations/security/security-best- practices?view=azure-devops#secure-azure-pipelines
- 43. Useful resources • Azure Pipelines Tasks Zips https://github.com/jessehouwing/azure-pipelines-tasks-zips • RenovateBot for Azure Pipelines https://jessehouwing.net/azure-pipelines-enable-renovatebot/ • Azure DevOps Extension tasks https://marketplace.visualstudio.com/items?itemName=ms- devlabs.vsts-developer-tools-build-tasks