Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)
1 like574 views
DevOps is increasingly blending the work of both application and network security professionals. In a quest to move faster, organisations can end up creating security vulnerabilities using the tools and products meant to protect them. What happens when these tools are used insecurely or - even worse - they are just insecure? Technologies discussed will encompass AWS, Puppet, Hudson/Jenkins, Vagrant, Docker and much, much more. Everything from common misconfigurations to remote code execution.
![• $ mkdir website-source-code
• $ cd website-source-code
• $ wget –mirror –include-directories=/.git http://
www.example.com/.git
• $ cd www.example.com
• $ git reset –hard
HEAD is now at […]
.git exposure / DirectoryIndex ON](https://image.slidesharecdn.com/devooops-151124153836-lva1-app6891/85/Gianluca-Varisco-DevOoops-Increase-awareness-around-DevOps-infra-security-17-320.jpg)
![[HES2013] Virtually secure, analysis to remote root 0day on an industry leadi...](https://cdn.slidesharecdn.com/ss_thumbnails/day1-04-140412114404-phpapp02-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security) (20)
Ad
More from Codemotion (20)
Ad
Recently uploaded (20)
Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)
- 1. Increase awareness around DevOps infra security DevOoops Gianluca Varisco @gvarisco
- 2. $ whoami VP Security @ Rocket Internet SE Formerly at Red Hat, Lastminute.com Group, PrivateWave
- 3. DevOps is about creating a conveyor belt to systematically pull together all of the pieces that need to go into production using automation to create a safe and reliable application deployment. What is DevOps?
- 4. Why Security needs DevOps
- 5. How vulnerabilities get introduced Configuration errors Missing patch Coding mistakeHuman mistake (bad OPSEC)
- 6. We learned (the HARD WAY) that DevOps is more than giving root to developers…. alias devops=sudo
- 7. DevOps Borat
- 8. 01 GitHub 02 RCS tools 03 CI tools 04 AWS config files 05 Client provisioning tools 06 Elasticsearch 07 In-memory databases Agenda 08 Docker
- 9. GitHub
- 10. It does support “advanced” search operators, eg. • extension:conf ftp server configuration • extension:pem private • extension:xls mail • extension:sql mysql dump • extension:php “preg_replace(“/(.+)/e” (RCE) • OSINT (within companies’ and employees’ repos) GitHub - Search
- 11. Pushing code to GitHub as Linus Torvalds? GitHub – Impersonating others People trust pictures!
- 12. • Hey, look! Linus is the main committer of my github.com:gvarisco/swag.git repo! • Sad truth! Design flaw or targeted feature? Official response below.. GitHub – Impersonating others / 2
- 13. • Always audit who has access to your repos • Be suspicious of pull requests with other authors’ code within the PR. • Always delete a private fork of a private organization repository if a member leaves your organization. • Audit organization members for 2-step verification GitHub – Learnings / TODOs
- 14. RCS Tools
- 15. Does your website expose the .git/ folder on a webserver out there? ▪ Access to such content lets you download the full source code ▪ tl;dr: NO, Turning DirectoryIndex (Apache) / autoindex (nginx) ON/OFF is NOT the fix! .git exposure
- 16. • Source code, config files, credentials, developer names, public/private keys, SSL certificates, e-mail addresses, etc. • Repo HISTORY (security issues fixed, password wrongly committed and removed later) • Archives / backups {My,Postgre,XYZ}SQL dumps • Session generation keys .git exposure / What can you get?
- 17. • $ mkdir website-source-code • $ cd website-source-code • $ wget –mirror –include-directories=/.git http:// www.example.com/.git • $ cd www.example.com • $ git reset –hard HEAD is now at […] .git exposure / DirectoryIndex ON
- 18. • Git-fsck to the rescue! • Bruteforce: Predictable file names and known object hashes, etc. • DVCS-{Pillage,Ripper} do it for you • Many admins tend to answer either 403 or 404 for .git/ but .git/config works just fine. • Git stores file information within the objects folder. .git exposure / DirectoryIndex OFF
- 19. • See the SHA-1 for index.php: • $ git cat-file –p master^{tree} • Take the SHA-1 and give it to ‘git cat-file’ to print out the file contents: Abusing the .git/ Objects folder
- 20. • 1.6.x and earlier • Check for .entries files • Walk SVN chain to retrieve all files. Metasploit does it for you (auxiliary/scanner/http/svn_scanner) Subversion 1.6.x
- 21. • 1.7.x uses SQLite. • Metasploit’s auxiliary/scanner/http/ svn_wcdb_scanner to the rescue! It will retrieve SVN’s wc.db for you • As we know the file name and the SHA-1 used, we can map all files. Subversion 1.7.x
- 22. $ sqlite3 wc.db 'select local_relpath, ".svn/pristine/" || substr(checksum,7,2) || "/" || substr(checksum,7) || ".svn-base" as alpha from NODES;’ index.php|.svn/pristine/4e/4e6a225331f9ae872db25a8f85ae7be05cea6d51.svn-base style/style.js|.svn/pristine/2s/2cc5590e0ba024c3db77a13896da09b39ea74799.svn- base ... $ wget -O - http://www.example.com/.svn/pristine/4e/ 4e6a225331f9ae872db25a8f85ae7be05cea6d51.svn-base <?php Subversion 1.7.x
- 23. CI Tools
- 24. • The leading open-source continuous integration server. • Built in Java, it provides 985 plug-ins to support building and testing virtually any project. • Latest and greatest release: 1.638 • A “few” security advisories… Jenkins
- 25. Jenkins – Searches on Shodan
- 26. • Metasploit modules: • auxiliary/scanner/http/jenkins_enum (It enumerates a remote Jenkins installation in an unauthenticated manner, including host OS and Jenkins installation details) • exploit/multi/http/jenkins_script_console (It uses the Jenkins Groovy script console to execute OS commands using Java.) • If no authentication is required, it is trivial to gain remote code execution via script console. Abusing Jenkins
- 27. • Wanna display jenkins’ user private SSH key? No problem! It is as simple as executing: println new ProcessBuilder('sh','-c','cat /Users/batman/.ssh/ id_rsa').redirectErrorStream(true).start().text Abusing Jenkins – Script console
- 28. • Last, but not least: • If you have access to /view/All/newJob, create new builds and run commands. • Browse WORKSPACEs, read config / folders containing sensitive data, eg. credentials, API keys Abusing Jenkins
- 29. AWS config files
- 30. • ALL credentials are stored in plain-text in “”hidden files””, eg. /home/gvarisco/.foo/bar • Typically privileged accesses • Once credentials are found, any of the OSS libraries available out there can interact with AWS (eg. Nimbostratus, AWS CLI tools) • OSINT / Information leakage via GitHub, Pastebins, etc. AWS config files
- 32. • If you expose a dashboard (eg. PuppetBoard/PuppetDB) be careful with your custom FACTS • Encrypt your sensitive YAML files’ information (if you use HIERA, a key/value lookup tool for config data) with HIERA-EYAML • It does provide asymmetric encryption of sensitive data • Store the keys securely when using puppet, as only the puppetmaster needs access to them in order to perform decryption when the agent runs on a remote node Puppet
- 34. • Web Interface (Chef Server), Rails powered, uses admin / p@ssw0rd1 as default credentials Chef
- 35. • Databags items (eg. MySQL data) can be encrypted • Use knife – a cli tool that provides an interface between a local chef-repo and the Chef server Chef
- 36. • Did you change your SSH keys? • Vagrant 1.7+ embeds vagrant-rekey-ssh plug-in Vagrant
- 37. • Common user/passwords: root/vagrant OR vagrant/vagrant • NO pass to sudo ☹ Vagrant
- 38. Vagrant – Scans using the default private key
- 39. • Vagrant workflows encourage you to edit your code outside the VM. • That’s why it helpfully shares the project directory as /vagrant/ in the VM. “Put evil things in /vagrant/.git/hooks/post-commit and wait for the user to commit some code. Since the /vagrant/ directory is mounted from the host, such hook will persist even if the user destroys the VM.” Vagrant – breaking in!
- 40. • Root passwords are either set: • During installation • Crypted hash defined in the KS file (rootpw –iscrypted) • Clear text defined in the KS file (rootpw –plaintext) Kickstart files (Red Hat, CentOS, …)
- 41. Elasticsearch
- 42. • A distributed full-text search engine with a RESTful web interface and schema-free JSON documents • 9200/TCP (GET request shows version) • No authentication • Can search stored data via HTTP API • Update data with PUT requests • Join an existing, open cluster and get all the data • REMOTE CODE EXECUTION prior to 1.2.0 Elasticsearch
- 43. • Own a server with a query like this (as the search function allows dynamic scripts execution): • 1.3.x adds a sandbox to control what classes and functions can be executed. • Add ‘script.disable_dynamic: true’ to your elasticsearch.yml • Make sure your instance is only binding on localhost Elasticsearch
- 44. Elasticsearch – read inside /etc
- 46. • Default config comes with: • NO encrypted communication • NO credentials • 6379/TCP • Binds to all interfaces (now FIXED in ) Redis
- 47. Redis – Shodan results
- 49. • FLUSHALL (Remove all keys from all databases) • SCRIPT LOAD • EVAL / EVALSHA Redis – “Funny” commands
- 50. • Free & open-source • High-performance, distributed memory object caching system • Fun things get put into memcache • SECURE IT: • First and always, FIREWALL • Check your bindings (interfaces) • If you need it, use SASL • DO NOT RUN AS ROOT Memcache
- 51. Memcache – interesting findings…
- 52. Docker
- 53. • It automates the deployment of applications inside software containers • Docker works as a client that communicates with a daemon process (dockerd) via a Unix domain socket called /var/run/ docker.sock • Highly privileged, effectively having root access Docker
- 54. • Error: # docker run -t -i -v /var/run/docker.sock:/var/run/docker.sock • PoC: • The container gets a docker client of its own, pointed at the /var/run/docker.sock • The container launches a new container mounting / on /host (It’s the host root filesystem, not the first container’s) • The second container chroots to /host, and is now effectively root on the host.. Don’t expose the Docker socket!
- 55. Video
- 57. • Add authentication to Jenkins • Make sure all your tools / systems are only available from/to hosts that need it • Change default private keys / credentials EVERYWHERE • Update to latest versions of all your devops tools And now what?
- 58. • Don’t push DevOps back but rather embrace it. • Participate in or create cookbooks/modules/scripts for security • Check for known security items you don’t want going into production by creating audit scripts Go forward
- 60. • Ken Johnson • Chris Gates • Laurens Van Houtven • Rocket Internet’s Security Team Credits
- 61. © 2015 Rocket Internet SE. All rights reserved.