SlideShare a Scribd company logo
Mateo Burillo - @mateobur
CI / CD / CS
Mateo Burillo
Tech Marketing aka container gamer @ Sysdig
Github: github.com/mateobur
Twitter: @mateobur
About me
Continuous Integration.
Continuous Security
Continuous Delivery.
Continuous Security
Continuous Security
Continuous Security.
How we do (did) security?.
- Establish trust boundaries (dev vs prod)
- Identify, minimise, and harden attack
surfaces
- Reduce scope and access (whitelist whenever
possible)
- Onion layers of protection
- Traceability and testing
Continuous Security.
- DevOps: agile and faster
- Security team: less incidents
- Modify process to bring security
- Security as Code
- Failure: open an issue vs abort deployment
Are containers secure?
- New infrastructure, new layers, new risks
- But we have seen them before:
- DDoS, isolation break-out, injections
- Fast pipeline: skip security?
- This is an opportunity
- More steps: more security onion layers
Security pipeline.
- Build
- Development
- Artifacts (Docker images, sw packages)
- Shipment
- Run-time
Security pipeline: Build.
- Code analysis
- specific vulnerabilities
- licensing
- style
- branch policies: PR, check, merge
- Test Driven Security (TDS)
- OWASP ZAP Scanning
Security pipeline: Build artifacts.
- Container build
- Trusted base image
- Restrict functionality
- Restrict libraries / dependencies
- Multi-stage builds
- Restrict privileges
- root, privileged, host, mounts
Security pipeline: Registry.
- Container scan
- At the CI -> registry step
- Find known vulnerabilities
- CoreOS Red Hat Clair
- Red Hat OpenScap
- Vuls.io
- Other commercial vendors
What we are looking for?.
- Package lists
- Software installed manually (pip, rake, …)
- Static binaries
- Hashes of known vulnerabilities
- Lost credentials
- Docker image layers
Ubuntu: 14.04Ubuntu: 14.04
Apache: 2.2Apache: 2.2
Wordpress: 4.6Wordpress: 4.6 PHP: 7.0PHP: 7.0
Ubuntu: 14.04Ubuntu: 14.04
Apache: 2.2Apache: 2.2
Wordpress: 4.6Wordpress: 4.6 PHP: 7.0PHP: 7.0
Ubuntu: 14.04Ubuntu: 14.04
Apache: 2.2Apache: 2.2
Wordpress: 4.6Wordpress: 4.6 PHP: 7.0PHP: 7.0
How to bring this into the pipeline?.
- CI, build the image
- Scan:
- https://github.com/optiopay/klar
- https://gitlab.com/gitlab-org/clair-scanner
- Push into the registry
Container registries.
- Often they already include scanning:
- DockerHub
- Quay.io
- GCR
- ECR
- ACR
- many!
Security pipeline.
- Build
- Shipment
- Run-time
Security pipeline: Shipment.
- Trust
- enable image signing
- DOCKER_CONTENT_TRUST=1
- Kubernetes ValidatingAdmissionWebhook
- Google Grafeas
- https://github.com/kelseyhightower/grafeas-tutorial
Kubernetes Security pipeline
Security pipeline.
- Build
- Shipment
- Run-time
Security pipeline: Run-time.
- Infrastructure security configuration:
- Host security
- Docker Engine security
- Kubernetes security
- Docker CIS benchmark: docker-bench
- Kubernetes CIS benchmark: kube-bench
Security pipeline: Kubernetes.
- RBAC
- namespaces
- Subjects: users and serviceAccounts
- resources
- Role and ClusterRole
- verbs: LIST, WATCH, GET, UPDATE, PATCH, DELETE
- RoleBindings and ClusterRoleBindings
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]
# This role binding allows "jane" to read pods in the "default" namespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: default
subjects:
- kind: User
name: jane
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Security pipeline: Kubernetes.
- Admission controllers
- DenyEscalatingExec
- NodeRestriction
- PodSecurityPolicy (beta)
- At definition time or Admission,
Grafeas, etc
Security pipeline: Kubernetes.
- Admission controllers: PodSecurityPolicy
- Privileged/hostPID/hostIPC/hostNetwork/
hostPorts
- runAsUser
- volumes/allowedHostPaths/ReadOnlyRootFi
lesystem
- Capabilities
- SELinux/AppArmor/seccomp
Security pipeline: Kubernetes.
- Resource management
resources:
requests:
memory: 512Mi
limits:
memory: 700Mi
- Network Policies:
- Pod overlay networks: cilium, etc
- Classic “iptables” doesn’t work
Security pipeline: Run-time scanning.
- Threat detection
- Network inspection
- Privilege escalation
- Post-mortem analysis and forensics
- Attacks are multiple steps
- Successful attack those are unknown to us
- But we just need to recognize one
Seccomp.
- Application syscall sandboxing
- Create filter (BPF) with allowed syscalls
- Failures-> log message, error return,
and/or kill process
- Docker runs containerized process under a
seccomp profile
- Notable disallowed syscalls:
- clone (creating new namespaces)
- reboot (reboot the host)
- setns (change namespaces)
MAC: SELinux / AppArmor.
- Kernel-level interception/filtering
- features++ && complexity++
- Higher level:
- Actors (process)
- Actions (read/write on files/sockets)
- Targets (files, IPs, ports)
https://sysdig.com/blog/selinux-seccomp-falco-technical-discussion/
- An open-source behavioral activity monitor
- Detects suspicious activity defined by a
set of rules
- Uses Sysdig’s flexible and powerful
filtering expressions
- With full support for
containers/orchestration
Sysdig Falco: sysdig.com/opensource/falco/
A shell is run in a container container.id != host and proc.name = bash
Overwrite system binaries
fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
and write
Container namespace change
evt.type = setns and not proc.name in
(docker, sysdig)
Non-device files written in /dev
(evt.type = create or evt.arg.flags contains O_CREAT)
and proc.name != blkid and fd.directory = /dev and
fd.name != /dev/null
Process tries to access camera
evt.type = open and fd.name = /dev/video0
and not proc.name in (skype, webex)
- macro: proc_is_new
condition: proc.duration <= 5000000000
- rule: Read secret file after startup
desc: >
an attempt to read any secret file (e.g. files containing user/password/authentication
information) Processes might read these files at startup, but not afterwards.
condition: fd.name startswith /etc/secrets and open_read and not proc_is_new
output: >
Sensitive file opened for reading after startup (user=%user.name
command=%proc.cmdline file=%fd.name)
priority: WARNING
Falco real rule example.
Incident response.
Post-mortem and forensics.
What?
Where?
Who?
Why?
logs? SSH into prod? pods are gone
How we did this in the past?.
Monitoring & Security Platform for Containers.
Demo
Danke!
Mateo Burillo - @mateobur
Questions?
Ad

More Related Content

What's hot (20)

Enhancing OpenShift Security for Business Critical Deployments
Enhancing OpenShift Security for Business Critical DeploymentsEnhancing OpenShift Security for Business Critical Deployments
Enhancing OpenShift Security for Business Critical Deployments
DevOps.com
 
Lessons from running potentially malicious code inside containers
Lessons from running potentially malicious code inside containersLessons from running potentially malicious code inside containers
Lessons from running potentially malicious code inside containers
Ben Hall
 
Docker up and running
Docker up and runningDocker up and running
Docker up and running
Victor S. Recio
 
On-Demand Image Resizing from Part of the monolith to Containerized Microserv...
On-Demand Image Resizing from Part of the monolith to Containerized Microserv...On-Demand Image Resizing from Part of the monolith to Containerized Microserv...
On-Demand Image Resizing from Part of the monolith to Containerized Microserv...
Docker, Inc.
 
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
Омские ИТ-субботники
 
Amazon EC2 Container Service in Action
Amazon EC2 Container Service in ActionAmazon EC2 Container Service in Action
Amazon EC2 Container Service in Action
Remotty
 
Docker summit 2015: 以 Docker Swarm 打造多主機叢集環境
Docker summit 2015: 以 Docker Swarm 打造多主機叢集環境Docker summit 2015: 以 Docker Swarm 打造多主機叢集環境
Docker summit 2015: 以 Docker Swarm 打造多主機叢集環境
謝 宗穎
 
Dockerize Me: Distributed PHP applications with Symfony, Docker, Consul and A...
Dockerize Me: Distributed PHP applications with Symfony, Docker, Consul and A...Dockerize Me: Distributed PHP applications with Symfony, Docker, Consul and A...
Dockerize Me: Distributed PHP applications with Symfony, Docker, Consul and A...
Alexey Petrov
 
Introduction to Ansible
Introduction to AnsibleIntroduction to Ansible
Introduction to Ansible
Michael Bahr
 
Ansible Oxford - Cows & Containers
Ansible Oxford - Cows & ContainersAnsible Oxford - Cows & Containers
Ansible Oxford - Cows & Containers
jonatanblue
 
Алексей Петров "Dockerize Me: Distributed PHP applications with Symfony, Dock...
Алексей Петров "Dockerize Me: Distributed PHP applications with Symfony, Dock...Алексей Петров "Dockerize Me: Distributed PHP applications with Symfony, Dock...
Алексей Петров "Dockerize Me: Distributed PHP applications with Symfony, Dock...
Fwdays
 
Under the Hood with Docker Swarm Mode - Drew Erny and Nishant Totla, Docker
Under the Hood with Docker Swarm Mode - Drew Erny and Nishant Totla, DockerUnder the Hood with Docker Swarm Mode - Drew Erny and Nishant Totla, Docker
Under the Hood with Docker Swarm Mode - Drew Erny and Nishant Totla, Docker
Docker, Inc.
 
Docker orchestration using core os and ansible - Ansible IL 2015
Docker orchestration using core os and ansible - Ansible IL 2015Docker orchestration using core os and ansible - Ansible IL 2015
Docker orchestration using core os and ansible - Ansible IL 2015
Leonid Mirsky
 
Kubernetes Boston — Custom High Availability of Kubernetes
Kubernetes Boston — Custom High Availability of KubernetesKubernetes Boston — Custom High Availability of Kubernetes
Kubernetes Boston — Custom High Availability of Kubernetes
Mike Splain
 
Docker 1.11 Presentation
Docker 1.11 PresentationDocker 1.11 Presentation
Docker 1.11 Presentation
Sreenivas Makam
 
IaC and Immutable Infrastructure with Terraform, Сергей Марченко
IaC and Immutable Infrastructure with Terraform, Сергей МарченкоIaC and Immutable Infrastructure with Terraform, Сергей Марченко
IaC and Immutable Infrastructure with Terraform, Сергей Марченко
Sigma Software
 
Scaling Next-Generation Internet TV on AWS With Docker, Packer, and Chef
Scaling Next-Generation Internet TV on AWS With Docker, Packer, and ChefScaling Next-Generation Internet TV on AWS With Docker, Packer, and Chef
Scaling Next-Generation Internet TV on AWS With Docker, Packer, and Chef
bridgetkromhout
 
From pets to cattle - powered by CoreOS, docker, Mesos & nginx
From pets to cattle - powered by CoreOS, docker, Mesos & nginxFrom pets to cattle - powered by CoreOS, docker, Mesos & nginx
From pets to cattle - powered by CoreOS, docker, Mesos & nginx
QAware GmbH
 
Infrastructure Deployment with Docker & Ansible
Infrastructure Deployment with Docker & AnsibleInfrastructure Deployment with Docker & Ansible
Infrastructure Deployment with Docker & Ansible
Robert Reiz
 
Real World Lessons on the Pain Points of Node.JS Application
Real World Lessons on the Pain Points of Node.JS ApplicationReal World Lessons on the Pain Points of Node.JS Application
Real World Lessons on the Pain Points of Node.JS Application
Ben Hall
 
Enhancing OpenShift Security for Business Critical Deployments
Enhancing OpenShift Security for Business Critical DeploymentsEnhancing OpenShift Security for Business Critical Deployments
Enhancing OpenShift Security for Business Critical Deployments
DevOps.com
 
Lessons from running potentially malicious code inside containers
Lessons from running potentially malicious code inside containersLessons from running potentially malicious code inside containers
Lessons from running potentially malicious code inside containers
Ben Hall
 
On-Demand Image Resizing from Part of the monolith to Containerized Microserv...
On-Demand Image Resizing from Part of the monolith to Containerized Microserv...On-Demand Image Resizing from Part of the monolith to Containerized Microserv...
On-Demand Image Resizing from Part of the monolith to Containerized Microserv...
Docker, Inc.
 
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
Омские ИТ-субботники
 
Amazon EC2 Container Service in Action
Amazon EC2 Container Service in ActionAmazon EC2 Container Service in Action
Amazon EC2 Container Service in Action
Remotty
 
Docker summit 2015: 以 Docker Swarm 打造多主機叢集環境
Docker summit 2015: 以 Docker Swarm 打造多主機叢集環境Docker summit 2015: 以 Docker Swarm 打造多主機叢集環境
Docker summit 2015: 以 Docker Swarm 打造多主機叢集環境
謝 宗穎
 
Dockerize Me: Distributed PHP applications with Symfony, Docker, Consul and A...
Dockerize Me: Distributed PHP applications with Symfony, Docker, Consul and A...Dockerize Me: Distributed PHP applications with Symfony, Docker, Consul and A...
Dockerize Me: Distributed PHP applications with Symfony, Docker, Consul and A...
Alexey Petrov
 
Introduction to Ansible
Introduction to AnsibleIntroduction to Ansible
Introduction to Ansible
Michael Bahr
 
Ansible Oxford - Cows & Containers
Ansible Oxford - Cows & ContainersAnsible Oxford - Cows & Containers
Ansible Oxford - Cows & Containers
jonatanblue
 
Алексей Петров "Dockerize Me: Distributed PHP applications with Symfony, Dock...
Алексей Петров "Dockerize Me: Distributed PHP applications with Symfony, Dock...Алексей Петров "Dockerize Me: Distributed PHP applications with Symfony, Dock...
Алексей Петров "Dockerize Me: Distributed PHP applications with Symfony, Dock...
Fwdays
 
Under the Hood with Docker Swarm Mode - Drew Erny and Nishant Totla, Docker
Under the Hood with Docker Swarm Mode - Drew Erny and Nishant Totla, DockerUnder the Hood with Docker Swarm Mode - Drew Erny and Nishant Totla, Docker
Under the Hood with Docker Swarm Mode - Drew Erny and Nishant Totla, Docker
Docker, Inc.
 
Docker orchestration using core os and ansible - Ansible IL 2015
Docker orchestration using core os and ansible - Ansible IL 2015Docker orchestration using core os and ansible - Ansible IL 2015
Docker orchestration using core os and ansible - Ansible IL 2015
Leonid Mirsky
 
Kubernetes Boston — Custom High Availability of Kubernetes
Kubernetes Boston — Custom High Availability of KubernetesKubernetes Boston — Custom High Availability of Kubernetes
Kubernetes Boston — Custom High Availability of Kubernetes
Mike Splain
 
Docker 1.11 Presentation
Docker 1.11 PresentationDocker 1.11 Presentation
Docker 1.11 Presentation
Sreenivas Makam
 
IaC and Immutable Infrastructure with Terraform, Сергей Марченко
IaC and Immutable Infrastructure with Terraform, Сергей МарченкоIaC and Immutable Infrastructure with Terraform, Сергей Марченко
IaC and Immutable Infrastructure with Terraform, Сергей Марченко
Sigma Software
 
Scaling Next-Generation Internet TV on AWS With Docker, Packer, and Chef
Scaling Next-Generation Internet TV on AWS With Docker, Packer, and ChefScaling Next-Generation Internet TV on AWS With Docker, Packer, and Chef
Scaling Next-Generation Internet TV on AWS With Docker, Packer, and Chef
bridgetkromhout
 
From pets to cattle - powered by CoreOS, docker, Mesos & nginx
From pets to cattle - powered by CoreOS, docker, Mesos & nginxFrom pets to cattle - powered by CoreOS, docker, Mesos & nginx
From pets to cattle - powered by CoreOS, docker, Mesos & nginx
QAware GmbH
 
Infrastructure Deployment with Docker & Ansible
Infrastructure Deployment with Docker & AnsibleInfrastructure Deployment with Docker & Ansible
Infrastructure Deployment with Docker & Ansible
Robert Reiz
 
Real World Lessons on the Pain Points of Node.JS Application
Real World Lessons on the Pain Points of Node.JS ApplicationReal World Lessons on the Pain Points of Node.JS Application
Real World Lessons on the Pain Points of Node.JS Application
Ben Hall
 

Similar to Continuous Security (20)

WTF my container just spawned a shell!
WTF my container just spawned a shell!WTF my container just spawned a shell!
WTF my container just spawned a shell!
Sysdig
 
Lions, Tigers and Deers: What building zoos can teach us about securing micro...
Lions, Tigers and Deers: What building zoos can teach us about securing micro...Lions, Tigers and Deers: What building zoos can teach us about securing micro...
Lions, Tigers and Deers: What building zoos can teach us about securing micro...
Sysdig
 
Containerizing your Security Operations Center
Containerizing your Security Operations CenterContainerizing your Security Operations Center
Containerizing your Security Operations Center
Jimmy Mesta
 
Introduction to InSpec and 1.0 release update
Introduction to InSpec and 1.0 release updateIntroduction to InSpec and 1.0 release update
Introduction to InSpec and 1.0 release update
Alex Pop
 
Scaleable PHP Applications in Kubernetes
Scaleable PHP Applications in KubernetesScaleable PHP Applications in Kubernetes
Scaleable PHP Applications in Kubernetes
Robert Lemke
 
Road to Opscon (Pisa '15) - DevOoops
Road to Opscon (Pisa '15) - DevOoopsRoad to Opscon (Pisa '15) - DevOoops
Road to Opscon (Pisa '15) - DevOoops
Gianluca Varisco
 
Dockers zero to hero
Dockers zero to heroDockers zero to hero
Dockers zero to hero
Nicolas De Loof
 
Docker - Demo on PHP Application deployment
Docker - Demo on PHP Application deployment Docker - Demo on PHP Application deployment
Docker - Demo on PHP Application deployment
Arun prasath
 
Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)
Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)
Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)
Codemotion
 
Securing Hadoop with OSSEC
Securing Hadoop with OSSECSecuring Hadoop with OSSEC
Securing Hadoop with OSSEC
Vic Hargrave
 
Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)
HungWei Chiu
 
Containerization is more than the new Virtualization: enabling separation of ...
Containerization is more than the new Virtualization: enabling separation of ...Containerization is more than the new Virtualization: enabling separation of ...
Containerization is more than the new Virtualization: enabling separation of ...
Jérôme Petazzoni
 
Container security
Container securityContainer security
Container security
Anthony Chow
 
Containerization Is More than the New Virtualization
Containerization Is More than the New VirtualizationContainerization Is More than the New Virtualization
Containerization Is More than the New Virtualization
C4Media
 
Uyuni Saltboot - automated image deployment and lifecycle with Uyuni
Uyuni Saltboot - automated image deployment and lifecycle with Uyuni Uyuni Saltboot - automated image deployment and lifecycle with Uyuni
Uyuni Saltboot - automated image deployment and lifecycle with Uyuni
Ondrej Holecek
 
Container Monitoring with Sysdig
Container Monitoring with SysdigContainer Monitoring with Sysdig
Container Monitoring with Sysdig
Sreenivas Makam
 
Kubernetes 101 for_penetration_testers_-_null_mumbai
Kubernetes 101 for_penetration_testers_-_null_mumbaiKubernetes 101 for_penetration_testers_-_null_mumbai
Kubernetes 101 for_penetration_testers_-_null_mumbai
n|u - The Open Security Community
 
How Honestbee Does CI/CD on Kubernetes - Vincent DeSmet
How Honestbee Does CI/CD on Kubernetes - Vincent DeSmetHow Honestbee Does CI/CD on Kubernetes - Vincent DeSmet
How Honestbee Does CI/CD on Kubernetes - Vincent DeSmet
DevOpsDaysJKT
 
Secrets management vault cncf meetup
Secrets management vault cncf meetupSecrets management vault cncf meetup
Secrets management vault cncf meetup
Juraj Hantak
 
Docker-v3.pdf
Docker-v3.pdfDocker-v3.pdf
Docker-v3.pdf
Bruno Cornec
 
WTF my container just spawned a shell!
WTF my container just spawned a shell!WTF my container just spawned a shell!
WTF my container just spawned a shell!
Sysdig
 
Lions, Tigers and Deers: What building zoos can teach us about securing micro...
Lions, Tigers and Deers: What building zoos can teach us about securing micro...Lions, Tigers and Deers: What building zoos can teach us about securing micro...
Lions, Tigers and Deers: What building zoos can teach us about securing micro...
Sysdig
 
Containerizing your Security Operations Center
Containerizing your Security Operations CenterContainerizing your Security Operations Center
Containerizing your Security Operations Center
Jimmy Mesta
 
Introduction to InSpec and 1.0 release update
Introduction to InSpec and 1.0 release updateIntroduction to InSpec and 1.0 release update
Introduction to InSpec and 1.0 release update
Alex Pop
 
Scaleable PHP Applications in Kubernetes
Scaleable PHP Applications in KubernetesScaleable PHP Applications in Kubernetes
Scaleable PHP Applications in Kubernetes
Robert Lemke
 
Road to Opscon (Pisa '15) - DevOoops
Road to Opscon (Pisa '15) - DevOoopsRoad to Opscon (Pisa '15) - DevOoops
Road to Opscon (Pisa '15) - DevOoops
Gianluca Varisco
 
Docker - Demo on PHP Application deployment
Docker - Demo on PHP Application deployment Docker - Demo on PHP Application deployment
Docker - Demo on PHP Application deployment
Arun prasath
 
Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)
Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)
Gianluca Varisco - DevOoops (Increase awareness around DevOps infra security)
Codemotion
 
Securing Hadoop with OSSEC
Securing Hadoop with OSSECSecuring Hadoop with OSSEC
Securing Hadoop with OSSEC
Vic Hargrave
 
Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)
HungWei Chiu
 
Containerization is more than the new Virtualization: enabling separation of ...
Containerization is more than the new Virtualization: enabling separation of ...Containerization is more than the new Virtualization: enabling separation of ...
Containerization is more than the new Virtualization: enabling separation of ...
Jérôme Petazzoni
 
Container security
Container securityContainer security
Container security
Anthony Chow
 
Containerization Is More than the New Virtualization
Containerization Is More than the New VirtualizationContainerization Is More than the New Virtualization
Containerization Is More than the New Virtualization
C4Media
 
Uyuni Saltboot - automated image deployment and lifecycle with Uyuni
Uyuni Saltboot - automated image deployment and lifecycle with Uyuni Uyuni Saltboot - automated image deployment and lifecycle with Uyuni
Uyuni Saltboot - automated image deployment and lifecycle with Uyuni
Ondrej Holecek
 
Container Monitoring with Sysdig
Container Monitoring with SysdigContainer Monitoring with Sysdig
Container Monitoring with Sysdig
Sreenivas Makam
 
How Honestbee Does CI/CD on Kubernetes - Vincent DeSmet
How Honestbee Does CI/CD on Kubernetes - Vincent DeSmetHow Honestbee Does CI/CD on Kubernetes - Vincent DeSmet
How Honestbee Does CI/CD on Kubernetes - Vincent DeSmet
DevOpsDaysJKT
 
Secrets management vault cncf meetup
Secrets management vault cncf meetupSecrets management vault cncf meetup
Secrets management vault cncf meetup
Juraj Hantak
 
Ad

More from Sysdig (15)

The top 5 Kubernetes metrics to monitor
The top 5 Kubernetes metrics to monitorThe top 5 Kubernetes metrics to monitor
The top 5 Kubernetes metrics to monitor
Sysdig
 
Behavioural activity monitoring on CoreOS with Sysdig Falco
Behavioural activity monitoring on CoreOS with Sysdig FalcoBehavioural activity monitoring on CoreOS with Sysdig Falco
Behavioural activity monitoring on CoreOS with Sysdig Falco
Sysdig
 
How to Monitor Microservices
How to Monitor MicroservicesHow to Monitor Microservices
How to Monitor Microservices
Sysdig
 
Trace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdminsTrace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdmins
Sysdig
 
You're monitoring Kubernetes Wrong
You're monitoring Kubernetes WrongYou're monitoring Kubernetes Wrong
You're monitoring Kubernetes Wrong
Sysdig
 
The Dark Art of Container Monitoring - Spanish
The Dark Art of Container Monitoring - SpanishThe Dark Art of Container Monitoring - Spanish
The Dark Art of Container Monitoring - Spanish
Sysdig
 
Building Trustworthy Containers
Building Trustworthy ContainersBuilding Trustworthy Containers
Building Trustworthy Containers
Sysdig
 
A brief history of system calls
A brief history of system callsA brief history of system calls
A brief history of system calls
Sysdig
 
Designing Tracing Tools
Designing Tracing ToolsDesigning Tracing Tools
Designing Tracing Tools
Sysdig
 
Extending Sysdig with Chisel
Extending Sysdig with ChiselExtending Sysdig with Chisel
Extending Sysdig with Chisel
Sysdig
 
Intro to sysdig in 15 minutes
Intro to sysdig in 15 minutesIntro to sysdig in 15 minutes
Intro to sysdig in 15 minutes
Sysdig
 
Troubleshooting Kubernetes
Troubleshooting KubernetesTroubleshooting Kubernetes
Troubleshooting Kubernetes
Sysdig
 
Find the Hacker
Find the HackerFind the Hacker
Find the Hacker
Sysdig
 
How to Secure Containers
How to Secure ContainersHow to Secure Containers
How to Secure Containers
Sysdig
 
Sysdig Meetup - San Francisco, December 2014
Sysdig Meetup - San Francisco, December 2014Sysdig Meetup - San Francisco, December 2014
Sysdig Meetup - San Francisco, December 2014
Sysdig
 
The top 5 Kubernetes metrics to monitor
The top 5 Kubernetes metrics to monitorThe top 5 Kubernetes metrics to monitor
The top 5 Kubernetes metrics to monitor
Sysdig
 
Behavioural activity monitoring on CoreOS with Sysdig Falco
Behavioural activity monitoring on CoreOS with Sysdig FalcoBehavioural activity monitoring on CoreOS with Sysdig Falco
Behavioural activity monitoring on CoreOS with Sysdig Falco
Sysdig
 
How to Monitor Microservices
How to Monitor MicroservicesHow to Monitor Microservices
How to Monitor Microservices
Sysdig
 
Trace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdminsTrace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdmins
Sysdig
 
You're monitoring Kubernetes Wrong
You're monitoring Kubernetes WrongYou're monitoring Kubernetes Wrong
You're monitoring Kubernetes Wrong
Sysdig
 
The Dark Art of Container Monitoring - Spanish
The Dark Art of Container Monitoring - SpanishThe Dark Art of Container Monitoring - Spanish
The Dark Art of Container Monitoring - Spanish
Sysdig
 
Building Trustworthy Containers
Building Trustworthy ContainersBuilding Trustworthy Containers
Building Trustworthy Containers
Sysdig
 
A brief history of system calls
A brief history of system callsA brief history of system calls
A brief history of system calls
Sysdig
 
Designing Tracing Tools
Designing Tracing ToolsDesigning Tracing Tools
Designing Tracing Tools
Sysdig
 
Extending Sysdig with Chisel
Extending Sysdig with ChiselExtending Sysdig with Chisel
Extending Sysdig with Chisel
Sysdig
 
Intro to sysdig in 15 minutes
Intro to sysdig in 15 minutesIntro to sysdig in 15 minutes
Intro to sysdig in 15 minutes
Sysdig
 
Troubleshooting Kubernetes
Troubleshooting KubernetesTroubleshooting Kubernetes
Troubleshooting Kubernetes
Sysdig
 
Find the Hacker
Find the HackerFind the Hacker
Find the Hacker
Sysdig
 
How to Secure Containers
How to Secure ContainersHow to Secure Containers
How to Secure Containers
Sysdig
 
Sysdig Meetup - San Francisco, December 2014
Sysdig Meetup - San Francisco, December 2014Sysdig Meetup - San Francisco, December 2014
Sysdig Meetup - San Francisco, December 2014
Sysdig
 
Ad

Recently uploaded (20)

F-Secure Freedome VPN 2025 Crack Plus Activation New Version
F-Secure Freedome VPN 2025 Crack Plus Activation  New VersionF-Secure Freedome VPN 2025 Crack Plus Activation  New Version
F-Secure Freedome VPN 2025 Crack Plus Activation New Version
saimabibi60507
 
Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)
Allon Mureinik
 
Adobe After Effects Crack FREE FRESH version 2025
Adobe After Effects Crack FREE FRESH version 2025Adobe After Effects Crack FREE FRESH version 2025
Adobe After Effects Crack FREE FRESH version 2025
kashifyounis067
 
PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025
mu394968
 
Designing AI-Powered APIs on Azure: Best Practices& Considerations
Designing AI-Powered APIs on Azure: Best Practices& ConsiderationsDesigning AI-Powered APIs on Azure: Best Practices& Considerations
Designing AI-Powered APIs on Azure: Best Practices& Considerations
Dinusha Kumarasiri
 
Download Wondershare Filmora Crack [2025] With Latest
Download Wondershare Filmora Crack [2025] With LatestDownload Wondershare Filmora Crack [2025] With Latest
Download Wondershare Filmora Crack [2025] With Latest
tahirabibi60507
 
WinRAR Crack for Windows (100% Working 2025)
WinRAR Crack for Windows (100% Working 2025)WinRAR Crack for Windows (100% Working 2025)
WinRAR Crack for Windows (100% Working 2025)
sh607827
 
How to Optimize Your AWS Environment for Improved Cloud Performance
How to Optimize Your AWS Environment for Improved Cloud PerformanceHow to Optimize Your AWS Environment for Improved Cloud Performance
How to Optimize Your AWS Environment for Improved Cloud Performance
ThousandEyes
 
Expand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchangeExpand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchange
Fexle Services Pvt. Ltd.
 
Landscape of Requirements Engineering for/by AI through Literature Review
Landscape of Requirements Engineering for/by AI through Literature ReviewLandscape of Requirements Engineering for/by AI through Literature Review
Landscape of Requirements Engineering for/by AI through Literature Review
Hironori Washizaki
 
How can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptxHow can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptx
laravinson24
 
FL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full VersionFL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full Version
tahirabibi60507
 
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
Andre Hora
 
Not So Common Memory Leaks in Java Webinar
Not So Common Memory Leaks in Java WebinarNot So Common Memory Leaks in Java Webinar
Not So Common Memory Leaks in Java Webinar
Tier1 app
 
Adobe Lightroom Classic Crack FREE Latest link 2025
Adobe Lightroom Classic Crack FREE Latest link 2025Adobe Lightroom Classic Crack FREE Latest link 2025
Adobe Lightroom Classic Crack FREE Latest link 2025
kashifyounis067
 
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Andre Hora
 
How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...
How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...
How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...
Egor Kaleynik
 
Revolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptxRevolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptx
nidhisingh691197
 
Solidworks Crack 2025 latest new + license code
Solidworks Crack 2025 latest new + license codeSolidworks Crack 2025 latest new + license code
Solidworks Crack 2025 latest new + license code
aneelaramzan63
 
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Lionel Briand
 
F-Secure Freedome VPN 2025 Crack Plus Activation New Version
F-Secure Freedome VPN 2025 Crack Plus Activation  New VersionF-Secure Freedome VPN 2025 Crack Plus Activation  New Version
F-Secure Freedome VPN 2025 Crack Plus Activation New Version
saimabibi60507
 
Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)Who Watches the Watchmen (SciFiDevCon 2025)
Who Watches the Watchmen (SciFiDevCon 2025)
Allon Mureinik
 
Adobe After Effects Crack FREE FRESH version 2025
Adobe After Effects Crack FREE FRESH version 2025Adobe After Effects Crack FREE FRESH version 2025
Adobe After Effects Crack FREE FRESH version 2025
kashifyounis067
 
PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025
mu394968
 
Designing AI-Powered APIs on Azure: Best Practices& Considerations
Designing AI-Powered APIs on Azure: Best Practices& ConsiderationsDesigning AI-Powered APIs on Azure: Best Practices& Considerations
Designing AI-Powered APIs on Azure: Best Practices& Considerations
Dinusha Kumarasiri
 
Download Wondershare Filmora Crack [2025] With Latest
Download Wondershare Filmora Crack [2025] With LatestDownload Wondershare Filmora Crack [2025] With Latest
Download Wondershare Filmora Crack [2025] With Latest
tahirabibi60507
 
WinRAR Crack for Windows (100% Working 2025)
WinRAR Crack for Windows (100% Working 2025)WinRAR Crack for Windows (100% Working 2025)
WinRAR Crack for Windows (100% Working 2025)
sh607827
 
How to Optimize Your AWS Environment for Improved Cloud Performance
How to Optimize Your AWS Environment for Improved Cloud PerformanceHow to Optimize Your AWS Environment for Improved Cloud Performance
How to Optimize Your AWS Environment for Improved Cloud Performance
ThousandEyes
 
Expand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchangeExpand your AI adoption with AgentExchange
Expand your AI adoption with AgentExchange
Fexle Services Pvt. Ltd.
 
Landscape of Requirements Engineering for/by AI through Literature Review
Landscape of Requirements Engineering for/by AI through Literature ReviewLandscape of Requirements Engineering for/by AI through Literature Review
Landscape of Requirements Engineering for/by AI through Literature Review
Hironori Washizaki
 
How can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptxHow can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptx
laravinson24
 
FL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full VersionFL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full Version
tahirabibi60507
 
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
Andre Hora
 
Not So Common Memory Leaks in Java Webinar
Not So Common Memory Leaks in Java WebinarNot So Common Memory Leaks in Java Webinar
Not So Common Memory Leaks in Java Webinar
Tier1 app
 
Adobe Lightroom Classic Crack FREE Latest link 2025
Adobe Lightroom Classic Crack FREE Latest link 2025Adobe Lightroom Classic Crack FREE Latest link 2025
Adobe Lightroom Classic Crack FREE Latest link 2025
kashifyounis067
 
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)
Andre Hora
 
How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...
How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...
How Valletta helped healthcare SaaS to transform QA and compliance to grow wi...
Egor Kaleynik
 
Revolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptxRevolutionizing Residential Wi-Fi PPT.pptx
Revolutionizing Residential Wi-Fi PPT.pptx
nidhisingh691197
 
Solidworks Crack 2025 latest new + license code
Solidworks Crack 2025 latest new + license codeSolidworks Crack 2025 latest new + license code
Solidworks Crack 2025 latest new + license code
aneelaramzan63
 
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Requirements in Engineering AI- Enabled Systems: Open Problems and Safe AI Sy...
Lionel Briand
 

Continuous Security

  • 1. Mateo Burillo - @mateobur CI / CD / CS
  • 2. Mateo Burillo Tech Marketing aka container gamer @ Sysdig Github: github.com/mateobur Twitter: @mateobur About me
  • 9. How we do (did) security?. - Establish trust boundaries (dev vs prod) - Identify, minimise, and harden attack surfaces - Reduce scope and access (whitelist whenever possible) - Onion layers of protection - Traceability and testing
  • 10. Continuous Security. - DevOps: agile and faster - Security team: less incidents - Modify process to bring security - Security as Code - Failure: open an issue vs abort deployment
  • 11. Are containers secure? - New infrastructure, new layers, new risks - But we have seen them before: - DDoS, isolation break-out, injections - Fast pipeline: skip security? - This is an opportunity - More steps: more security onion layers
  • 12. Security pipeline. - Build - Development - Artifacts (Docker images, sw packages) - Shipment - Run-time
  • 13. Security pipeline: Build. - Code analysis - specific vulnerabilities - licensing - style - branch policies: PR, check, merge - Test Driven Security (TDS) - OWASP ZAP Scanning
  • 14. Security pipeline: Build artifacts. - Container build - Trusted base image - Restrict functionality - Restrict libraries / dependencies - Multi-stage builds - Restrict privileges - root, privileged, host, mounts
  • 15. Security pipeline: Registry. - Container scan - At the CI -> registry step - Find known vulnerabilities - CoreOS Red Hat Clair - Red Hat OpenScap - Vuls.io - Other commercial vendors
  • 16. What we are looking for?. - Package lists - Software installed manually (pip, rake, …) - Static binaries - Hashes of known vulnerabilities - Lost credentials - Docker image layers
  • 17. Ubuntu: 14.04Ubuntu: 14.04 Apache: 2.2Apache: 2.2 Wordpress: 4.6Wordpress: 4.6 PHP: 7.0PHP: 7.0
  • 18. Ubuntu: 14.04Ubuntu: 14.04 Apache: 2.2Apache: 2.2 Wordpress: 4.6Wordpress: 4.6 PHP: 7.0PHP: 7.0
  • 19. Ubuntu: 14.04Ubuntu: 14.04 Apache: 2.2Apache: 2.2 Wordpress: 4.6Wordpress: 4.6 PHP: 7.0PHP: 7.0
  • 20. How to bring this into the pipeline?. - CI, build the image - Scan: - https://github.com/optiopay/klar - https://gitlab.com/gitlab-org/clair-scanner - Push into the registry
  • 21. Container registries. - Often they already include scanning: - DockerHub - Quay.io - GCR - ECR - ACR - many!
  • 22. Security pipeline. - Build - Shipment - Run-time
  • 23. Security pipeline: Shipment. - Trust - enable image signing - DOCKER_CONTENT_TRUST=1 - Kubernetes ValidatingAdmissionWebhook - Google Grafeas - https://github.com/kelseyhightower/grafeas-tutorial
  • 25. Security pipeline. - Build - Shipment - Run-time
  • 26. Security pipeline: Run-time. - Infrastructure security configuration: - Host security - Docker Engine security - Kubernetes security - Docker CIS benchmark: docker-bench - Kubernetes CIS benchmark: kube-bench
  • 27. Security pipeline: Kubernetes. - RBAC - namespaces - Subjects: users and serviceAccounts - resources - Role and ClusterRole - verbs: LIST, WATCH, GET, UPDATE, PATCH, DELETE - RoleBindings and ClusterRoleBindings
  • 28. kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: pod-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["get", "watch", "list"] # This role binding allows "jane" to read pods in the "default" namespace. kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-pods namespace: default subjects: - kind: User name: jane apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io
  • 29. Security pipeline: Kubernetes. - Admission controllers - DenyEscalatingExec - NodeRestriction - PodSecurityPolicy (beta) - At definition time or Admission, Grafeas, etc
  • 30. Security pipeline: Kubernetes. - Admission controllers: PodSecurityPolicy - Privileged/hostPID/hostIPC/hostNetwork/ hostPorts - runAsUser - volumes/allowedHostPaths/ReadOnlyRootFi lesystem - Capabilities - SELinux/AppArmor/seccomp
  • 31. Security pipeline: Kubernetes. - Resource management resources: requests: memory: 512Mi limits: memory: 700Mi - Network Policies: - Pod overlay networks: cilium, etc - Classic “iptables” doesn’t work
  • 32. Security pipeline: Run-time scanning. - Threat detection - Network inspection - Privilege escalation - Post-mortem analysis and forensics - Attacks are multiple steps - Successful attack those are unknown to us - But we just need to recognize one
  • 33. Seccomp. - Application syscall sandboxing - Create filter (BPF) with allowed syscalls - Failures-> log message, error return, and/or kill process - Docker runs containerized process under a seccomp profile - Notable disallowed syscalls: - clone (creating new namespaces) - reboot (reboot the host) - setns (change namespaces)
  • 34. MAC: SELinux / AppArmor. - Kernel-level interception/filtering - features++ && complexity++ - Higher level: - Actors (process) - Actions (read/write on files/sockets) - Targets (files, IPs, ports) https://sysdig.com/blog/selinux-seccomp-falco-technical-discussion/
  • 35. - An open-source behavioral activity monitor - Detects suspicious activity defined by a set of rules - Uses Sysdig’s flexible and powerful filtering expressions - With full support for containers/orchestration Sysdig Falco: sysdig.com/opensource/falco/
  • 36. A shell is run in a container container.id != host and proc.name = bash Overwrite system binaries fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin) and write Container namespace change evt.type = setns and not proc.name in (docker, sysdig) Non-device files written in /dev (evt.type = create or evt.arg.flags contains O_CREAT) and proc.name != blkid and fd.directory = /dev and fd.name != /dev/null Process tries to access camera evt.type = open and fd.name = /dev/video0 and not proc.name in (skype, webex)
  • 37. - macro: proc_is_new condition: proc.duration <= 5000000000 - rule: Read secret file after startup desc: > an attempt to read any secret file (e.g. files containing user/password/authentication information) Processes might read these files at startup, but not afterwards. condition: fd.name startswith /etc/secrets and open_read and not proc_is_new output: > Sensitive file opened for reading after startup (user=%user.name command=%proc.cmdline file=%fd.name) priority: WARNING Falco real rule example.
  • 40. How we did this in the past?.
  • 41. Monitoring & Security Platform for Containers.
  • 42. Demo
  • 43. Danke! Mateo Burillo - @mateobur Questions?

Editor's Notes

  • #2: &amp;lt;number&amp;gt;
  • #3: &amp;lt;number&amp;gt;
  • #4: Let’s start defining CI CD and their relation to the container world
  • #5: &amp;lt;number&amp;gt;
  • #7: &amp;lt;number&amp;gt;
  • #8: &amp;lt;number&amp;gt;
  • #10: &amp;lt;number&amp;gt;
  • #11: &amp;lt;number&amp;gt;
  • #12: &amp;lt;number&amp;gt;
  • #13: &amp;lt;number&amp;gt;
  • #14: &amp;lt;number&amp;gt;
  • #15: &amp;lt;number&amp;gt;
  • #16: &amp;lt;number&amp;gt;
  • #17: &amp;lt;number&amp;gt;
  • #18: &amp;lt;number&amp;gt;
  • #19: &amp;lt;number&amp;gt;
  • #20: &amp;lt;number&amp;gt;
  • #21: &amp;lt;number&amp;gt;
  • #22: &amp;lt;number&amp;gt;
  • #23: &amp;lt;number&amp;gt;
  • #24: &amp;lt;number&amp;gt;
  • #25: &amp;lt;number&amp;gt;
  • #26: &amp;lt;number&amp;gt;
  • #27: &amp;lt;number&amp;gt;
  • #28: &amp;lt;number&amp;gt;
  • #29: &amp;lt;number&amp;gt;
  • #30: &amp;lt;number&amp;gt;
  • #31: &amp;lt;number&amp;gt;
  • #32: &amp;lt;number&amp;gt;
  • #33: &amp;lt;number&amp;gt;
  • #34: &amp;lt;number&amp;gt;
  • #35: &amp;lt;number&amp;gt;
  • #36: &amp;lt;number&amp;gt;
  • #37: Workhourse - Basic
  • #38: &amp;lt;number&amp;gt;
  • #40: &amp;lt;number&amp;gt;
  • #41: &amp;lt;number&amp;gt;
  • #42: &amp;lt;number&amp;gt;