[Konveyor] adding security to dev ops for your kubernetes native applications

Editor's Notes

• #2: Hello All, Welcome and Thank you for taking your time and joining us today
• #5: I am Krishnan Narayana swamy, started my career as a software developer and right after my graduation, few bunch of college mates, we Founded a startup named telesto based on open source lucene search project in 2008 and i have had great association and passion with open source technologies since then. I have in the past and continue to help lot of customers to innovate and adopt open source technologies, especially kubernetes and applications that run on kubernetes, in the best way possible and that is exactly my role as a Specialist Solutions Architect at Redhat ANZ. I am based out of Melbourne in Australia. Feel free to connect with me on linked in and explore my github repositories for my experimentations or demos in the kubernetes space.
• #6: I’m a developer and I worked in infrastructure before and I don’t think I thought about security a single time. Basically, what I would do was develop code, make sure it worked on docker containers, push it, and hope for the best; and now looking at it from a security point-of-view, it probably wasn’t the best way to go. So, how do we build security into rolling out our software? How to bake security into your Dev and Ops processes for your applications running in our kubernetes clusters. Kubernetes is Awesome, but the out of the box open source version has little for security. We will discuss why container security is important, some of the kubernetes security best practices and how stackrox kind of bridges the gap here. Stackrox is a recent acquisition by Red hat and we have decided to open source and we are in the process of making the platform open source soon. We will look at the architecture of the platform and its components and how it enables cloud native security for kubernetes clusters and the applications that run in them We will see a demonstration of the product and how it helps in continuous security across your DevOps process and how it integrates into your CI/CD process enabling your Shift Left on security. Let us assume, you have a kubernetes native application deployed and have a solid DevOps practice set up Continuous Integration = Development practice in which developers integrate code into a shared repository (main code path/trunk) every commit; Automated build and unit tests are run to validate the changes. In this way, defects and integration errors are verified and remedied earlier in the process. Continuous Delivery = Orchestrating your build through a series of quality gates with automated/manual approval processes, however the build stops before reaching production - typically additional validations are required (manual, compliance) Continuous Deployment = And extension of Continuous Delivery where the build is pushed automatically from commit to production if all the quality gates are met DevOps-driven adoption of new technologies and processes may leave security as an afterthought or, in some instances, expose new gaps in security coverage and risk management.
• #7: Let us assume, you have a kubernetes native application deployed and have a solid DevOps practice set up Continuous Integration = Development practice in which developers integrate code into a shared repository (main code path/trunk) every commit; Automated build and unit tests are run to validate the changes. In this way, defects and integration errors are verified and remedied earlier in the process. Continuous Delivery = Orchestrating your build through a series of quality gates with automated/manual approval processes, however the build stops before reaching production - typically additional validations are required (manual, compliance) Continuous Deployment = And extension of Continuous Delivery where the build is pushed automatically from commit to production if all the quality gates are met DevOps-driven adoption of new technologies and processes may leave security as an afterthought or, in some instances, expose new gaps in security coverage and risk management.
• #8: The defaults in upstream Kubernetes can not be assumed to provide adequate protection based on our use case A common reason for attacking containers today is to abuse compute resources, for example, for cryptocurrency mining. can also offer access to customer or workload data. But attacking the container chance to propagate to other nodes in the cluster and also gain persistent access to valuable user code, compute and/or data receives privileged access, they could potentially access information running in the other containers. The Kubernetes master controls your cluster. An attacker that can compromise the master can control the environment, including the ability to take it offline. And a compromised etcd can mean the ability to modify or destroy the cluster, steal secrets and credentials, or gain enough information about the application it’s running to go recreate it somewhere else.
• #9: Lets discuss some of the best practices out there At the infrastructure level, i recommend keeping the kubernetes cluster up to date, Make sure that your network access is firewalled off correctly. Make sure the Kubernetes API server (which is basically the entry point into all of your infrastructure) is locked down. (I know a lot of people will restrict it to a VPC or VPN. Just ensure that the traffic to the API server is protected.) Lastly, let’s make sure that the actual host itself is locked down. Leverage a matured CI/CD processes to deploy applications to kubernetes cluster. One service account authorized against your API server. Obviously, you still have people who need access to the cluster, you still need to debug and fix, but now since you have fewer users and groups, you have a much smaller number of people you need to manage in RBAC. From an application / container stand point, It starts with building your image, which means looking at the dependencies, the packages, and most importantly analyzing the vulnerabilities to discover what we already know is bad. Then, you look at how the app is configured: What privileges does it have on host os? What privileges does it have against the API server? Are there weak secrets or other sensitive data (API keys for registries, sensitive databases)? What labels and annotations are used? This is key because it allows you to to answer the question of who owns a given service (annotation of owner, email, or team) and is very valuable for debugging for operations. This also allows you to route the issue quickly to the right owner as opposed to playing a game of murder mystery and trying to figure out who owns this application and who deployed it. Some of the most important configurations best practices that I we highly recommend for containers and Kubernetes are as follows: Mount host-sensitive directories as read-only - this means no one can write to your file system. Set up appropriate SELinux profiles to limit containers mounting unix socket. This gives permissions to run containers, create images and so on.. Use capabilities to grant fine-grained privileges - use CAP DROP and CAP ADD to limit what Linux capabilities containers are allowed to run. Do not run ssh services inside a container. Ensure that there are no unnecessary ports exposed ensure that the container’s ports are not mapped to host ports Network policies – use network policies to limit ingress and egress network communication between Pods (east-west) and from outside to Pods (north-south). Secure the host OS - SELinux RESTRICT A CONTAINER FROM ACQUIRING NEW PRIVILEGES So, the last thing to always remember is that security is hard, more akin to a marathon than a sprint. There’s no such thing as perfect security; it’s always about monitoring, iterating, and making sure that the tools are available for people who are building the code to integrate it into their process and drive security.
• #10: Throwing another Buzz word here , the DevSecOps. The idea here is to embed Security early in your development process to prevent any security incidents and enabling continuous security in a cloud native fashion. Another key is the constant feedback loop between the build/deploy phase and the runtime phase. Enable Developers and provide them with the security issues during development and educate them on how they can be fixed. Culture change when it comes to security right, how do we fix a container image that has vulnerability? , we don’t kill the container process and leave it there, we go back and fix them in our source, build and push a new image into production. Cloud native CI/CD processes allow us to do that.
• #11: Stackbox runs in kubernetes, supports declarative style and same Cloud native way of doing things, does remediate using kubernetes constructs ..leverages most of the kubernetes out of the box flexibility to achieve things. protection Eliminate blinspots Threats unique kubernetes vulnerabilities Time to implement security and learning cost Remediates in the kubernetes way Operational security - greater uptime Operational conflict - inconsistent configurations Ability to bring security to your build phase, deploy phase and Run phase It supports all kubernetes Disconnected installs on prem and runs anywhere Highly flexible and Integrates with some of the platforms For example, for package scanning scanning for vulnerabilities, it can work with tenable, Quay or anchore or you can use inbuilt scanning. Supports number of different languages including .NET core Integrate with your CI/CD tools including jenkins, DevOps notifications for Jira, slack pagerduty and others.. SIEM toold it can
• #12: Central Core function: UI and API server; policy engine How it works: Runs as a Kube service in cluster/VM/cloud; must be able to receive inbound connections from all monitored clusters What it does: Facilitates all UI and customer-facing API interactions Performs policy evaluation Facilitates multi-cluster management of all services environments across various providers and data centers Sensor Core function: Point of integration with Kubernetes; processing engine How it works: Runs as a Kube service with permissions to listen to orchestrator events and CRUD rights for enforcement What it does: Inventories cloud-native components (namespaces, running services, network policies, and orchestrator secrets) Performs detection rule processing Facilitates enforcement Collector a Daemonset Core function: syscall-level data collector How it works: Runs as a container on each host and inserts a kernel module for syscall visibility What it does: Facilitates collection of data for runtime detection Scanner (Optional) Core function: Vulnerability scanning of images How it works: Runs as a Kube HPA in the same cluster as Central. What it does: Handles requests to scan image layers from Central Pulls image layers from registry using configured image integrations Retrieves updated CVE data from stackrox.io or from Central Scanner-db Core function: Database for caching image layer data. How it works: Runs as a single Kube pod with PostgreSQL DB What it does: Caches results of layer scanning Customers do not need to provide redundancy / backup AdmissionController Core function: Plugin to Kube AdmissionController to approve / deny API actions How it works: Runs as a Replicaset, registered as ValidatingWebhookConfiguration What it does: Examines Kubernetes API commands that create pods, deployments, daemonsets, etc. Offer stackrox admins the ability to reject these API requests based on policy. Can also examine Update API commands to reject changes for already deployed Can examine & prevent ‘exec’ and ‘port-forward’ API requests Sensor talks to kube apis to get data , what deployments, what configuration Collector - one pod at the node level, what process. What network traffic
• #13: From a cluster level, most of the kubernetes distributions, you can leverage out of the box functionalities to extend it for applying security best practices. For example., From OKE, the upstream open source version of OpenShift few security aspects comes default Secure OS/Infrastructure The core components of the Linux kernel that are used for containers are cgroups — control groups, which define the resources like CPU and memory which are available to a given process — and namespaces, which are a way of separating processes by restricting what each process can see, so that system resources “appear” isolated to the process. Along with cgroups and namespaces, you can also use a Linux Security Module (LSM) like SELinux to configure a container’s capabilities. SELinux. Both deny undesirable default capabilities, like the ability to write to the proc filesystem RHEL CoreOS Cri-o container runtime - provides a smaller footprint and reduced attack surface SELinux Security Built-In Identity & Access Management Compliance Operator - describe the desired compliance state of a cluster and provides them with an overview of gaps and ways to remediate them Volume Encryption File Integrity Operator Security Context Constraints - allow administrators to control permissions for pods Secrets Management All the above from a more cluster level security Stackrox elevates it to the next level or application level and brings in the ability to customize your policies and gives a better visibility and ability respond to those events through notifications. Build Security Vulnerability scanning Vulnerability management CI/CD integration to fail builds Registry integration to block deployments CI/CD Integration for CIS Docker checks/failures Image Content Analysis Deploy Security Declarative Policy Creation Multi-factor risk profiling Compliance benchmarks RBAC analysis for Kubernetes Network policy enforcement Service Configuration Admission Controls Posture Management Kubernetes Events (Execs/Forwards) Runtime Security Intrusion detection & behavioral analysis Network policy visibility/incident detection Deep data collection and correlation for forensics Baselines/whitelists of runtime activity profiles Killing of compromised pods Detection of real-time intrusions, cryptojacking, and orchestrator threats