AWS Case Study: Kubernetes on CIS Benchmarked Ubuntu AMIs using KOPS

In a recent blog, we shared the AWS case study of a major US airline and how we used the Kubernetes project for managing production-grade Kubernetes (K8) clusters, KOPS, to run its AWS-based K8 clusters. The goal was to host the company’s applications in an AWS-enabled framework, which the team at Flux7 helped implement in the form of its Enterprise DevOps Framework (EDF). As promised, today we will share the second part of their story, illustrating how we used Ubuntu CIS benchmarked images to help proactively safeguard against security threats.


First, for those of you unfamiliar with the Ubuntu CIS benchmark, it is a widely accepted set of security configuration guidelines for the Ubuntu Linux Operating System. (You can download them for free yourself.) CIS publishes both controls and benchmarks, which are globally-recognized best practice standards for securing IT systems against attack. Its guidelines have been proven trustworthy and are continually refined and updated with a consensus-driven process. CIS benchmarks are very much aligned with Flux7’s AWS cloud security by design approach that allows us to effectively balance security and agility for our customers. (Enjoy a copy of our white paper on balancing security with agility here.)

As we mentioned in Part 1 of this story, the Flux7 DevOps team used CIS benchmarked AMIs for the host OS, Ubuntu Linux. By building these configuration guidelines for Linux OS into the solution, it will help the airline proactively safeguard against security threats.

 

The Challenge

The challenge in this is that KOPS assumes the use of a vanilla (non-CIS benchmarked) AMI. As a result, when we change AMI id in KOPS config to a CIS-benchmarked AMI, KOPS fails. Thus, our job was to determine which security rules on the CIS benchmark needed to be modified in order to get it working with KOPS.  

 

The Solution

While the CIS benchmarked AMI comes with iptables enabled by default, this makes it difficult to establish communication between the services required to bootstrap Kubernetes. To solve for this, we adjusted our Packer template in order to open the appropriate ports and make it persistent. Those TCP ports were:

4001:4002	Etcd events
2379:2381	Etcd
443	AWS Elb api
8443/6443	Dashboard/heapster
4194	cAdvisor
10250	Worker node Kubelet API for exec and logs
10255	Worker node Kubelet API for exec and logs (read-only)
179	Calico BGP network
30000:32767	Default port range for external service ports

 

The CIS Benchmarked AMI

With the communication issue solved, the AMIs can now provide the information required to launch an instance — including a template for the root volume for the instance which includes the Linux Ubuntu operating system. The AMI also includes launch permissions that control which AWS accounts can use the AMI to launch instances and mapping that specifies the volumes to attach to the instance when it’s launched.

We based the AMI on the CIS Ubuntu 16.04 image, baked it using Packer and configured it with Ansible. We attached an Amazon CloudWatch Logs Agent to the AMI using Instance profile and with EC2 permissions. The workflow we used to bake the CIS-based AMIs looks like this:


Last, we verified job success by logging into the console and affirming that the latest AMIs are built with the Pipeline, a process designed specifically to automate the delivery of services into the landing zone.

With the newly deployed CIS Ubuntu Linux benchmarks, the airline benefits from the ability to create infrastructure with security best practices built-in from the beginning. Further, by automating best practice security processes through hardened AMIs, the airline is able to boost its security benefits and provide a greater level of control over its architecture and new elements being created. As illustrated by this carrier, AWS cloud security can be automated and built in to allow developers and IT operations to simultaneously increase their security and business agility.

For additional AWS case studies and how to effectively balance security with agility:

• Voyant Grows Data Security and Customer Satisfaction with Advanced AWS services
• TN Marketing Scales Performance, Elasticity, and Security through AWS CloudFront, ELB and WAF expertise
• RentACenter Builds Innovation, Availability, and Security-By-Design