Upgrade Gives Developer Weekly Time Savings

Upgrades are a way of life. From new features and bug repair to security patches, updating your software helps ensure system compatibility and team productivity. Yet, upgrades can be easier said than done, which is why our client asked for help upgrading its commercial managed security service offering to a custom baked image with Ubuntu 18.04 CIS. 

As a security solution, it’s imperative that the firm ensure the utmost security of its offering while bringing to market a solid, reliable and flexible solution. To do so, the firm wanted to upgrade its platform to a custom baked image with Ubuntu 18.04 CIS. As we assessed the project, we determined that the work would also include upgrading its Puppet modules as well as handling the company’s software artifacts and converting its build pipelines to code.  

Upgrading Puppet and Ubuntu 

We began the project by upgrading the company’s Puppet code to the newest version, simultaneously implementing Puppet best practices. As part of that process, we refactored the managed service node profiles and adopted a Puppet Development Kit (PDK) format. We documented the solution using Puppet-Strings such that developed code met the standards set for the client’s product line. 

Specifically, we ensured that:  

• The Puppet profile and role exist for the client’s solution to efficiently manage product code. 
  
  
• Its build plans effectively manage the artifacts created for Puppet modules and stores them in Artifactory for efficient artifact management. This also ensures the client can meet its compliance needs with tools monitoring artifact repositories. 
  
  
• The image creation process produces secure images by implementing CIS benchmarks controls that are validated before image creation.  
  
  
• The OS is upgraded to a custom baked image with Ubuntu 18.04 CIS.  
  
  
• Acceptance testing for the Puppet components meet established best practices.  
  
  
• Last, we provided the company with release support and helped to update builds to the client’s build specifications. 
  
  

Improved testing 

In addition to upgrading Ubuntu and Puppet, the NTT DATA team also helped improve the company’s user acceptance testing (UAT) tests to make its tests modular, thereby reducing the client’s test cycle times. The team also improved the test plans at the module level as well as the client’s end to end tests.   

In addition to reducing test cycle times, testing improvements had a major impact on the build time of the client’s Amazon Machine Image (AMI).  Improved tests together with staging that now runs tests in parallel saves the firm approximately 30% of the time previously spent in builds. Ultimately, this saves the company up to three hours of troubleshooting and fixes in get to green each week.  

 

The underlying OS image that is used is custom CIS-hardened which is built by a pipeline and tested by producing a report. 
 

In concert with AWS services 

This customer relies on AWS services and this project was no exception. Amazon EC2 was used for AMI creation, predominantly through HashiCorp Packer, and to facilitate end-to-end testing. Amazon EC2 spun up instances, ran tests and then terminated them. In addition, AWS Systems Manager Parameter Store stored sensitive information in SecureStrings and application specific values as another measure to help ensure security. AWS Systems Manager Session Manager connected instances. Last, Amazon CloudWatch log groups were created to retain the log output of the entire testing procedure. 

The pipeline 

The NTT DATA team created new continuous integration deployment pipelines as code for all builds, the Puppet module and role clean up. They did so by removing redundant modules, upgrading code and testing the pipelines on the new Ubuntu 18.04 CIS custom baked images. With this new level of automation in place, the client has achieved several new benefits, including: 

• Get to green improvements. Whereas the client’s previous pipeline returned green once for every 20 runs, the new IaC pipeline achieves green one out of every four runs, a 5x improvement. 
  
  
• Faster feedback to developer  
  
  
• A 40% time savings. Previously, deployments took between 120 and 130 minutes to execute. Now, deployments run in just 70 to 80 minutes, saving the team approximately one hour each time the pipeline runs.  
  
  
• Credentials are now protected during testing processes. 
  
  
• End to end (E2E) issues are resolved.  
  
  
• Enhanced robot tests that feature improved recording and rerunning failures capabilities. 
  
  
• Greater consistency, reduced complexity and increased durability. 
  
  

NTT DATA helped this client upgrade to a custom baked image with Ubuntu 18.04 CIS – and so much more. With updated Puppet modules, improvements in its testing regime and enhanced build processes, the firm can now securely release new builds into production faster. Moreover, by building secure product images on top of CIS hardened images, the client can now build processes for commercial and gov cloud releases more securely – soundly delivering on the brand promise of its security service.