Data Analytics Startup Matures Approach with Microservices Architecture

Profile

This organization is the leading data solutions company for identifying, understanding and reaching passionate consumers. Over 400+ of the top talent and companies in Music, Sports & Entertainment leverage them to securely manage and monetize their consumer data.

Challenge

As a data-driven organization, this start-up has a massive data pipeline from which it parses and applies intelligence for its customers. With some automation in place, the company’s engineering leadership knew that a greater application of automation framed in DevOps principles would help streamline processes for the firm. In addition, due to the nature of its setup, this company was always paying for the worst-case-scenario as they didn’t have the resources to properly scale up and scale down their instances. Already using AWS, the company sought an expert in cloud-based DevOps who could partner with them in setting up a new architecture, re-establishing their foundation as one for successful future growth.

Solution

Flux7 AWS experts began with a thorough Assessment of this firm’s architecture and technology approach. Automation that freed up developers to focus on coding, not time-consuming manual processes, was a key goal of the project. As a result, Flux7 recommended an AWS-based microservices architecture steeped in DevOps principles.

Scalability: One of the first tasks was to cost-effectively address this firm’s spikes in demand. Specifically, they experienced days where they would see 25,000 users or more 15-20 minutes, with traffic then dying down for the rest of the day. With a need to scale to over one billion requests a month with very spiky traffic patterns, the firm had resorted to paying for the worst-case-scenario due to internal delays in identifying these spikes. Flux7 addressed this concern by recommending that the company break up its application into autonomous services, each of which could automatically scale up and scale back down based on immediate demand for that individual service. This capacity elasticity now allows each service to meet appropriate demand without manual intervention or implications to other services.

Automation: This startup company’s solution had grown quickly to meet rapid customer demand. And while it had some automation in place, they knew that greater automation could give developers and operations much needed time to focus on more strategic work. With that in mind, the Flux7 team applied automation best practices throughout the implementation. Using its tool chain for DevOps automation, Flux7 was able to architect and orchestrate the delivery pipeline, creating an automated workflow for repeatable provisioning, testing, and deployments.

Developer Agility: The new microservices architecture allowed developers to focus more on creating and delivering code, and less on coordination with other team members as they were assigned to and focused on a discrete service, independent of the work of other teammates. This, combined with a smaller, simpler code base allowed developers to easily identify issues in their code, and quickly update the service — both of which made product delivery faster and of a higher quality.

Benefits

One of the biggest benefits of this project was that the Flux7 team broke a cycle of developers spending too much time with manual processes. Setting up automation to replace manual tasks reversed a downward spiral, turning it into a virtuous cycle of more time to spend on coding and experimenting with new approaches. Indeed, developers now have the luxury to fail fast and try again.

Moreover, the microservices architecture coupled with automation and DevOps principles allowed this organization to effectively achieve scalability without manual intervention or consistently planning — and paying for — the worst-case scenario. Now traffic spikes are handled automatically on a per-service basis ensuring cost-efficient availability.

Most importantly, the microservices architecture gave this quickly growing startup a foundation for future growth and success. With complete knowledge transfer, this company’s Development and Operations teams are ready to manage and extend their new platform dynamically meeting customer needs now and in years to come.

Technical Details

AWS Services: ECS, EC2 Container, Service, Elastic Load Balancing, VPC, CloudWatch, CloudFormation, Cloudtrail, S3, AWS Config, HashiCorp Consul, and HashiCorp Vault