Case Study: Cloud & Operations
SITUATION & BUSINESS CHALLENGE
The eServices division of a company with 200+ contract workers had learned to develop and deliver technologies more efficiently with Agile development but was still experiencing significant waste as well as lowered morale.
Although the group had been using Agile/Scrum for years, automation was limited and roles and responsibilities were somewhat confused, resulting in duplicate effort and other efficiency challenges. In addition, a constant change in priorities was producing chaos across the group’s 12 Scrum teams. The team’s culture had become all about delivery deadlines with little focus on individual or team growth. In fact, teams weren’t communicating when a project wasn’t going well because under previous leadership the one team that had been transparent in its reporting was disbanded for failure. Frustration pervaded the organization.
A new corporate vice president overseeing the division became aware of the issues and mandated that the group transition to a DevOps model. DevOps is a significant transformation that melds development, operations, and support by extending Agile principles and test automation through the deployment process, thus increasing software quality and reliability along with waste reduction and efficiency gains. DevOps also empowers teams by promoting a culture of cooperation, accountability, continuous improvement, and growth.
However, transitioning such a large group to DevOps posed a considerable challenge. The dev manager tasked with the DevOps roadmap realized the need for outside expertise, a knowledgeable resource that would not be affected by internal politics and could leverage industry experience to produce a roadmap customized precisely for the division. After vetting numerous agencies, eServices management chose AIM Consulting to drive the project.
SOLUTION
AIM deployed a consultant from its Delivery Leadership practice to perform an assessment of the organization’s readiness for DevOps and craft a detailed roadmap for the eServices division to reach full DevOps maturity with predictable stability in all environments. The project entailed several steps:
Vision and Measurable Goals
The first critical step was to outline the project vision and goals. Interviews with stakeholders revealed that the organization did not have an agreed upon definition for DevOps or a plan for how the transformation would take place or impact teams.
Four goals were established for the DevOps transformation project:
- Establish a new Continuous Integration / Continuous Delivery (CI/CD) pipeline
- Streamline roles and responsibilities
- Make quality the focus throughout the development cycle, starting with improving the format for user stories with rigorous Definitions of Done, as well as employing better automation
- Build team morale
Assessment and Report
Deep research was conducted by observing current development and operations processes and interviewing 20+ personnel at every level in eServices. The AIM consultant identified several feedback themes, which enabled teams and leaders to learn about team perception and the inefficiencies of the organization and how to improve from an objective, outside point of view.
The themes included:
- Excessive status reporting – Multiple roles were reporting status on the same projects or efforts. This resulted in managers spending more time on redundant statuses, leaving very little time for strategy and other tasks.
- Immature Scrum Masters – Every two weeks, an Agile Coach would spend up to eight hours providing assessments for each of the 12 Scrum teams. This highlighted the need to bring in trained Scrum Masters to help remove the need for regular assessments, which should not be required after more than two years of Agile coaching and executing Scrum as an organization.
- Environment stability issues – Developers and testers spent a significant amount of time being blocked by, or troubleshooting, test environment issues often outside of eService’s control.
- Anxiety – People often felt randomized due to changing priorities and external dependencies. Inconsistent prioritization and planning of features, unclear areas of responsibility across the Product Development organization, and a lack of communication with external dependency groups consistently interrupted the flow of feature development and testing
Solutions to address known problems and areas of waste were included in the final roadmap.
Phased Roadmap for DevOps
AIM Consultant crafted a five-phase roadmap to fully transition eServices to a healthy DevOps model, including structure, culture, and technological solutions to embrace collaboration, accountability, visibility, and efficiency.
The phases, including variable timeframes based on the rate of success for arriving at maturity at each level, were laid out as follows:
- Phase One, split into two parts, implements Lean fundamentals and the initial set-up of the CI/CD pipeline.
- In Phase Two, one of the 12 scrum teams is chosen to pilot the structure and processes to ensure responsibilities are clearly defined for efficient and reliable test automation.
- In Phase Three, learnings from the previous phase will expand to all 12 scrum teams. Unit test code coverage requirements will gradually ramp up for all code checked in until the desired minimum bar is set.
- In Phase Four, pre-production resiliency will be established by making the testing environment stable for monitoring, ensuring the ability to fail smoothly without affecting the customer, and introducing technologies to help load balance the traffic in the event of a service failure.
- In Phase 5, concrete production resiliency will be established by taking the lessons learned in Phase 4 and applying them to production.
Obtain Executive Buy-In
With a solid plan in place, the AIM consultant created a presentation summarizing the roadmap to obtain buy-in from leadership to execute, with the assurance that the range of each phase was flexible to accommodate the maturity of the teams and the speed at which the organization could adopt change.
Create User Stories to Address Objectives
Even with executive buy-in on goals, targets and milestones, DevOps transformations can stall from a loss of momentum as organizations figure out how to get started. To prevent stagnation and jumpstart realization of actual value, the AIM consultant delivered actual user stories in the areas where the teams needed to improve to serve as actionable items as well as examples.
These user stories encompassed measurable goals for eServices such as:
- “When checking in code, ensure that it gets in production in an hour or less”
- “When customers have great ideas, go from concept to solution within three weeks”
- “Have less than five critical customer reported incidents per month”
- “Achieve an 85% approval rating within an internal employee satisfaction survey.”
AIM created a backlog of the detailed user stories for each objective tied to all five phases of the DevOps roadmap. This additional effort was a huge differentiation from other consulting companies. The final roadmap, complete with goals and user stories, came with a recommended timeline for completion of 10–20 months, depending on the rate of progress of individual Scrum teams.
RESULTS
The eServices group began acting upon the roadmap’s first phase immediately, even before executives provided formal sign-off. Lean education seminars commenced and progress was made on redefining roles. One dev manager was so enthused upon viewing the roadmap that he persuaded leadership to agree to a leadership backlog to track this effort as well as all initiatives the managers were working on.
Ultimately, AIM delivered tangible, immediate value through a DevOps strategy and roadmap tailored to the exact needs of the eServices group, complete with a vision, measurable goals, and recommendations that were immediately actionable. Successful execution of the fifth phase excited leadership, as it meant a transition from DevOps to NoOps, where optimization is built into testing and automation to such an extent that operations is “hands off” in the deployment process. The realized transformation would result in a better product, happier teams, and significant efficiency gains, both in the software development process itself and in the ability to redirect the attention of personnel to other areas for improvement.
