12/01/2015 | Delivery Leadership | Thought Leadership
Imagine this scenario: Your organization is eight months into a large-scale agile transformation. You have multiple transition teams at work moving your organization along the transformation path as well as pilot teams working to perfect the processes and adapt them to your organization. This is a large, complicated effort and your monthly spend reflects this, being in excess of $200,000. You sit down to review the program, budgets and progress with your CFO, who notices the monthly burn rate of your transformation program and asks you directly if the organization is starting to see any benefits from the effort.
What do you say? How do you know if your agile transformation is producing any benefits?
The truth is that many organizations do not track the success of agile itself. If you are one of these organizations, you should definitely consider the benefits of doing so. Agile transformations are lengthy and costly initiatives, so it behooves an organization to understand the costs involved and to be able to communicate to leadership and key stakeholders what type of return they are realizing for the money and time invested in the effort. Progress can be demonstrated through transformational metrics aligned to organizational objectives, but before we discuss these metrics and how to kickoff such a program, let us review why organizations transform to agile to begin with.
Why Firms Go Agile
Organizations go down an agile path for a variety of reasons, but their objectives are most commonly a combination of one or more of the following:
Speed to Market – Agile organizations operating efficiently can significantly lessen the time it takes them to get new features into production and into the hands of customers. This shortened lag time can help to minimize the cost of delay the organization might normally experience. It also helps to shorten customer feedback loops, allowing the organization to obtain valuable feedback on their product in a timely manner and adjust accordingly.
Code Quality – Teams operating in an agile fashion, and especially teams adhering to more engineering-centric methods such as Extreme Programming, can significantly improve the quality of their code while also reducing time required for quality assurance and testing cycles.
Transparency – On agile teams, there is no place to hide. The reason for this is that agile processes have built-in mechanisms that provide visibility and transparency to the work performed by the team; indeed, transparency is one of the core values of agile. Transparency allows business partners to grasp easily where their features are in the process. It also provides leadership visibility into bottlenecks hindering performance, enabling rapid feedback and the ability to change course rapidly.
Job Satisfaction – Studies have shown that people working in an agile environment experience higher job satisfaction. There are many reasons for this, but it is often a result of the team’s ability to self-organize and make its own decisions. Making a commitment to complete work over a series of sprints imbues team members with a sense of ownership and accomplishment, further improving teamwork as well as individual morale.
Whatever the reason for transforming to agile, the desired results are worth the effort of funding the program. However, it’s not uncommon to overlook tracking the progress of the transformation.
Eight months in – Do you know how it is going?
The question of whether or not you are getting any results from your agile transformation is inevitable, as C-level executives are keen to understand the return on their investment. Unfortunately, many organizations are not able to determine this. There might be numerous reasons as to why, but here are a few common ones:
No baseline metrics – Oftentimes, an organization does not have a solid understanding of items such as its speed to market or overall job satisfaction from before the time that the agile transformation program launched, so there is nothing against which to compare current metrics. Ideally, this is recognized early on and the framework and efforts suggested later in this article are implemented in order to provide a benchmark against which to measure future progress.
No ability to capture metrics – Many organizations do not have a measurement framework in place for the transition team to capture key performance indicators. Information lies in multiple disparate repositories such as Rally, Clarity, physical task and scrum boards, or other systems, and there is no mechanism and process in place to extract this data and transform it into understandable, meaningful information.
Lack of understanding of metrics – Metrics and measures may exist, but they might not align to corporate transformational objectives or be truly indicative of progress. Worse yet, they may exist but might not be fully understood by senior leadership or may be contentious and misleading. I have been in daylong meetings where the bulk of the meeting time was spent arguing about the meaning or calculation of metrics rather than discussing the critical decisions to be made based on the information in question.
No access to data (no reporting framework) – Metrics and reliable information for critical decision-making may be available to the organization, but there may be no reliable or predictable means by which this information is distributed to various stakeholders within the organization. Canned reports from point systems such as Clarity or Rally often do not provide the right information at the right level for the right audience, and thus many organizations spend valuable time tweaking reports, creating custom reports, or not reporting at all.
Obviously, failing to track the success of your agile transformation program across any of these points is less than ideal. Realizing the oversight midway through your program only makes things more difficult, so it’s best to plan how you will measure success at the beginning of your program.
What you can do
There are a few, easy steps that you can take to begin your agile transformation on the right foot and set yourself and your organization up to be able to answer the tough questions from your CFO.
1. Start Small
Identify a small set of metrics to start with that tightly align to your corporate objectives and to your transformational objectives. I recommend including metrics from both the team level, e.g. those that measure team performance, such as velocity and committed story points versus accepted points, as well as transformational metrics that measure how your organization is progressing toward the transition, such as changes in speed to market, decrease in post-production defects, etc. However, note that transformational metrics are the only ones that truly demonstrate progress toward your transformational goals. A small set of five to ten well understood and simple-to-calculate metrics will serve your purpose and allow you to evolve to meet business informational needs.
Here are some recommended metrics to choose from:
| Team-Level Metrics (from SAFe) | Transformational Metrics |
| Velocity – Planned- Velocity – ActualStories – PlannedStories – Actual% Stories Accepted% Unit Test CoverageNumber of DefectsNumber of New Test CasesNumber of New Test Cases AutomatedTotal TestsRelease Frequency | Cycle Time (Idea to Deploy)- Customer SatJob SatisfactionPost Prod Defects% Automation% Test Case CoverageVelocityCycle Time (Build to Deploy)Burn DownCumulative Flow |
2. Educate Leadership and Gain Buy-In
Metrics are no good unless leadership fully understands and supports them. To use metrics from an agile process to assist in decision-making and help guide the direction of your transformation, leadership must understand concepts around lean and agile thinking and the value of chunking things up into small batch sizes to reduce variability and increase predictability. They must also understand specific agile metrics and outputs such as velocity, burn up charts, release burn ups, and other standard agile measures, as well as know how these compare to and relate to traditional metrics such as milestones, earned value, etc. Bolster leadership’s understanding of agile metrics by establishing a data governance framework and capturing and defining metrics in a catalog that includes how they each are calculated, where the source data comes from, how frequently they will be reported, who their target consumer is, and why we care about them.
3. Establish a Cadence
Finally, metrics, even if well understood and supported, are worthless unless they are captured and communicated. Your organization should establish a simple, lightweight process to gather information from your teams and convert this information into metrics. Teams in homogenous environments can use standard reporting tools as long as the canned reports provide the appropriate level of information. Teams in heterogeneous tool environments might need some ETL (extract, transform, load) processes as well as visualization provided by a tool such as Tableau or QlikView. These reports can be generated relatively quickly and can be distributed through a variety of means. In addition, as business needs change, reports developed in this manner can be quickly refactored to supply newly desired information to the business in a timely manner, rather than having to wait on cumbersome report request processes that can take weeks or longer.
In summary, it definitely pays dividends to think ahead about how you are going to report on the progress and success of your agile transformation. Start early and start small, and focus on the transformational metrics that will allow you to tell how well you are doing and how much benefit you are realizing. That way, when the CFO asks you how your agile transformation efforts are progressing, you will have a solid story to tell.
If you are investigating consulting services for Agile development, tell us about your challenge to learn more about the approach we would take with your organization.
