CASE STUDY – Part 3 – Proven, Practical Tactics For Agile IT Release Management (THE CORE SOLUTION)

OVERVIEW:

This article is the third in a series of five which explain how an IT organization delivered a release management process that exceeded its management’s expectations and provided a foundation for continued success. The series includes:

  1. How did we get here – THE CONTEXT
  2. First solution steps – DEFINITIONS AND TRIAGE
  3. Intake and Release Planning – THE CORE SOLUTION
  4. Production Change Control – FINAL QUALITY CONTROL
  5. Metrics and Insights – LESSONS LEARNED

SUMMARY:

Many Information Technology organizations flounder when they are tasked to understand, organize and implement change to the system and application software serving their clients and end customers over a period of several years. This third article focuses on the key processes of the solution I developed for the Release Management consulting engagement.

Please refer to the first Article – THE CONTEXT for a full discussion of the problem domain and organization, and to the second article – DEFINITIONS AND TRIAGE for a discussion of the get-ready steps.   What was the secret sauce that made Release Management processes a success? The sauce begins with three foundation ingredients or elements – a paradigm shift, visualizing the whole problem domain, and knowledgeable collaboration.

FOUNDATION ELEMENTS:

The Core Solution was enabled by three dramatic changes in the mindset and actions of IT management.

  • Paradigm Shift – From Resource Balancing to IT Hydraulics and Kanban

The CIO and others initially assumed that the path to improved productivity would be an outcome of matching forecasted demand for service and forecasted supply (essentially of programming and QA staff), enabled by capturing a lot of size and capacity metrics, personal vacations, hours spent on work in progress, etc. In simple terms, each change request would be estimated (repeatedly) for hours of remaining effort from each source group or individual, run through a number cruncher that was time-sensitive, and out would come a resource-balanced schedule of releases with their associated change requests. Let’s call this the MSProject model for short. In theory this works for standard parts and production lines, but the model fails miserably in the face of human nature, software craft, and changing business priorities and demands.

The first shift was to recognize that the IT environment behaved according to the principles of hydraulics. Imagine for a moment a large funnel fed from above by fire hoses gushing change requests.  At the bottom of the funnel is a gate/valve operated to flow CRs into a pipeline with a fairly constant flow and capacity.  Along the pipeline were release valves allowing CRs to enter production.  This is the mental model of IT Hydraulics.

Our funnel (holding tank) for Projects and Change Requests represented unscheduled customer demand that was fed via “fire hoses” from the department heads. There was no limit on how many CRs could be pushed through a fire hose in a time period. The holding tank consisted of the 250-350 change request “forms” for the year of this case study.  The IT work capacity was a pretty steady state of 60 people working full time each week. Of these, about 40 were software developers and QA staff. These resources could work on any of the change requests from the holding tank, and it was management’s job to direct them to work productively and collectively on the most important ones. Some change requests could be fulfilled with mere hours of effort. Many took person-weeks and person-months across IT organizational lines. Also, software developers operated with a limiting principle that they would not have more than 2 open “code branches” for an application. During this case study, IT software development capacity (pipeline) was held constant by a fixed budget, fixed headcount, and existing toolware.   This hydraulics system had relief valves – called releases that were scheduled for every 2 weeks. The system would thus “flush out” a variable number of change requests into production, preventing an overflow of the holding tank. Adding additional pressure from the holding tank above could result in some short-term flow improvements, but excess pressure (or push) invariably resulted in pipeline cracks and disruptive failures. Another primary limiting consideration is that only certain people could work productively on certain CRs due to application or toolware expertise. A prime example was that only 2 programmers knew how to implement change using Oracle Forms software, and had the appropriate licenses and training. There were a number of similar cases defining who worked on what. IT was teetering on the boundaries of failure because an important determinant of the success of production scheduling based on “pushing” the demand is the quality of the demand forecast which can receive such “push”.    So IT management reconsidered its position regarding a “push” form of scheduling. In IT’s environment we did not have standard production line equivalent inputs and outputs. As a consultant, I introduced a pull-based approach, based on my exposure to Kanban theories.

To paraphrase an encyclopedia  Kanban, by contrast(to push), is part of an approach of receiving the pull from the demand. Therefore the supply, or production is determined according to the actual priority demands of the customers. In contexts where supply time is lengthy and demand is difficult to forecast, the best one can do is to respond quickly to observed demand. The theory sounded better than pushing stuff toward the programmers, but how do you pull it off?

  • Visualizing the Whole Problem Domain

The second ingredient for the secret sauce was absolutely crucial. We found that during the triage effort (working from Excel spreadsheets of 300 items and Change Request form documents) that it was literally impossible to use the many lists and pieces of paper, on projectors and laptops, and simultaneously make priority decisions in context of the whole set. It was a condition of too much information, too many people looking different places at once, etc. I was failing to facilitate the priority-setting and release-bundling effort. Then we got really agile in technique. I decided to transform our favorite conference room into a “Visual Decision Space” using:

1.      One colored 4X6 index card for each of the 300+ change requests – all info on a card was readable from 15 feet away

2.      One 8’High X 15’Wide fabric-covered wall that allowed me to use stickpins to tack up the cards, and easily move them from spot to spot

3.      Six Vertical columns on the wall each representing a 2-week release period

4.      One overflow portion of the wall for CRs not assigned to a release period.

The puzzle of what should be positioned in each release could now be readily represented and dynamically manipulated by moving the cards from column to column or put in the overflow (funnel). Everyone in the room could simultaneously see and evaluate each move in context of the whole wall of CRs. Each manager could consider in real time how a move would affect the resources and throughput of their group in the context of 6 release cycles. All the lightbulbs flashed on at once.   This concept of a planning/production storyboard was certainly not new in the literature of production management, but the adaptation developed within our IT department in 1999 certainly preceded many of the authored works on enterprise requirements management and SCRUM techniques of the early Agile Alliance advocates. The Agile Manifesto itself can be dated around February 2001.

With this second ingredient in place, and a very large DO NOT TOUCH sign on the wall of cards, the secret sauce was nearly complete.

  • Knowledgeable Collaboration – Making It Happen

The third ingredient of the sauce was the IT management staff’s willingness and ability to collaborate on making decisions in an informed and friendly way. The wall of index cards provided the puzzle space to let managers propose how they could arrange the priority work in packages that made sense given their resource constraints and track records. No longer were we trying to sum up work estimates and do math problems and forecasts. Instead we could pick out 3 yellow cards from the overflow and say they should go together in the third release because they “hit the same code” or collectively made business sense for the client. Instead of agonizing over potentially crushing any one bottleneck, when a manager felt their unit had reached their max effort for a release, all managers would quickly adjust and move on. Sure, a lot of guesswork was involved, but we also knew if we were over or under committed in a release, our staff would read the wall and point out some possible adjustments that we could make in the next planning session. The process of rearranging THE WALL took place every week with all 5 IT managers, and took one hour or less, even when 10-20 new CRs arrived from the customer departments that week.    By blending knowledgeable collaboration with a paradigm shift to pull-based planning and a visual space for the whole problem domain, the secret sauce was ready to apply!

The Intake and Release Planning Cycle – IT

People abhor unproductive business meetings. People love solving problems collectively. Everyone likes efficiency.   Simply put, our intake and release planning cycle was a weekly cycle that went like this:

  1. Every day the Release Manager would receive zero to many Change Requests on emailed forms.
  2. That same day, the Release Manager assigned CR numbers, updated an Excel-based Change Request Log, acknowledged the author, and routed the CR form to the members of the Architectural Review Board.
  3. Throughout the week it was the job of the Applications Architect and the Operations Architect to do a first cut investigation and resource estimate of these new CRs for use in an ARB meeting.
  4. On Tuesday mornings at 9AM the Architecture Review Board met to consider the CRs submitted from the previous 5 business days, hear from the Architects, discuss the client’s priorities, assess risks, and assign a SPOC for each CR. The meeting ended faithfully on or before 10AM.
  5. Tuesday between 10 and 10:30, the Release Manager made up new CR index cards for the wall, choosing the right color (colors were unique to each application area, plus some special colors for projects and IT internal stuff).
  6. Tuesdays at 10:30AM, the members of the Release Planning Group convened in the conference room holding THE WALL of CR index cards. The first order of business was for the Release Manager to ask for an immediate consensus on where each new CR should be placed. The default position was in the holding tank, but some could quickly be slotted for a pending release package. As a second order of business, the RPG would then begin adjusting the content of the six release columns on the wall, based on known problems or opportunities. Members would propose moving cards back and forth, discuss what they thought, make a decision and move on.
  7. By 11:20 the Release Manager would call a halt to the discussions and do a “session wrap”. It was common to see 15 to 30 cards change position in the course of a single meeting. No one kept any minutes, but the new WALL clearly showed the Release Plan for 6 more Releases.
  8. By 1:00PM on Tuesday, the Release Manager made all corresponding Release date changes in the Excel file (Change Request Log/Release Plan) and routed it to the entire IT department.
  9. By 5:00PM on Tuesday, the individual new CR authors were emailed on the status of their submitted requests. For “older” CRs, the assigned SPOCs were responsible for communicating any Release date adjustments.

Intake and Release Planning Cycle – CLIENT

The commitment to keep the customers apprised of the status or disposition of the requests took several forms. As noted above, on every Tuesday the Release Plan was distributed as an Excel document to the VPs of the organization. All the new CRs submitted that week had individual feedback emails. The second part of the client cycle was a monthly review meeting scheduled with each VP, with the Project Management Manager and Release Manager attending. This meeting was facilitated with Change Request Log (Excel) reports which exposed solely the departments’ own Change Requests. The goal of this meeting was to clearly identify that department’s current “Top 5” Change Requests. Month to month these items changed with completion of work and shifting business interests. In turn, the Release Manager updated the Change Request Log and pasted a “Top 5 dot” on those CR index cards on THE WALL. This proved to be a great way to let all IT managers visualize the top priority work during Release Planning sessions. These monthly client meetings were frequently held in the conference room where the VPs could easily see the scope of the IT workload and where their own CRs were queued up.

CONCLUSION / TRANSITION

The Intake and Release Planning cycle operated with friendly precision. People did not miss meetings. Everyone actively participated within their role because it was efficient. The Release Manager performed all record keeping, and was acknowledged for the accuracy and timeliness of the Change Request Log as a “routable” version of THE WALL. But whenever anyone in the organization wanted a comprehensive appreciation of the whole business, they would sneak into the conference room and read THE WALL. The CIO would occasionally bring in the President to demonstrate where everything stood. This was truly an information radiator (thank you Alistair Cockburn).   The rest, they say, is “agile” history. To learn what happens next for Releases, our story continues with PRODUCTION CHANGE CONTROL.

(c) By David W. Larsen

Article Source

Share

.......................