CASE STUDY – PART 2 – Proven, Practical Tactics For Agile IT Release Management (DEFINITIONS AND TRIAGE)

OVERVIEW:

This article is the second in a series of five which explain how an IT organization delivered a release management process that exceeded its management’s expectations and provided a foundation for continued success. The series includes:

1. How did we get here – THE CONTEXT
2. First solution steps – DEFINITIONS AND TRIAGE
3. Intake and Release Planning – THE CORE SOLUTION
4. Production Change Control – FINAL QUALITY CONTROL
5. Metrics and Insights – LESSONS LEARNED

SUMMARY:

Many Information Technology organizations flounder when they are tasked to understand, organize and implement numerous changes to the system and application software serving their clients and end customers over a period of several years. This second article focuses on the very first steps of the solution I developed for the Release Management consulting engagement. Please refer to the first Article – THE CONTEXT for a full discussion of the problem domain and organization.   What was the secret sauce that made Release Management a success? The answer begins with core problem analysis, proposed solutions and a triage effort.

CORE PROBLEM ANALYSIS:

The company and its IT department clearly wanted a solution implemented that avoided historical mistakes. As a retained consultant, I conducted a series of one-on-one interviews, examined remnants of Change Control documents from a predecessor, investigated commercial off-the-shelf packages for both processes and operational tracking, and discovered the following core problems.

• Problem #1 – What Trees are in this Forest? A substantial source of confusion and discussion involved defining the scope of what things should be controlled under the umbrella of Release Management and Change Control. There were divergent opinions about whether to include/exclude major projects, infrastructure changes from operations, bug fixes, hardware changes, Customer service changes, emergency patches, etc. As with most debates within IT, the stakeholders frequently used similar terms to mean very different things. Very specific language needed to be written, socialized and implemented that reduced this ambiguity and confusion.

• Problem #2 – Who is Responsible for What? Projects were very clearly controlled end-to-end by Project Managers, and at any given time the 4 PMs would have 2-5 projects underway. These generally covered scope for multiple applications and multiple person-months of programming effort. Beyond that good start, Client Change Requests, Bug Fixes, Operations infrastructure changes, Customer service changes (move/update/fix my workstation) and emergency patches had no one role identified for control, communications and accountability throughout their life span. The IT assembly line for such work was disjointed at best and lacked fundamental structure. The role of Release Manager itself was undefined, with various stakeholders holding unique viewpoints on the scope of the assignment.

• Problem #3 – How To Introduce Order Upon Chaos? This was a very critical concern, as the inflow of new requests from the business could not be halted, due to political and practical matters. If the organization knew tactically where it stood on Monday for every item, by Wednesday the landscape had changed, and priorities for older work were being adjusted on-the-fly, either overtly or covertly by business leadership.

• Problem #4 – How Frequently Should We Release Change Packages? A practical concern was whether IT should embark on weekly, biweekly, monthly, quarterly or other frequency of planned production software change. The frequency of change would end up driving the timing of the real-world series of gates and meetings necessary for control and adjustment.   The first solution proposal entailed purchasing a complete software application and documentation package from a market leader that promised to cover the full scope of their interests. The alternative proposal was founded upon a low level of automation, and a high degree of inter-personal collaboration to achieve similar ends.

CONCLUSION / TRANSITION

The CIO, facing this situation, agreed to allow the Manager for Project Management and Analysis to contract for a resource to implement Release Management (Version 2).   The CIO believed that she could deliver better results to her constituency by implementing change in a series of well-understood application package upgrades at regular intervals. She also wanted to take back to her peers a plan that they could understand and use to directly influence the order of implementation for their changes. The Manager of PM retained me as the Release Manager with the mandate to institute the processes and controls needed, and engaging all IT staff and VPs in business departments as needed for success.    The rest, they say, is “agile” history. To learn what it really takes, our story continues next with DEFINITIONS AND TRIAGE.

Definitions

I will remark on Definitions first, because the management team needed to ground itself and communicate in a consistent fashion about the key objects and controls with Release Management processes.

Problem #1 – What Trees are in this Forest? – SOLUTION:

We took the perspective that anything that is planned to change the configuration of the production computing environment within the controlled data center and network configurations was subject to Release Management processes. As a result, we included:

·        Application software changes requested by clients or from IT itself (re-factoring, etc)
·        Application software fixes that were “not immediately damaging” clients’ business
·        Each software change package from projects (projects typically had more than one release package over several months)
·        Network or server infrastructure upgrades (OS, DBMS, middleware, hardware, etc.)

We excluded from Release Management processes:

·        Customer Service requests (fix/move my workstation, office moves, etc)
·        Emergency production software application fixes (fix it now)

This last exclusion was troublesome, but necessary. We assigned total responsibility for managing the emergency fixes to the Software Development Manager, and set an overall target to keep these to fewer than 5% of the total changes made into production. (We tracked the numbers, but didn’t stand in the way).   The practical outcome of these agreements was that each individual thing included in Release Management was a Change Request, to be initiated with a simple form. Each would be assigned a unique Number (and key attributes) and controlled in immediate form by an Excel spreadsheet updated and distributed by the Release Manager.

Problem #2 – Who is Responsible for What? — SOLUTIONS:

• Single Point of Contact (SPOC) The organization had a good model of behavior and accountability for projects, but there was disjointed accountability for all the other Change Request types. To solve this, we defined a role called the Single Point of Contact (SPOC). The role was accountable for conveying the requirements, correct status of IT progress, and sponsoring the Change Request for its ultimate release to production. The SPOC was individually accountable for telling our clients the timing and impact of the implementation of a change, so that our clients were adequately prepped for a release. The assignments to this role were expected to last for the duration of the Change Request, as opposed to the previous pattern of shifting responsibility. As a practical matter, the SPOC assignments for 75% of the Change Requests fell into the Project Manager/Business Analysis team.

• Architecture Review Board (ARB) The IT organization had a defined group called the Architecture Review Board (ARB) which convened to assess the technical and organizational impact and risks of major changes to the environment. This group consisted of the 5 IT Managers, the Applications Architect and the Operations Architect, and occasionally the CIO. As part of our definitional work it was determined that this group would exercise an expanded role – to quickly and routinely review each incoming change request. The Release Manager was added to the Board. This board was the “neck of the funnel” for all new items and through discussion, they determined rough size, priority, and impacts. The ARB also made the specific assignment of a SPOC to each new Change Request. More on the role of the ARB is covered in Article Three- Intake and Release Planning.

• Release Planning Group (RPG) The primary organizational element that needed to be set in place was a new group titled Release Planning. This team, facilitated by the Release Manager, met with great discipline and regularity to organize, re-organize, and commit to a comprehensive, concrete order of implementation for all Change Requests. While this sounds straightforward on paper, remember that the context for this role was to organize +/- 325 Change Requests into a series of planned releases – and do this repeatedly as new things got added each week. This was a puzzle with ever-moving parts. The Release Planning Group consisted of the 5 IT Managers and the Release Manager. The Release Manager published the current Release Schedule as an outcome of each of the group’s meetings.

• Change Control Board (CCB) This group was chaired by the Configuration Management leader, and had the responsibility to review and approve or defer the completed Change Requests for implementation in production. The Operations Manager and QA Manager played strong roles within this forum. The SPOCs for each Change Request were questioned for preparedness items, including the advance notification of the client communities. The CCB made a consensus decision on each Change Request and the outcome of these decisions allowed the Configuration Management Team to prepare the scripts and code packages for production upgrades.

Problem #3 – How To Introduce Order Upon Chaos?-SOLUTION

This fundamental problem afflicts all business organizations. Customer requirements constantly change in nature, new ones are added, and old ones wither yet refuse to die. Progress on the production line in IT is swift, stuttering, under-resourced, or overwhelmed. Managers independently made decisions from their own perspective of the best interests of the company. Frequency of change was sporadic.   To introduce order, the Release Manager defined a disciplined business cycle for Release Management Processes. The business cycle was a repetitive set of scheduled meetings of the Architectural Review Board, Release Planning Group, and Change Control Board that would be executed with defined agendas and deliverables, without failure, and with full participation of the people in their assigned roles. This business cycle would commence as soon as a triage of the Change Request queue of work could be completed. Customer feedback loops were defined for each stage of the process. Triage was a critical first step and is discussed further in this article.   The plan for this business cycle was presented to the CIO. She approved the plan for this business cycle, committed her management team to its principles, and successfully sold the plan to her peers in the company.

Problem #4 – How Frequently Should We Release Change Packages?- SOLUTION

The debate on this was not difficult – once we had made the earlier decisions to include operations change requests and exclude the emergency software fixes. We settled on a 2-week release change cycle. Our internal customers were already seeing changes made (or attempted) weekly with mid-week exceptions and surprises, so this could have been viewed as a step back by IT. However, the IT managers saw many shortcomings with more rapid attempts at change and were far more confident that the company would be well-served on a 2-week cycle. Specifically, change requests would be bundled together into a Release Change Package for implementation on alternate Thursday evenings. Our fallback position, if the Release did not succeed on Thursday night, would be to switch to a Friday evening implementation.

Triage

At this stage, the CIO evaluated how quickly all this good foundation work could be put into operation. As Release Manager, I advocated for a process solution supported by an industrial-strength commercial application that could be leveraged toward portfolio management, with many people updating their component parts, and project-specific support and tracking. However, finding, funding, purchasing and implementing such a baseline tool would require an estimated 3-4 months under ideal conditions. The CIO opted to proceed with the alternative “low-tech” approach for her organization. The mandate was to “find a way” to implement the essential processes by using the lowest-budget approach.   The mandate was daunting. The prior “Change Coordinator” person had worked from a Change Control log in Excel that had fallen into disuse. The root causes for that condition appeared to be that the information could never be kept current, plus it only covered some of the Change Requests. No one had previously been assigned the responsibility to actual know and communicate the status of “small” change requests – of which there were several hundred. The log also attempted to store a lot of interim dates on small changes, and it duplicated interim dates that Project Managers maintained in MSProject on regular projects. The SPOC role had not been defined. As far as we could tell, 285 Change Requests were “open” – meaning submitted by clients and not yet completed. That number fluctuated each week as new ones got added and some got finished, but no one was certain of the status of each item.

It seems appropriate to define the term triage (courtesy of dictionary.com)   -noun

1. the process of sorting victims, as of a battle or disaster, to determine medical priority in order to increase the number of survivors.
2. the determination of priorities for action in an emergency.

The IT managers knew we couldn’t cope much longer with incorrect information about all the victims (Change Requests) that were littering our battlefield. As Release Manager, I asked them to devote the resources necessary to obtain a current, accurate view of the following for each Change Request:

1. Change Request Number
2. Customer Name
3. Change Request Label (very short description)
4. First Requested Date
5. Status (Open, Completed, Being Worked, Cancelled, or Duplicate) (if completed, wanted a completion date)
6. Target Release Date (Not Available was OK)
7. SPOC Name

I was responsible for facilitating the triage process. This roughly consisted of some very long all-manager meetings, publishing many versions of a new Release/Change Request log in Excel, assigning segments of the list to the most knowledgeable workers for update, and numerous interventions. The sorting process consisted of IT managers agreeing on an initial High, Medium or Low priority for an “early” Release Target per Change Request. Customer VPs were also polled on their priority settings for Change Requests. The process was declared “finished” in 3 weeks. We achieved a stable state of Change Request information in the log and were ready to begin Release Management processes and the business cycle for control.   The saga of Agile IT Release Management continues with THE CORE SOLUTION.

(c) By David W. Larsen

Article Source