This document will give an overview of the L2L Change Management policy. This policy describes the responsibilities, policies, and procedures to be followed by Leading2Lean when making changes to the production infrastructure. All employees are required to follow this policy.
The Leading2Lean Technology Team is tasked with providing a stable and reliable IT infrastructure for our customers. The purpose of the Change Management policy is to minimize service disruptions to our data center environment and promote system availability. The Chief Operating Officer (COO), is responsible for maintaining and updating the Change Control Policy as needed.
Definitions
Change: to transform, alter, or modify the operating environment or standard operating procedures; any modification that could have a potential and/or significant impact on the stability and reliability of the infrastructure and impacts conducting normal business operation by our customers and Leading2Lean.
Event: any activity outside of the normal operating procedures that could have a potential and/or significant impact on the stability and reliability of the infrastructure.
Maintenance: any activity executed by the Operations team that follows established operating policies and procedures that has been deemed safe and necessary for normal day-to-day operation of the Leading2Lean environment. Potential for impact on stability or reliability is extremely low because established procedures are in place that have been extensively tested.
Change Request: the official notification of the change/event submitted.
Change Management Administrator: the person in charge of the Change Management process. Currently the COO.
Purpose
The Change Management Process is designed to provide an orderly method in which changes to the production environment are requested and approved prior to the installation or implementation. The purpose is to question the rationale for the change, ensure that all elements are in place, the change plan is adequate, all parties are notified in advance, and the schedule for implementation is coordinated with all other activities within the organization.
Most problems that occur in the production environment are the result of changes. The more successfully changes are controlled the more likely we are to minimize problems that arise as a result of changes.
Scope
Change Management provides a process to apply changes, upgrades, or modifications to the production environment. This covers any and all changes to the hardware, software or applications. This process also includes modifications, additions or changes to the LAN/WAN, Network or Server hardware and software, and any other environmental touch points (generator, UPS, electrical). The process is used for any change that might affect one or all of the environments that the Leading2Lean customers rely on to conduct normal business operations. It also includes any event that may alter the normal operating procedures.
Changes to the production environment arise from many circumstances, such as:
- Periodic maintenance,
- User requests,
- Provisioning activities,
- Hardware and/or software upgrades,
- Acquisition of new hardware and/or software,
- Changes or modifications to the infrastructure,
- Environmental changes,
- Operations schedule changes,
- Changes in hours of availability, and
- Unforeseen events.
The above list is not all-inclusive. For a more comprehensive list see Appendix A. If you are unsure if a change needs to be submitted through the Change Management process contact the Change Management Administrator.
Change Management Process
Members of the Leading2Lean Technology Team are responsible for pro-active planning in managing their areas in the production environment. Change Requests should be submitted as soon as all planning has been completed.
Submission of Change Request
- Create a ticket to track the change. If the change is part of an existing case the original case may be used.
- Set the status of the ticket to 'Ready To Deploy'. In the Comments section give a high level overview of the change.
- Attach any required documents (project plans, test plans, screenshots, instructions, etc…) to the ticket.
- Once all documentation is in place, notify the Change Administrator. This constitutes the Change Request submittal.
- The Change Request must include enough detail understand the relative impact of the change and how it may affect other areas. Change Requests not completed properly will be rejected and returned to the Requester with an explanation for the denial.
- The Change Administrator will reply with approval to proceed or with follow-up questions or actions that need to be completed before moving forward.
- The Change Administrator or his delegate will manage all communication with the customer regarding any change.
Execution of Change
- Before proceeding with the change you must notify the Change Administrator that you are about to begin.
- After completing the change notify the Change Administrator that you are finished and give them a status update.
Change Completion
- The Change Requester must mark the ticket as 'Fixed' and will log all information about the change including outcome into the ticket. If successful the ticket can be closed.
Emergencies
Emergencies exist only as a result of:
- a customer is completely out of service,
- there is a severe degradation of service needing immediate action,
- a system/application/component is inoperable and the failure causes a negative impact,
- a response to a natural disaster, or
- a response to an emergency business need.
All emergencies are handled on an as-required basis with the approval of the COO or in his absence the VP of Customer Support and must follow the guidelines below:
- COO or in his absence the VP of Customer Support must be notified before proceeding.
- The notification(s) shall include at a minimum the following information:
-
- Will the change cause an interruption in service?
- What additional customers will be affected (in the event a change is needed to fix an outage) and who needs to be notified?
- What is the possible work around until the problem is resolved?
- What is the approximate length of the outage?
- Notification of resolution.
- Completion of a ticket to accurately describe the outage.
Emergencies after normal business hours, on the weekend or holidays, will be resolved immediately. A ticket will be generated and staff will notify affected customers, as applicable. A completed ticket must be submitted through the regular reporting process on the first work day immediately following when the change was made.
The Change Management Administrator will review all emergency submissions to ensure the change met the criteria for an “emergency change” and to prevent the process from becoming normal practice to circumvent the Change Management Process. Any questions will be directed to the manager who approved the change.
Responsibilities
Change Management Administrator (CMA)
The CMA will direct the Change Management Process. The CMA responsibilities include the following tasks:
- Analyze and evaluate a Change Request as it relates to the impact on the production infrastructure.
- Approve or deny the schedule for the change in accordance with the Change Management Policy and Procedures, and report any deviations to the COO and/or to the Requester.
- Perform impact/risk analysis to eliminate potential conflicts.
- Coordinate the changes/events.
- Notify parties of the change schedule.
- Provide notifications of any emergency changes.
Change Requester
It is the primary responsibility of the individual submitting a request to evaluate the change prior to submission. The Change Requester’s responsibilities include the following tasks:
- Perform risk benefits/risk analysis.
- Verify that all equipment, software, hardware, and updates are available.
- Research the requirements to achieve a successful change (required patches and stability of upgrade).
- Evaluate the impact to the system/network and to the customers.
- Document and coordinate a fallback plan. This should explain the steps that must be taken to restore access in the event that the change has a negative impact.
- Develop a plan of action to reduce the risk to an acceptable level.
- Develop a plan of action to lessen the affects on the customer if the change should cause an outage.
- Complete any internal checklist that may be required by the CMA.
- Obtain approval from the CMA or designee for requesting the change.
- Submit a complete, concise, and descriptive Change Request. Change Request not complete will be rejected and returned to the Requester with an explanation for denial.
Once the request is approved:
Ensure that the customer is aware of any possible impact. Typically the Customer Service staff will handle all communications with the customer but the Requester should ensure that the proper communication has been delivered.
Coordinate proper on-call support as needed to resolve any problems or answer any questions that may occur during the change, or immediately subsequent to the change. Contact names and numbers should be available to support staff to obtain additional or outside support.
Report unplanned outages or problems immediately to the CMA.
Provide a status update of the change results in the requesting ticket, upon completion of the requested change. The completed ticket must provide an update on the success or failure of the change in detail.
Appendix A – Approved Activities
All operational activities fall into one of three general categories-
Approved Maintenance: Activity has approval to be executed as any time. These activities have been certified as safe but executor should always proceed with caution.
Approved Provisioning: Activity has approval to proceed but activity must be completed by CMA or under his supervision.
Change Management: Activity must go through Change Management to complete.
Below are general guidelines for deciding which bucket an activity fits into and also a list of the most common activities which an explanation of how they should be handled.
General Guidelines
It is impossible to list all operational and provisioning activities therefore general guidelines are given. Common sense is the best guide. If an activity has the potential to impact our service and/or any customer it should be handled very carefully. When in doubt escalate to the Change Management Administrator. All major activities, including upgrades and migrations, must always follow the change control process.
Approved Maintenance Activities List
- Restart of webservers
- Restart of databases
- Restart of application servers
- Routine passive server monitoring / troubleshooting
- Database backups
- Disk space maintenance
Approved Provisioning
- New customer database creation
- New customer server creation
- Non-production DNS changes
- File storage provisioning
- Database data updates & additive schema changes
- Hot swap deployment of minor bug fixes
- Staging of beta application modules not accessible by production users
Change Management
- Database schema changes
- Database restores
- Major application upgrades
- New module production deployments
- Firewall/Security Changes
- Production DNS changes
- Any server maintenance requiring downtime
Appendix B – Urgency of Change Definitions
The change requester with help of the CMA if necessary, prior to acceptance, will determine the priority of the change. The CMA has final determination as to the correct priority of each Change Request.
Emergency
The problem requires immediate attention where either system failure or essential customer functionality is not available and no work around exists. This problem can apply to the system as a whole or in part. This type of Change Request will receive an immediate initial analysis; however, the initial analysis is not mandatory for approval. The corrective action is implemented as soon as the fix is available regardless of change management schedule, however appropriate organizational reviews and approvals must be submitted to change management as soon as possible. Resolving and implementing a fix to a HIGH Priority problem is worked until completed.
High/Urgent
The problem is of an urgent nature and can justify an out-of-cycle change. This priority is used for problems that meet HIGH Priority requirements, except that a work around exists, or performance degradation for which no temporary work-around is available however delay would not cause adverse customer impact beyond that of inconvenience. These changes must still be controlled, tested and approved prior to implementation on a production system. Change Requests that fall into this category may, at the approval of the CMA, be scheduled for any time.
Medium (Default)
Routine Change Requests that are less operationally important and/or the time frame is not critical for implementation. This priority may be used for important software/hardware/network maintenance issues such as version upgrades, utility software, etc. This priority may be used to improve very difficult or awkward implementations for heavily used subsystems on a selective basis. This priority may be used for development activity or new functionality providing that the activity cannot be accomplished with the lower priority. These problems are resolved and implemented in the next scheduled change cycle.
Low
This priority is intended primarily for new functionality and for fixing capabilities that are currently operational but are difficult or awkward to use. It applies also to non-standard implementations, and other assorted irritants.