This document will give an overview of the Leading2Lean Development Processes and Procedures. This process describes the responsibilities, polices, and procedures to be followed by Leading2Lean when making changes to the production service and infrastructure. All employees are required to follow this policy.
The Leading2Lean Technology Team is tasked with providing a stable and reliable IT infrastructure for our customers. The purpose of the Development Process is to minimize service disruptions to our data center environment and promote system availability and integrity. The Chief Technical Officer (CTO), is responsible for maintaining and updating the Development Process and Procedures as needed. This process is further governed by organizational controls defined as part of our AICPA SOC 2 Type 2 compliance program. These controls are audited by our external auditor on an annual basis.
The development process starts with a developer reviewing the ticket, coordinating with colleagues regarding architecture and design, writing code, receiving feedback from the QA team, to finally, releasing the software to customers.
This document defines the lifecycle of feature requests and bugs. It also describes the processes and procedures used to ensure proper testing and validation occurs for each release. This document does not describe the Change Management Process as defined in our Change Control Policy Overview.
Backlog: A collection of bugs and features that have not begun development. The Backlog is where all lower priority tickets reside.
Bug: A feature that is not working as it was intended to work. Bugs can range from non-severe to extremely severe. The Bugs list is where all lower priority bugs reside. Tickets are placed on the Bugs list during the Triage meeting. Tickets are assigned to the development team members from the Bugs list.
Feature or Enhancement: Behavior that has not been developed for the software. This can be a modification of existing functionality or the creation of new functionality.
Project or Epic: A collection of Bugs and Features. Projects are managed on the Roadmap.
Roadmap: A collection of Projects. The Roadmap is the planning board for larger projects. The Roadmap drives development of new features, large and small.
Triage: Collection of new tickets. The Triage list is where all new tickets are created. The Triage team will then place tickets into either the Bugs list, Roadmap, or Backlog.
There are 3 main phases to each development ticket: 1. Planning, 2. Development, and 3. Quality Assurance Testing. Each phase will be discussed below.
We utilize a multi-disciplinary triage approach to the planning process. Our Support, Development, and Quality teams meet on a regular basis for a Triage Meeting to assess new issues, feature requests, and enhancements. Based on the severity, these issues may trigger an escalated break/fix response cycle to restore service in the event of high priority items or represent a degradation in the service. Otherwise, they are given an initial priority and passed to our Product Team for additional planning, prioritization, and scheduling.
Features are typically planned with the input of a small number of people. This may include members of the Product Team, subject matter experts, customers, etc. This is driven by our product management team.
The majority of feature requests will be discussed during the Roadmap Planning meeting. During this meeting, projects and escalated feature requests will be discussed. Projects will be prioritized against other projects, and feature requests will be reviewed and moved to the backlog. Feature requests may be linked to a project on the roadmap.
Once the planning process is complete, this work is assigned to Development. The Development Team has its own set of development environments that is a mirror of our production environment. This is an essential part of our quality program. At each step of our development lifecycle (Dev, QA, Staging, & Production) we maintain separate infrastructure to ensure that any work completed will be properly tested and verified before it enters our production environment.
We employe a full set of industry-standard development tools designed for revision control, automated testing, static code analysis, and code cleanliness. These tools provide early and frequent feedback to the developer to ensure the best possible code quality. They also help maintain proper separation of development branches to avoid cross-contamination between developers working on different projects, tasks, and elements of the codebase. These tools provide a standardized method for peer code reviews and approval processes to ensure any code ready for Quality Assurance Testing has passed all automated testing and validation checks.
Our developers communicate directly with the support team, the product team, or other developers when questions arise about how to implement a ticket. All UI changes require approval from the product team. Since UI changes impact customers and may require training, we work with customers, support, and the product team to ensure proper care is made in designing any UI changes to minimize the impact on customers. All UI changes are also reviewed by our support team during the Quality Assurance Testing phase to validate the proper functionality exists and customers are provided the proper notifications and training.
Developers are required to run our automated test suite successfully on any changes as part of their normal operating procedures. Once the ticket passes these automated tests, it is ready to be reviewed by someone other than the developer. This event allows key team members to know it is time for feedback as well as indicating that the ticket is getting closer to release.
All code is peer-reviewed by another developer before being passed to the QA team. This helps ensure that more than one person knows and understands what is being changed and why. The purpose of a code review should be to think about the overarching goal of the changes, review the code for simple errors, scan the code for inefficiencies, ask for clarification when something doesn’t look right, and try to maintain consistency with the coding standards and best practices. At times the original developer will refactor the code due to feedback in the testing phase. If this leads to a significant amount of code change, an additional review is performed. When the code review consensus is recorded and documented, the code is good to be released to the QA team
Quality Assurance Testing
Once the automated test suite has completed successfully without errors, and the code review has completed successfully, the QA team can begin the Quality Assurance Testing process.
The two main activities involved with the testing process are running automated test scripts and manual testing. L2L has a suite of UI and unit test cases that are run from the command line. There is also a linting process that runs against the code to check for basic coding errors. All developers should run both the test suite and the linting process before submitting their code for review. The QA team does its own review of automated test suite results, after deploying the code to into the QA test environment. This ensures we have a clean test environment to assess the code update's ability to function as designed and according to our quality standard.
The QA team will coordinate the manual testing of code to ensure additional quality. Feedback from the QA team will go into the development ticket. Often, code will need refactoring and updating before QA gives a stamp of approval. QA will indicate the code is ready to be merged by approving the pull request. QA will merge the code into the main codebase once everything has been approved. This ensures that developers cannot merge code into our main codebase and bypass the QA process.
During the Quality Assurance Testing phase, the QA team will also utilize support team members and the customer to perform user acceptance testing. These tests are vital to ensure the code changes are usable, fit the customers needs, and are compliant with any industry regulations and requirements.
Change Control Process and Deployment into Production
To ensure any changes to our production environment follows a standard process, we have implemented a Change Management Process as defined in our Change Control Policy Overview. This ensures we have the proper checks and balances in the organization to ensure the quality, security, and availability of the production environment. For more information about this process, see the above link.
It is important to note that as part of the change control process, L2L uses an automated deployment script to release software from the code repository to customer servers. At no point do employees manually update the production environment. This automated system creates an audit trail for each new release/deployment and verifies the proper approvals are in place. Once this is done, the release scripts will be allowed to update production servers. This automated script also provides internal company notification of the deployments. Once the deployment is complete, this automated system also performs "smoke tests" to validate the deployment was successful and that the production environment is operating properly.
In the event released code resurfaces an old problem or causes the system to alert, a retrospective meeting, aka a post-mortem, will occur. The purpose of the meeting will be to gather the facts relating to the release of the bad code, discuss ways to prevent similar events from happening in the future, assign tasks relating to the prevention of future events, and educating the team about the incident. These meetings should happen within 36 hours of the event and should involve as many developers and quality assurance team members as possible for root cause analysis and training purposes.
Customers notifications can occur at multiple times during this process. Customers are often consulted during the planning, development, QA phases as well as once code is released. The customer will be notified in the following methods:
- Personal Contact / Scheduled Meetings. Customers involved in specific development projects may be invited to attend regular update meetings to review plans, designs, and project status.
- Personal email to customers. All originating support tickets will be updated informing the customer that the change has been released to production servers, and the ticket will be closed. If a support ticket was not created, a support representative may contact the customer directly. If only one customer was impacted by a change, other forms of notification may not be used.
- Change log. All changes that impact customers will contain a change log. This includes bug fixes, minor updates, and feature requests. When a change only affected a single customer and the impact was minor, and if a customer was contacted directly, a change log may not be added. Customers can access the change log from the Support menu in the Application.
- Mass email. An email will be sent to all Administrators for all large changes to the app. Mass emails are used mainly for product announcements and large UI enhancements but can be used for additional scenarios.
- In-app notifications may be used for impacting changes.