Chapter 21: Change management

Why Change Management:

To manage the system development and maintenance processes effectively, you need discipline and structure to help conserve resources and enhance effectiveness. Change management, like risk management, is often considered expensive, nonproductive, unnecessary, and confusing - an impediment to progress. However, like risk management, change management can be scaled to control and manage the development and maintenance of systems effectively. Change management should be used in all phases of a system's life: development, testing, quality assurance, and production. Short development cycles have not changed the need for an appropriate amount of management control over software development, maintenance, and operation. In fact, short turnaround times make change management more necessary, because once a system goes active in today's service based environments, it often cannot be taken offline to correct errors it must stay up and online, or else business will be lost and brand recognition damaged. The following scenarios exemplify the need for appropriate change management policy and for procedures over software, hardware, and data: -The developers can't find the latest version of the production source code. Change management practices support versioning of software changes. -A bug corrected a few months ago mysteriously reappears. Proper change management ensures developers always use the most recently changed source code. -Fielded software was working fine yesterday but does not work properly today. Good change management controls have access to previously modified modules so that previously corrected errors aren't reintroduced into the system. -Development team members overwrote each other's changes. Today's change management tools support collaborative development. -A programmer spent several hours changing the wrong version of the software. Change management tools support viable management of previous software versions. -New tax rates stored in a table have been overwritten with last year's tax rates. Change controls prevents inadvertent overwriting of critical reference data -Network administrator inadvertently brought down a server because he incorrectly punched down the wrong wires. Just like a blueprint shows key electrical paths, data center connection paths can be version controlled -A newly installed server is hacked soon after installation because it was improperly configured. Network and system administrators use change management to ensure configurations consistently meet security standards.

Configuration Control

is the process of controlling changes to items that have been baselined. It ensures that only approved changes to a baseline are allowed to be implemented.

Staging

The staging environment is optional, but it is commonly found when there are multiple production environments. After passing testing the system moves to staging, from where it can be deployed to different production systems. The primary purpose of staging is as a sandbox after testing so the test system can test the next set, while the current set is deployed across the enterprise. One method of deployment is the staged deployment, where software is deployed to part of the enterprise and then the process is paused to watch for unforeseen problems. If none occur, the deployment process continues, stage by stage, until all of the production systems are changed. By moving software in this manner, you never lose the old production system until the end of the move, giving you time to judge and catch any unforeseen problems. This also prevents the total loss of production due to a failed update.

Test

The test environment is one that closely mimics the production environment, with the same versions of software, down to patch levels, and the same permissions, file structures and so on. The purpose of the test environment is to enable a system to be fully tested prior to being deployed into production.

The purpose of a Change Control Board:

To oversee the change management process, most organizations establish a Change Control Board (CCB). In practice, a CCB not only facilitates adequate management oversight, but also facilitates better coordination between projects. The CCB convenes on a regular basis, usually weekly or monthly, and can meet up on an emergency or as needed basis. Figure 21.2 shows the process for implementing and properly controlling hardware and software during changes. The CCB uses standard documents, such as Change Requests, in concert with business schedules and other elements of operational data, with a focus on system stability. The CCB also ensures that all elements of the change policy have been complied with before approving changes to production systems. The CCB's membership should consist of development project managers, network administrators, system administrators, test/QA managers, an information security manager, operation center manager, and a help desk manager. Others could be added as necessary.

Secure Baseline:

To secure the software on a system effectively and consistently, you must take a structured and logical approach. This starts with an examination of the system's intended functions and capabilities to determine what processes and applications will be housed on the system. As a best practice, anything that is not required for operations should be removed or disabled on the system; then, all the appropriate patches, hotfixes, and settings should be applied to protect and secure it. This becomes the system's secure baseline. After administrators have finished patching, securing, and preparing a system, they often create an initial baseline configuration. This represents a secure state for the system or network device and a reference point for the software and its configuration. This information establishes a reference that can be used to help keep the system secure by establishing a known safe configuration. If this initial baseline can be replicated, it can also be used as a template when systems and network devices are deployed.

Identifying Separation of Duties

Using figure 21.1 observe the separation of duties between development, test/QA, and production. The functions of creating, installing, and administrating are assigned to different individuals. Note also appropriate management review and approval. This implementation also ensures that no compiler is necessary on the production system. Indeed, compilers should not be allowed to exist on the production system.

Environment:

Within a modern environment, there are multiple, separate environments designed to isolate functions of development, test, and production. These are primarily to prevent accidents arising from untested code ending up in production. These environments are segregated by access control list as well as hardware, preventing users from accessing multiple different levels of environments, special accounts that can access both are used, thus eliminating issues of crosstalk.

Production

is the environment where the systems work with real data, doin the business that the system is supposed to perform. This is an environment where there are by design virtually no changes, except as approved and tested via the system's change management process.

Configuration Auditing

is the process of verifying that the configuration items are built and maintained according to the requirements, standards, or contractual agreements. It is similar to how audits in the financial world are used to ensure that generally accepted accounting principles and practices are adhered to and financial statements properly reflect the financial status of the systems in operation. Configuration Auditing takes 2 forms: 1. Functional Configuration audit: verifies that the configuration items performs as defined by the documentation of the system requirements 2. Physical Configuration Audit: confirms that all configuration items to be included in a release, install, change, or upgrade are actually included, and that no additional items are included - no more, no less.

System Problem Report (SPR)

is used to track changes through the CCB. The SPR documents changes or corrections to a system. It reflects who requested the change and why, what analysis must be done and by whom, and how the change was corrected or implemented.

Change Control Board (CCB)

A formal group of people responsible for approving or rejecting changes on a project

The Key Concept:

A foundation for change management is the recognition that involving more than one individual in a process can reduce risk. Good business control practices require that duties be assigned to individuals in such a way that no individual can control all phases of a process or the processing and recording of a transaction. This is called Separation of Duties ( aka segregation of duties). It is an important means by which errors and fraudulent or malicious acts can be discouraged and prevented. Information Technology (IT) organizations should design, implement, monitor, and enforce appropriate separation of duties for the enterprise's information systems and processes. Some of the best practices for ensuring proper separation of duties in an IT organization are as follows: -Separation of duties between development, testing, QA, and production should be documented in written procedures and implemented by software or manual processes. -The activities of program developers and program testers should be conducted on "test" data only. They should be restricted from accessing "live" production data. This will assist in ensuring an independent and objective testing environment without jeopardizing the confidentiality and integrity of production data. -End users or computer operations personnel should not have direct access to program source code. This control helps lessen the opportunity of exploiting software weaknesses or introducing malicious code (or code that has not been tested) into the production environment either intentionally or unintentionally. -Functions of creating, installing, and administering software programs should be assigned to different individuals. For example, since developers create and enhance programs, they should not be able to install these programs on the production system. Likewise, database administrators should not be program developers on database systems they administer. -All accesses and privileges to systems, software, or data should be granted based on the principle of least privilege, which gives users no more privileges than are necessary to perform their jobs. Access privileges should be reviewed regularly to ensure that individuals who no longer require access have had their access removed. -Formal change management policy and procedures should be enforced throughout the enterprise. Any changes in hardware and software components (including emergency changes) that are implemented after the system has been placed into production must go through the approved formal change management mechanism. Managers at all levels should review existing and planned processes and systems to ensure proper separation of duties. Smaller business entities may not have the resources to implement all of the preceding practices fully, but other control mechanisms, including hiring qualified personnel, bonding contractors, and using training, monitoring, and evaluation practices, can reduce any organization's exposure to risk.

Change Management

Refers to a standard methodology for performing and recording changes during software development and system operation. The methodology defines steps that ensure system changes are required by the organization and are properly authorized, documented, tested, and approved by management. Configuration Management is considered synonymous with Change Management

Sandboxing

Sandboxing refers to the quarantine or isolation of a system from its surroundings. It has become standard practice for some programs with an increased risk surface to operate within a sandbox, limiting the interaction with the CPU and other processes such as memory. This works as a means of quarantine, preventing problems from getting out of the sandbox and onto the OS and other applications on a system. Virtualization can be used as a form of sandboxing with respect to an entire system. You can build a VM, test something inside the VM, and, based on the results, make decision with regard to stability or whatever concern was present.

Baseline

Serves as a foundation for comparison or measurement

Steps to implement Separation of Duties:

Step 1. Identify an indispensable function that is potentially subject to abuse. Step 2. Divide the function into separate steps, each containing a small part of the power that enables the function to be abused. Step 3. Assign each step to a different person or organization.

Types of changes:

The ITIL v3 Glossary of terms, Definitions and Acronyms (https://www.axelos.com/glossaries-of-terms) defines the following types of changes (with examples added in parenthesis): -Change "the addition, modification or removal of anything that could have an effect on IT services". (for example, modification to a module to implement a new capability). -Standard Change "a preapproved change that is low risk, relatively common and follows procedure or work instruction" (for example, each month finance must make a small rounding adjustment to reconcile the general ledger to account for foreign currency calculations). -Emergency Change "a change that must be introduced as soon as possible" (for example, to resolve a major incident or implement a security patch. The change management process will normally have a specific procedure for handling emergency changes) see https://www.axelos.com/best-practice-solutions/itil.aspx for more information.

Release Management

The ITIL v3 glossary defines release management as "the process responsible for planning, scheduling and controlling the movement of releases to test and live environments. The primary objective of release management is to ensure that the integrity of the live environment is protected and that the correct components are released".

Development

The development system is one that is sized, configured, and set up for developers to develop applications and systems. The development platform does need to be of the same type of system. After code is successfully developed, it is moved to a test system.

The Capability Maturity Model Integrity

An important set of process models are the Capability Maturity Model Integration (CMMI) series developed at Carnegie Mellon University's Software Engineering Institute (SEI). SEI has created 3 capability maturity model integrations that replace the older Capability Maturity Model (CMM): 1. The Capability Maturity Model Integrations For Acquisition (CMMI-ACQ) 2. The Capability Maturity Model Integrations For Development (CMMI-DEV) 3. The Capability Maturity Model Integrations For Services (CMMI-SVC) CMMI-DEV is representative of the 3 models. One of the fundamental concepts of CMM-DEV is configuration or change management, which provides organizations with the ability to improve their software and other processes by providing an evolutionary path from ad hoc processes to disciplined management processes. The CMMI-DEV defines 5 maturity levels: 1. Level 1: Initial At maturity level 1, processes are generally ad hoc and chaotic. The organization does not provide a stable environment to support processes. 2. Level 2: Managed At maturity level 2, processes are planned and executed in accordance with policy. The projects employ skilled people who have adequate resources to produce controlled outputs; involve relevant stakeholders; are monitored, controlled, and reviewed; and are evaluated for adherence to their process descriptions. 3. Level 3: Defined At maturity level 3, processes are well characterized and understood, and they are described in standards, procedures, tools, and methods. These standard processes are used to establish consistency across the organization. 4. Level 4: Quantitatively Managed At maturity level 4, the organization establishes quantitative objects for quality and process performance and uses them as criteria in managing projects. Quantitative objects are based on the needs of the customer, end-users, organization, and process implementers. Quality and process performance is understood in statistical terms and is managed throughout the life of projects. 5. Level 5: Optimizing At maturity level 5, an organization continually improves its processes based on a quantitative understanding of its business objectives and performance needs. The organization uses a quantitative approach to understanding the variation inherent in the process and the causes of the process outcomes. Change management is a key process for implementing the CMMI-DEV in an organization. Change management is an essential management tool and control mechanism. The concept of segregation of duties ensures that no single individual or organization possesses too much control in a process, helping to prevent errors and fraudulent or malicious acts. The elements of change management - configuration identification, configuration control, configuration status accounting, and configuration auditing - coupled with a defined process and a change control board (CCB). Will provide management with proper oversight of the software lifecycle. Once such a process and management oversight exist, the company can use CCMI-DEV to move from ad hoc activities to a disciplined software management process.

Elements of Change Management

Change management has its roots in system engineering, where it is commonly referred to as configuration management. Most of today's software and hardware change management practices derive from long standing system engineering configuration management practices. Computer hardware and software development have evolved to the point that proper management structure and controls must exist to ensure the products operate as planned. Change and configuration management use different terms for their various phases, but they all fit into 4 general phases defined under configuration management: -Configuration Identification -Configuration Control -Configuration Status Accounting -Configuration Auditing It is important to understand that even though all servers may be initially configured to the same baseline. Individual applications might require a system specific configuration to run properly. Change management actually facilitates system specific configuration in that all exceptions from the standard configuration are documented. All people involved in managing and operating these systems will have documentation to help them quickly understand why a particular system is configured in a unique way.

Code Integrity

One key benefit of adequate change management is the assurance of code consistency and integrity. Whenever a modified program is moved to the production source library, the executable version should also be moved to the production system. Automated change management systems greatly simplify this process and are therefor better controls for ensuring executable and source code integrity. Remember that at no time should the user or application developer have access to the production source and executable code libraries in the production environment.

Implementing Change Management

Change management requires some structure and discipline in order to be effective. the change management function is scalable from small to enterprise level projects. figure 21.1 shows that developers never have access to the production system or data. It implies that a main separation exists between the development, testing and QA, and production environment. This workflow is for changes that have a major impact on production or the customer's business process. For minor changes that have minimal risk or impact on business processes, some of the steps may be omitted. The change management workflow proceeds as follows: 1. The developer checks out source code from the code-control tool archive to the development system. 2. The developer modifies the code and conducts unit testing of the changed modules. 3. The developer checks the modified code into the code control tool archive. 4. The developer notifies the buildmaster that changes are ready for a new build and testing/QA. 5. The buildmaster creates a build incorporating the modified code and compiles the code. 6. The buildmaster notifies the system administrator that the executable image is ready for testing/QA. 7. The system administrator moves the executables to the test/QA system. 8 QA tests the new executables. If the tests are passed, Test/QA notifies the manager. If the tests fail, the process starts over. 9. Upon manager approval, the system administrator moves the executable to the production system.

Configuration Status Accounting

Consists of the procedures for tracking and maintaining data relative to each configuration item in the baseline. It is closely related to configuration control. Status accounting involves gathering and maintaining information relative to each configuration item.

Backout Plan

One of the key elements of a change plan is a comprehensive backout plan. If, in the course of a planned change activity in production, a problem occurs that prevents going forward, it is essential to have a backout plan to restore the system to its previous operating condition. Having backups is essential.

Integrity Measurement

Integrity measurement is the measuring and identification of changes to a specific system away from an expected value. From the simple changing of data as measured by a hash value to the TPM-based integrity measurement of the system boot process and attestation of trust, the concept is the same. Take a known value, perform a storage of a hash or other keyed value, and then at the time of concern, recalculate and compare the two values. In the case of a TPM-mediated system, where the trusted platform module (TPM) chip provides a hardware based root of trust anchor, the system is specifically designed to calculate hashes of a system and store them in a platform configuration register (PRC). This register can be read later and compared to a known, or expected, value, and if they differ, there is a trust violation. Certain BIOSs, UEFIs, and boot loaders can all work with the TPM chip in this manner, providing a means of establishing a trust chain during system boot.

Configuration Identification

Is the process of identifying which assets need to be managed and controlled. These identified assets are called configuration items or computer software configuration items.