TST 102 Fundamentals of Test and Evaluation

One of the "lessons learned" in testing COTS software products is to assess the availability of documentation for operations and repair. Why is this an important COTS?NDI testing concern?

(All of these responses are correct) >Some essential documentation needed to support the COTS/NDI product may be proprietary or have data rights issues precluding access >Maintenance procedures may be substantially different in the commercial sector and incompatible with military ones >The expected maintenance skill sets for the COTS/NDI product may not be compatible with military's intended support structure

A "covered system" is the DoD term based on Public Law that is intended to include

>A system that is user occupied and designed to provide some degree of protection to its occupants in combat; >A conventional munition program or missile program or conventional munitions program planning to acquire more than 1,000,000 rounds (regardless of major system status); >A modification to a covered system that is likely to significantly affect the survivability or lethality of that system

"Gray box" testing involves partial knowledge of internal structure of an item and typically includes

>Access to the algorithms used >Access to documentation of internal data structures

What are two characteristics of Quick Reaction Tests used in Joint T&E? (Select TWO answers)

>Address urgent, specific, and focused joint issues >Typically less than one year in duration

What are two considerations to keep in mind when testing a software intensive system that employs a significant amount of COTS?

>COTS/NDI spectrum utilization assumptions should be carefully reviewed >The proprietary nature of some COT products may impact interoperability testing

Per DoD Instruction 8500.01, Cybersecurity, key elements of the DoD cybersecurity program include

>Incorporation of security early and continuously within the acquisition lifecycle >Continuous monitoring and timely correction of deficiencies for information systems

What are three typical levels of interoperability that are important considerations in interoperability testing?

>Physical compatibility >Data exchange interoperability >Data interpretation interoperability

Three of the main objectives of logistics testing and evaluation are to (Select THREE correct answers)

>Verify that the logistic support being developed for the material system is capable of meeting the required objectives. >Verify that the system will be found to be operationally suitable without unintended contractor support. >Verify that the developed system has desirable characteristics that reduce the maintenance and supply burdens required to support the system.

Sound judgement must be exercised when determining type and amount of data to be collected. Data should be

All are correct >Of sufficient quality >Based on Evaluation Plan contents >Timely >Cost effective

With regard to data management and the Contract Data Requirements List (CDRL), the contract must specify

All are correct >Organization that will receive the data >Data the contractor will supply >Whether any outside agencies will be included in direct distribution

Testers use capability documents to

All of these responses are correct (Help develop the TEMP; Understand user needs in specific, quantifiable terms; develop specific test criteria)

Modeling and Simulation (M&S), as a tool to augment and support the T&E process, can be used to:

All of these responses are correct (Increase safety, better evaluate the impact, simulate more scenarios, reduce the use of T&E resources)

Testing of DoD "net centric" systems is typically accomplished

As part of previously scheduled joint warfighting exercises

The formula UPTIME/(UPTIME+DOWNTIME) is one way to measure

Availability

Why would the "white box" testing be preferable in a given software testing circumstance?

By directly examining the structure of the code, more finely-tuned an effective test cases can be developed

Which contracting document outlines the data requirements for the entire test program?

Contract Data Requirements List (CDRL)

Because of the risks involved, the use of Modeling and Simulation (M&S) as a tool to augment and support the T&E process is prohibited by DoD policies.

False

Ethical dilemmas always involve choices between a clearly defined "right" course of action and a "wrong" course of action

False

Advantages of a concurrent DT/OT strategy include all of the following EXCEPT

Increasing the competition for access to limited test articles

Reliability Growth testing

Is one way test results can be used to improve designs

One of the key differences between OT&E and DT&E is

OT&E is conducted by an agency independent of both the program and the user

With regard to T&E for rapid acquisition programs

Schedule considerations dominate over cost and technical risk considerations

One of the "lessons learned" in testing COTS software products is to assess the amount of "glue code" being used. Why is glue code an important software consideration?

The use of undocumented and uncontrolled glue code increases the risk of downstream malfunctions

In addition to Public Law and DoD 5000-series documents, the military services publish guidance applicable to their own T&E activities.

True

Multiservice T&E is used for programs acquiring a materiel solution for more than one service

True

Since most systems in the combat environment operate independently of imbedded computers and electronics communication devices, potential damage to systems used in combat is not a significant concern for DoD.

False

The Operational Test Readiness Review (OTRR) is a multi-disciplined product and process assessment that guarantees successful completion of Initial Operational Test and Evaluation (IOT&E)

False

Technology readiness level (TRL) of 9 (Actual system proven through successful mission operations) likely indicates technology maturity encountered during

Fielding and deployment of a system

The purpose of _______ is to verify the correction of deficiencies and to evaluate the operational effect of changes, modifications, and upgrades that have been made to the system

Follow-on Operational Test and Evaluation (FOT&E)

Live fire test and evaluation (LFT&E)

Provides a realistic assessment of a system's vulnerability and lethality

Per the Risk Management Framework (RMF), the Authorizing Official (AO) ensures that all appropriate RMF tasks are initiated and completed, with appropriate documentation, for assigned information systems

True

Results of T&E help to mitigate risk by providing timely and accurate information to decision-makers over the acquisition life cycle.

True

What are the two key purposes of live fire test and evaluation (LFT&E)?

>To determine the vulnerability of certain types of systems to typical enemy threats; >To determine the lethality of certain types of systems against realistic threat targets

T&E provides the feedback on the effectiveness of engineering development efforts at discrete points in time

True

With regard to the Risk Management Framework step "categorize system," systems are categorized by the Information System Owner as high, moderate, or low, for all of the following EXCEPT

Cost

One advantage of heavy use of COTS software in a system is that, because these products are in commercial use, much of DT&E, LFT&E, and OT&E can be waived

False

Of the following, which best describe Environment, Safety, and Occupational Health (ESOH) requirements with respect to T&E activities?

>ESOH laws and regulations are complex and comprehensive, and they directly affect how T&E activities are conducted; >ESOH considerations have a big impact on range operations that must be factored into T&E planning

Implemented by DoD Instruction 8510.01, Risk Management Framework (RMF) for DoD Information Technology (IT), the RMF includes which of the following steps? (Select 2)

>Select security controls >Assess security controls

Which two of the following best characterize Developmental Test and Evaluation (DT&E)?

>Testing to verify that design risks are minimized >Testing to determine readiness for IOT&E

Which of the following would you expect to be essential elements of a test database?

All are correct >Accessible to all stakeholders >Current status of the data >Traceability to the originator of the data

Key parts of typical Test and Evaluation Master Plan (TEMP) include

All of these responses are correct (A summary of the overall integrated test program; a summary of needed T&E resources; An outline for DT&E and OT&E activities)

According to DoDD 5000.01, T&E programs should be integrated throughout the acquisition process in order to

All of these responses are correct (Assess attainment of technical performance, Determine if systems are operationally effective and suitable, Provide essential information to decision makers)

A Measure of Effectiveness (MOE) can be defined as

Capabilities that correspond to positive mission outcomes and achievement of desired results

The Deputy Director for Development Test and Evaluation is responsible for a number of key T&E activities. Of the following, which one is NOT one of them?

Conducting independent Operational Testing for each of the services

Two important criteria that should be evaluated to determine the "maturity" of a software product include

Configuration Management processes, Software fuctionality

A simulation that utilizes models and analytical tools to simulate conditions and evaluate the responses of systems to specific input conditions is an example of a

Constructive simulation

A number of key T&E events take place over the acquisition life cycle (DT&E, IOT&E, FOT&E). When considering the entire acquisition life cycle (without regard to any specific phase), which one of the following represents the order that these events are typically performed?

DT&E, IOT&E, FOT&E

Advantages of using Modeling and Simulation (M&S) as a tool to augment and support the T&E process include all of the following EXCEPT

Decreasing operator safety

Current policy found in CJCSI 5123.01 and DoDI 8330.01 on interoperability and certification requirements for Developmental Test and Evaluation (DT&E) and Operational Test and Evaluation (OT&E) requires that

DoD Components will ensure that both DT&E and OT&E processes include mission oriented interoperability assessments

When would you normally expect to see development of the TEMP for a program?

During materiel solution analysis phase, leading into milestone A

Which one of the following DT&E activities is usually NOT performed by the contractor?

Evaluation of government cost and schedule impacts on the test program

A test database should be difficult to use so that engineers trained on the tool will be the only people to access the information

False

Although it is an important task, Test and Evaluation has been found to increase the risks in acquiring, developing, and using a DoD system because of the time and effort involved in its performance

False

Because the best way to ensure a quality software product is to test it exhaustively, the best software testing approach is to defer testing until development has been totally completed.

False

The purposes of Test and Evaluation in defense system development include

Help to reduce development and acquisition risks Assess the operational suitability of a system Provide assessments of system maturity

The Defense Information Systems Agency and the subordinate Joint Interoperability Test Command are responsible for

Issuing Joint Interoperability Assessments for observed/conducted DT&E and OT&E interoperability testing

Two generic ways to classify software testing activities are: Human based testing and computer based testing. Which of the following most accurately describes computer based testing?

It involves primarily the use of running test scripts on a computer system

Which organization coordinates, monitors and executes interoperability testing?

Joint Interoperability Test Command (JITC)

Which one of the following is a characteristics of multiservice T&E?

Multiservice T&E is conducted according to lead service procedures, but T&E reports may contain service specific annexes

The Federal Information Security Management Act of 2002 tasked which agency with responsibilities for standards and guidelines for information security, including the development of minimum security requirements for information and information systems?

National Institute of Standards and Technology

The "overall degree of mission accomplishment of a system when ussed by typical users in a realistic environment" is

Operational Effectiveness

A software problem that is characterized as "adversely impacting mission accomplishment but a known work around exists has been reported. This is an example of a

Priority 3 software problem

A software problem that is characterized as "this would be nice to fix but is not essential for mission accomplishment" has been reported. This is an example of a

Priority 5 software problem

A technical test completed prior to the Full Rate Production (FRP) decision to ensure the effectiveness of the manufacturing process, equipment, and procedures is called

Production qualification testing

With regard to an incremental acquisition program

Since capabilities will improve between subsequent increments, it would not be appropriate to plan for an integrated test strategy that considers more than one increment

According to the Software Development "V" model, which ordering of activities reflects how software is typically developed and tested?

Software and coding and unit-level testing, Unit integration Testing, CSCI/HWCI Testing

Activities such as reviewing and analyzing sources or requirements and determining which operational issues to evaluate are typically performed in which step of the DoD T&E process?

Step 1: Identify Information Requirements

One of the key documents produced by the T&E WIPT is the

Test & Evaluation Master Plan (TEMP)

Which of the following key T&E activities typically takes place during the Materiel Solution Analysis phase of the acquisition life cycle?

The initial drafting of the TEMP

The development of Measures of Performance (MOPs) is an important part of setting up a T&E program. Why are MOPs considered to be so important?

They are directly measurable parameters that can be specifically tested and measured

A Key Performance Parameter (KPP) can be defined as

Those attributes that are considered most essential for an effective military capability

"Accreditation" is a particularly important concept because all M&S assets used for testing must be certified as acceptable for use in specific types of applications.

True

A T&E Process that strictly adheres to well-established exit and entrance criteria is one that is considered to be Event-Driven, not Schedule-Driven

True

A key criterion considered before starting Initial Operational Testing and Evaluation (IOT&E), is the "maturity level" of the software.

True

Any system deemed to be a "covered system" by the Secretary of Defense must undergo Life Fire Test and Evaluation (LFT&E)

True

Because hardware and software must work together to determine if the system can meet the user's needs, the software T&E process doesn't differ in concept from the general T&E process.

True

During the technology maturation and risk reduction phase, testing focuses on prototypes demonstrating technology capabilities (e.g. ATDs, ACTDs, mock-ups)

True

Each service has its own service-unique test and evaluation policies that supplement those found in DoD directives and instructions

True

Early OT&E activities may require support from system contractors, program office personnel, DT&E elements, and operational forces

True

The TEMP can play a key role in pin-pointing net centric testing opportunities

True

Two different acquisition approaches for developing and integrating systems include a "single step acquisition" where the system is developed in one large effort and "incremental acquisition" where core functionality is developed and then progressively improved.

True

When used with inferential statistics to predict the reliability of an entire system based upon the results from a limited number of tests, a confidence level quantifies the belief the tester has that a particular requirement has been met

True

The last step in the DoD T&E Process is

Utilization of Test Data

The answer to the T&E related question of "Was the product built right from a technical perspective?" is primarily answered by the Systems Engineering Technical Process of

Verification

The ability of an item to be retained in, or restored to a specific condition is a measure of its

maintainability