TST 102 Fundamentals of Test and Evaluation
One of the "lessons learned" in testing COTS software products is to assess the availability of documentation for operations and repair. Why is this an important COTS?NDI testing concern?
(All of these responses are correct) >Some essential documentation needed to support the COTS/NDI product may be proprietary or have data rights issues precluding access >Maintenance procedures may be substantially different in the commercial sector and incompatible with military ones >The expected maintenance skill sets for the COTS/NDI product may not be compatible with military's intended support structure
A "covered system" is the DoD term based on Public Law that is intended to include
>A system that is user occupied and designed to provide some degree of protection to its occupants in combat; >A conventional munition program or missile program or conventional munitions program planning to acquire more than 1,000,000 rounds (regardless of major system status); >A modification to a covered system that is likely to significantly affect the survivability or lethality of that system
"Gray box" testing involves partial knowledge of internal structure of an item and typically includes
>Access to the algorithms used >Access to documentation of internal data structures
What are two characteristics of Quick Reaction Tests used in Joint T&E? (Select TWO answers)
>Address urgent, specific, and focused joint issues >Typically less than one year in duration
What are two considerations to keep in mind when testing a software intensive system that employs a significant amount of COTS?
>COTS/NDI spectrum utilization assumptions should be carefully reviewed >The proprietary nature of some COT products may impact interoperability testing
Of the following, which best describe Environment, Safety, and Occupational Health (ESOH) requirements with respect to T&E activities?
>ESOH laws and regulations are complex and comprehensive, and they directly affect how T&E activities are conducted; >ESOH considerations have a big impact on range operations that must be factored into T&E planning
Per DoD Instruction 8500.01, Cybersecurity, key elements of the DoD cybersecurity program include
>Incorporation of security early and continuously within the acquisition lifecycle >Continuous monitoring and timely correction of deficiencies for information systems
What are three typical levels of interoperability that are important considerations in interoperability testing?
>Physical compatibility >Data exchange interoperability >Data interpretation interoperability
Implemented by DoD Instruction 8510.01, Risk Management Framework (RMF) for DoD Information Technology (IT), the RMF includes which of the following steps? (Select 2)
>Select security controls >Assess security controls
Which two of the following best characterize Developmental Test and Evaluation (DT&E)?
>Testing to verify that design risks are minimized >Testing to determine readiness for IOT&E
What are the two key purposes of live fire test and evaluation (LFT&E)?
>To determine the vulnerability of certain types of systems to typical enemy threats; >To determine the lethality of certain types of systems against realistic threat targets
Three of the main objectives of logistics testing and evaluation are to (Select THREE correct answers)
>Verify that the logistic support being developed for the material system is capable of meeting the required objectives. >Verify that the system will be found to be operationally suitable without unintended contractor support. >Verify that the developed system has desirable characteristics that reduce the maintenance and supply burdens required to support the system.
Which of the following would you expect to be essential elements of a test database?
All are correct >Accessible to all stakeholders >Current status of the data >Traceability to the originator of the data
Sound judgement must be exercised when determining type and amount of data to be collected. Data should be
All are correct >Of sufficient quality >Based on Evaluation Plan contents >Timely >Cost effective
With regard to data management and the Contract Data Requirements List (CDRL), the contract must specify
All are correct >Organization that will receive the data >Data the contractor will supply >Whether any outside agencies will be included in direct distribution
Key parts of typical Test and Evaluation Master Plan (TEMP) include
All of these responses are correct (A summary of the overall integrated test program; a summary of needed T&E resources; An outline for DT&E and OT&E activities)
According to DoDD 5000.01, T&E programs should be integrated throughout the acquisition process in order to
All of these responses are correct (Assess attainment of technical performance, Determine if systems are operationally effective and suitable, Provide essential information to decision makers)
Testers use capability documents to
All of these responses are correct (Help develop the TEMP; Understand user needs in specific, quantifiable terms; develop specific test criteria)
Modeling and Simulation (M&S), as a tool to augment and support the T&E process, can be used to:
All of these responses are correct (Increase safety, better evaluate the impact, simulate more scenarios, reduce the use of T&E resources)
Testing of DoD "net centric" systems is typically accomplished
As part of previously scheduled joint warfighting exercises
The formula UPTIME/(UPTIME+DOWNTIME) is one way to measure
Availability
Why would the "white box" testing be preferable in a given software testing circumstance?
By directly examining the structure of the code, more finely-tuned an effective test cases can be developed
A Measure of Effectiveness (MOE) can be defined as
Capabilities that correspond to positive mission outcomes and achievement of desired results
The Deputy Director for Development Test and Evaluation is responsible for a number of key T&E activities. Of the following, which one is NOT one of them?
Conducting independent Operational Testing for each of the services
Two important criteria that should be evaluated to determine the "maturity" of a software product include
Configuration Management processes, Software fuctionality
A simulation that utilizes models and analytical tools to simulate conditions and evaluate the responses of systems to specific input conditions is an example of a
Constructive simulation
Which contracting document outlines the data requirements for the entire test program?
Contract Data Requirements List (CDRL)
With regard to the Risk Management Framework step "categorize system," systems are categorized by the Information System Owner as high, moderate, or low, for all of the following EXCEPT
Cost
A number of key T&E events take place over the acquisition life cycle (DT&E, IOT&E, FOT&E). When considering the entire acquisition life cycle (without regard to any specific phase), which one of the following represents the order that these events are typically performed?
DT&E, IOT&E, FOT&E
Advantages of using Modeling and Simulation (M&S) as a tool to augment and support the T&E process include all of the following EXCEPT
Decreasing operator safety
Current policy found in CJCSI 5123.01 and DoDI 8330.01 on interoperability and certification requirements for Developmental Test and Evaluation (DT&E) and Operational Test and Evaluation (OT&E) requires that
DoD Components will ensure that both DT&E and OT&E processes include mission oriented interoperability assessments
When would you normally expect to see development of the TEMP for a program?
During materiel solution analysis phase, leading into milestone A
Which one of the following DT&E activities is usually NOT performed by the contractor?
Evaluation of government cost and schedule impacts on the test program
A test database should be difficult to use so that engineers trained on the tool will be the only people to access the information
False
Although it is an important task, Test and Evaluation has been found to increase the risks in acquiring, developing, and using a DoD system because of the time and effort involved in its performance
False
Because of the risks involved, the use of Modeling and Simulation (M&S) as a tool to augment and support the T&E process is prohibited by DoD policies.
False
Because the best way to ensure a quality software product is to test it exhaustively, the best software testing approach is to defer testing until development has been totally completed.
False
Ethical dilemmas always involve choices between a clearly defined "right" course of action and a "wrong" course of action
False
One advantage of heavy use of COTS software in a system is that, because these products are in commercial use, much of DT&E, LFT&E, and OT&E can be waived
False
Since most systems in the combat environment operate independently of imbedded computers and electronics communication devices, potential damage to systems used in combat is not a significant concern for DoD.
False
The Operational Test Readiness Review (OTRR) is a multi-disciplined product and process assessment that guarantees successful completion of Initial Operational Test and Evaluation (IOT&E)
False
Technology readiness level (TRL) of 9 (Actual system proven through successful mission operations) likely indicates technology maturity encountered during
Fielding and deployment of a system
The purpose of _______ is to verify the correction of deficiencies and to evaluate the operational effect of changes, modifications, and upgrades that have been made to the system
Follow-on Operational Test and Evaluation (FOT&E)
The purposes of Test and Evaluation in defense system development include
Help to reduce development and acquisition risks Assess the operational suitability of a system Provide assessments of system maturity
Advantages of a concurrent DT/OT strategy include all of the following EXCEPT
Increasing the competition for access to limited test articles
Reliability Growth testing
Is one way test results can be used to improve designs
The Defense Information Systems Agency and the subordinate Joint Interoperability Test Command are responsible for
Issuing Joint Interoperability Assessments for observed/conducted DT&E and OT&E interoperability testing
Two generic ways to classify software testing activities are: Human based testing and computer based testing. Which of the following most accurately describes computer based testing?
It involves primarily the use of running test scripts on a computer system
Which organization coordinates, monitors and executes interoperability testing?
Joint Interoperability Test Command (JITC)
Which one of the following is a characteristics of multiservice T&E?
Multiservice T&E is conducted according to lead service procedures, but T&E reports may contain service specific annexes
The Federal Information Security Management Act of 2002 tasked which agency with responsibilities for standards and guidelines for information security, including the development of minimum security requirements for information and information systems?
National Institute of Standards and Technology
One of the key differences between OT&E and DT&E is
OT&E is conducted by an agency independent of both the program and the user
The "overall degree of mission accomplishment of a system when ussed by typical users in a realistic environment" is
Operational Effectiveness
A software problem that is characterized as "adversely impacting mission accomplishment but a known work around exists has been reported. This is an example of a
Priority 3 software problem
A software problem that is characterized as "this would be nice to fix but is not essential for mission accomplishment" has been reported. This is an example of a
Priority 5 software problem
A technical test completed prior to the Full Rate Production (FRP) decision to ensure the effectiveness of the manufacturing process, equipment, and procedures is called
Production qualification testing
Live fire test and evaluation (LFT&E)
Provides a realistic assessment of a system's vulnerability and lethality
With regard to T&E for rapid acquisition programs
Schedule considerations dominate over cost and technical risk considerations
With regard to an incremental acquisition program
Since capabilities will improve between subsequent increments, it would not be appropriate to plan for an integrated test strategy that considers more than one increment
According to the Software Development "V" model, which ordering of activities reflects how software is typically developed and tested?
Software and coding and unit-level testing, Unit integration Testing, CSCI/HWCI Testing
Activities such as reviewing and analyzing sources or requirements and determining which operational issues to evaluate are typically performed in which step of the DoD T&E process?
Step 1: Identify Information Requirements
One of the key documents produced by the T&E WIPT is the
Test & Evaluation Master Plan (TEMP)
Which of the following key T&E activities typically takes place during the Materiel Solution Analysis phase of the acquisition life cycle?
The initial drafting of the TEMP
One of the "lessons learned" in testing COTS software products is to assess the amount of "glue code" being used. Why is glue code an important software consideration?
The use of undocumented and uncontrolled glue code increases the risk of downstream malfunctions
The development of Measures of Performance (MOPs) is an important part of setting up a T&E program. Why are MOPs considered to be so important?
They are directly measurable parameters that can be specifically tested and measured
A Key Performance Parameter (KPP) can be defined as
Those attributes that are considered most essential for an effective military capability
"Accreditation" is a particularly important concept because all M&S assets used for testing must be certified as acceptable for use in specific types of applications.
True
A T&E Process that strictly adheres to well-established exit and entrance criteria is one that is considered to be Event-Driven, not Schedule-Driven
True
A key criterion considered before starting Initial Operational Testing and Evaluation (IOT&E), is the "maturity level" of the software.
True
Any system deemed to be a "covered system" by the Secretary of Defense must undergo Life Fire Test and Evaluation (LFT&E)
True
Because hardware and software must work together to determine if the system can meet the user's needs, the software T&E process doesn't differ in concept from the general T&E process.
True
During the technology maturation and risk reduction phase, testing focuses on prototypes demonstrating technology capabilities (e.g. ATDs, ACTDs, mock-ups)
True
Each service has its own service-unique test and evaluation policies that supplement those found in DoD directives and instructions
True
Early OT&E activities may require support from system contractors, program office personnel, DT&E elements, and operational forces
True
In addition to Public Law and DoD 5000-series documents, the military services publish guidance applicable to their own T&E activities.
True
Multiservice T&E is used for programs acquiring a materiel solution for more than one service
True
Per the Risk Management Framework (RMF), the Authorizing Official (AO) ensures that all appropriate RMF tasks are initiated and completed, with appropriate documentation, for assigned information systems
True
Results of T&E help to mitigate risk by providing timely and accurate information to decision-makers over the acquisition life cycle.
True
T&E provides the feedback on the effectiveness of engineering development efforts at discrete points in time
True
The TEMP can play a key role in pin-pointing net centric testing opportunities
True
Two different acquisition approaches for developing and integrating systems include a "single step acquisition" where the system is developed in one large effort and "incremental acquisition" where core functionality is developed and then progressively improved.
True
When used with inferential statistics to predict the reliability of an entire system based upon the results from a limited number of tests, a confidence level quantifies the belief the tester has that a particular requirement has been met
True
The last step in the DoD T&E Process is
Utilization of Test Data
The answer to the T&E related question of "Was the product built right from a technical perspective?" is primarily answered by the Systems Engineering Technical Process of
Verification
The ability of an item to be retained in, or restored to a specific condition is a measure of its
maintainability