This preview shows page 1. Sign up to view the full content.
Unformatted text preview: How Do You Test an IT How Do You Test an IT Service?
Pete Green Tom Frey Purpose Purpose Explore evolving influence of IT systems and services on weapons systems Increasing complexity of testing Our experience foundation is in NMCI Foci are the test processes How Did We Get Here? How Did We Get Here?
NMCI Designated DOT&E Oversight Program NMCI Contract Awarded to EDS Master Test Plan Completed & Delivered TESP Input Delivered Master Test Plan Development CT&E 2 Testing DASN Directs TESP for NMCI 2000 2001 2002 JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN Veridian sole sourced By EDS to conduct NMCI CT&E BSA conducted at 4 test sites CT&E 1 Testing Conducted CT&E 3 Testing Contract SLAs mapped to COIs Interoperability Testing & Monitoring Plan Delivery Rolling In Rolling In Compressed Schedule No Traditional Program Documentation Our Response Based on 4step Test Process Build Up (Crawl, Walk, Run) Concentric Testing Familiar Fundamentals Faces Processes Test Approach Test Approach BSA Phased Testing SLA based derived CTPs, COIs, CTE1 DT (lab, bench) CTE2 DT (networks) CTE3 DT/OT (operational vignettes) OPEVAL OT Test Approach Test Approach
NMCI OPERATIONAL REQUIREMENTS NMCI MEASURES OF EFFECTIVENESS NMCI SERVICE LEVEL AGREEMENTS 1
Desktop Hardware and Operating System SECURITY INTEROPERABILITY & RELIABILITY NETWORK OPERATIONS & MAINTENANCE Electronic Survivability Average Availability Configuration Integrity Data Integrity Network Delays Application Delays Interoperability Reconfigurability Customer Satisfaction Help Desk Support Technology Insertion Latency Recovery Latency Security Report Latency USER UPGRADES 2 1 1 1 1 2 3 2,3 3
END USER SERVICES 1 3 2 2 3 4 10 12 15 E-mail Services Directory Services NMCI Intranet Performance Internet Access
Moves, Adds, and Changes 4 4,5,6 3 2 1,2,4 3 5 4 MAINTENANCE & HELP DESK SERVICES 23 Basic Help Desk Services COMMUNICATIONS SERVIC ES BAN/LAN 25 Communications Service s
NMCI Security Operational 33 Services General 1,2,3,4 ,5,6,7 1,3 4,5,6 9 7,8 1,2,4 2
INFORMATION ASSURANCE 6 4 36 Planning Services NMCI Security Test Approach Test Approach IT Testing IT Testing Similar to Early Aviation Testing How fast can you go? How high can you go? OSI layer 13 centric Parameter focus IT T&E and the OSI 7Layer IT T&E and the OSI 7Layer Model
Where We need to be Mission Based “Is This System Effective and Suitable?” Application Present Commercial Parametric IT Testing “How Fast Can This Car Go?” 7 Presentation 6 Session 5 Transport 4 Network 3 Data Link 2 Physical 1 Product Line Maturity Product Line Maturity Systems Maturity Ai f cra r t A
1950 1960 n vio ic s So ftw a re
1990 t Ne w k or s 1970 1980 2000 C4I Testing C4I Testing Success of testing tied to maturity of program execution OTRR Schedule Constraints IP systems generate lots of data Program Execution curve is fractal Use of surveys Ao Processes on critical path provide 1st order approximation Testing and Program Execution Testing and Program Execution
Idealized Program Execution Curve
% Complete Full Rate Production OPEVAL OTRR Time Test Issues Test Issues NMCI equal to or better than legacy networks Critical path processes SLAs to COIs not a reversible function Operationalizing the metrics Early and combined CT/OT to minimize risk Seat cutover Legacy applications Baseline System Assessment Baseline System Assessment Innovative contract vehicle Lack of traditional documentation Needed to measure the performance of the legacy systems for comparison with NMCI BSA Tools BSA Tools AMP NetIQ MIB scripts Help Desk Logs BelManage Surveys Use same tools and processes for OPEVAL End user CIO and IT professionals Information Assurance BSA Lessons Learned BSA Lessons Learned Configuration management is historical Good management makes an immediate impact Legacy applications Firewalls and IDS Routers Connectivity TESP Process TESP Process Similar to a TEMP From blank sheet to approval in under 11 months Represented a strategy for success for NMCI More responsive If it was not necessary, it was not included If it was necessary, it was tailored to NMCI The State of IT Testing Today The State of IT Testing Today Operational assessments of interoperability and information assurance in fielded systems Inverse correlation of interoperability and information assurance Mission Thread Analysis Blue, Green, and Red Teams Common Toolkits Information Operations Threat Capabilities Summaries Feedback and feed forward Interoperability vs. Information Interoperability vs. Information Assurance: A Complex Challenge
Information Assurance (IA) Interoperability (Iop) Iop(ax) = by IA
An Enterprise System Needs to Balance Iop and IA Testing Frameworks Testing Frameworks
Device Testing System Testing System of Systems Testing
A MissionFocused Testing
Z Work Flow Threads Example Mission Thread Example Mission Thread
TWM & CTB-AM Server 2 3 KCOIC BCD PLNS WS 2 ACC Target Nomination 1 GCC Prioritized KCOIC CTAPS WS KCOIC CTAPS WS 7 KCOIC Integrated ITO Sent to TL # 1,6 Target List Tasking Order GCCS-K C2/COP WS RELROK WAN ROK AF Suwon ( ) RELROK WAN 4 CTAPS ITODB Server RELROK WAN CTAPS TGT WS 5 KUNSAN CTAPS TGT WS 6 8 CP TANGO GCCS-K C2/COP WS RELROK WAN RELROK WAN HTACC CTAPS APS Server Training Facility GCCS-K Web Server HTACC CTAPS WS CTAPS ADS Server RELROK WAN Assessment Methodology Flow
Joint Mission Areas Assigned CoCom Missions Joint Doctrine TT&Ps UJTL Standards Conditions CoCom JMETs Lessons Learned CCTI Strategies Commanders Training Guidance Exercise Comm/Sys Architecture SYSTEM ISSUES
Integrated Architectures System SMEs C4ISP Operational Testers MISSION ISSUES
Scenario MSEL BLUFOR OPLAN OPFOR Campaign Exercise Objectives Mission SMEs Training Objectives Situation Level of Perf Control Plan Execution PERFORMANCE ASSESSMENT Assessment Plan Mission Functional Threads Performance Data Collection Management Plan After Action Review Performance Analysis JTS EXERCISE ISSUES QUESTIONS? QUESTIONS? ...
View Full Document
This note was uploaded on 11/14/2011 for the course COMP 6370 taught by Professor Staff during the Fall '08 term at Auburn University.
- Fall '08