Hubbry Logo
Software verification and validationSoftware verification and validationMain
Open search
Software verification and validation
Community hub
Software verification and validation
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Software verification and validation
Software verification and validation
Software verification and validation
View on Wikipedia
from Wikipedia
IEEE software life cycle

In software project management, software testing, and software engineering, verification and validation is the process of checking that a software system meets specifications and requirements so that it fulfills its intended purpose. It may also be referred to as software quality control. It is normally the responsibility of software testers as part of the software development lifecycle. In simple terms, software verification is: "Assuming we should build X, does our software achieve its goals without any bugs or gaps?" On the other hand, software validation is: "Was X what we should have built? Does X meet the high-level requirements?"

Definitions

[edit]

Verification and validation are not the same thing, although they are often confused. Boehm succinctly expressed the difference as[1]

  • Verification: Are we building the product right?
  • Validation: Are we building the right product?

"Building the product right" checks that the specifications are correctly implemented by the system while "building the right product" refers back to the user's needs. In some contexts, it is required to have written requirements for both as well as formal procedures or protocols for determining compliance. Ideally, formal methods provide a mathematical guarantee that software meets its specifications.

Building the product right implies the use of the Requirements Specification as input for the next phase of the development process, the design process, the output of which is the Design Specification. Then, it also implies the use of the Design Specification to feed the construction process. Every time the output of a process correctly implements its input specification, the software product is one step closer to final verification. If the output of a process is incorrect, the developers have not correctly implemented some component of that process. This kind of verification is called "artifact or specification verification".

Software verification

[edit]

It would imply to verify if the specifications are met by running the software but this is not possible (e.g., how can anyone know if the architecture/design/etc. are correctly implemented by running the software?). Only by reviewing its associated artifacts, can someone conclude whether or not the specifications are met.

Artifact or specification verification

[edit]

The output of each software development process stage can also be subject to verification when checked against its input specification (see the definition by CMMI below).

Examples of artifact verification:

  • Of the design specification against the requirement specification: Do the architectural design, detailed design and database logical model specifications correctly implement the functional and non-functional requirements specifications?
  • Of the construction artifacts against the design specification: Do the source code, user interfaces and database physical model correctly implement the design specification?

Software validation

[edit]

Software validation checks that the software product satisfies or fits the intended use (high-level checking), i.e., the software meets the user requirements, not as specification artifacts or as needs of those who will operate the software only; but, as the needs of all the stakeholders (such as users, operators, administrators, managers, investors, etc.). There are two ways to perform software validation: internal and external. During internal software validation, it is assumed that the goals of the stakeholders were correctly understood and that they were expressed in the requirement artifacts precisely and comprehensively. If the software meets the requirement specification, it has been internally validated. External validation happens when it is performed by asking the stakeholders if the software meets their needs. Different software development methodologies call for different levels of user and stakeholder involvement and feedback; so, external validation can be a discrete or a continuous event. Successful final external validation occurs when all the stakeholders accept the software product and express that it satisfies their needs. Such final external validation requires the use of an acceptance test which is a dynamic test.

However, it is also possible to perform internal static tests to find out if the software meets the requirements specification but that falls into the scope of static verification because the software is not running.

Artifact or specification validation

[edit]

Requirements should be validated before the software product as a whole is ready (the waterfall development process requires them to be perfectly defined before design starts; but iterative development processes do not require this to be so and allow their continual improvement).

Examples of artifact validation:

  • User Requirements Specification validation: User requirements as stated in a document called User Requirements Specification are validated by checking if they indeed represent the will and goals of the stakeholders. This can be done by interviewing the stakeholders and asking them directly (static testing) or even by releasing prototypes and having the users and stakeholders to assess them (dynamic testing).
  • User input validation: User input (gathered by any peripheral such as a keyboard, bio-metric sensor, etc.) is validated by checking if the input provided by the software operators or users meets the domain rules and constraints (such as data type, range, and format).

Validation vs. verification

[edit]

According to the Capability Maturity Model (CMMI-SW v1.1),[2]

  • Software Validation: The process of evaluating software during or at the end of the development process to determine whether it satisfies specified requirements. [IEEE-STD-610]
  • Software Verification: The process of evaluating software to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. [IEEE-STD-610]

Validation during the software development process can be seen as a form of User Requirements Specification validation; and, that at the end of the development process is equivalent to Internal and/or External Software validation. Verification, from CMMI's point of view, is evidently of the artifact kind.

In other words, software verification ensures that the output of each phase of the software development process effectively carries out what its corresponding input artifact specifies (requirement -> design -> software product), while software validation ensures that the software product meets the needs of all the stakeholders (therefore, the requirement specification was correctly and accurately expressed in the first place). Software verification ensures that "you built it right" and confirms that the product, as provided, fulfills the plans of the developers. Software validation ensures that "you built the right thing" and confirms that the product, as provided, fulfills the intended use and goals of the stakeholders.

This article has used the strict or narrow definition of verification.

From a testing perspective:

  • Fault – wrong or missing function in the code.
  • Failure – the manifestation of a fault during execution. The software was not effective. It does not do "what" it is supposed to do.
  • Malfunction – according to its specification the system does not meet its specified functionality. The software was not efficient (it took too many resources such as CPU cycles, it used too much memory, performed too many I/O operations, etc.), it was not usable, it was not reliable, etc. It does not do something "how" it is supposed to do it.
[edit]

Both verification and validation are related to the concepts of quality and of software quality assurance. By themselves, verification and validation do not guarantee software quality; planning, traceability, configuration management and other aspects of software engineering are required.

Within the modeling and simulation (M&S) community, the definitions of verification, validation and accreditation are similar:

  • M&S Verification is the process of determining that a computer model, simulation, or federation of models and simulation implementations and their associated data accurately represent the developer's conceptual description and specifications.[3]
  • M&S Validation is the process of determining the degree to which a model, simulation, or federation of models and simulations, and their associated data are accurate representations of the real world from the perspective of the intended use(s).[3]
  • Accreditation is the formal certification that a model or simulation is acceptable to be used for a specific purpose.[3]

The definition of M&S validation focuses on the accuracy with which the M&S represents the real-world intended use(s). Determining the degree of M&S accuracy is required because all M&S are approximations of reality, and it is usually critical to determine if the degree of approximation is acceptable for the intended use(s). This stands in contrast to software validation.

V&V methods

[edit]

Formal

[edit]

In mission-critical software systems, formal methods may be used to ensure the correct operation of a system. These formal methods can prove costly, however, representing as much as 80 percent of total software design cost.

Independent

[edit]

Independent Software Verification and Validation (ISVV) is targeted at safety-critical software systems and aims to increase the quality of software products, thereby reducing risks and costs throughout the operational life of the software. The goal of ISVV is to provide assurance that software performs to the specified level of confidence and within its designed parameters and defined requirements.[4][5]

ISVV activities are performed by independent engineering teams, not involved in the software development process, to assess the processes and the resulting products. The ISVV team independency is performed at three different levels: financial, managerial and technical.

ISVV goes beyond "traditional" verification and validation techniques, applied by development teams. While the latter aims to ensure that the software performs well against the nominal requirements, ISVV is focused on non-functional requirements such as robustness and reliability, and on conditions that can lead the software to fail.

ISVV results and findings are fed back to the development teams for correction and improvement.

History

[edit]

ISVV derives from the application of IV&V (Independent Verification and Validation) to the software. Early ISVV application (as known today) dates back to the early 1970s when the U.S. Army sponsored the first significant program related to IV&V for the Safeguard Anti-Ballistic Missile System.[6] Another example is NASA's IV&V Program, which was established in 1993.[7]

By the end of the 1970s IV&V was rapidly becoming popular. The constant increase in complexity, size and importance of the software led to an increasing demand on IV&V applied to software.

Meanwhile, IV&V (and ISVV for software systems) consolidated and is now widely used by organizations such as the DoD, FAA,[8] NASA[7] and ESA.[9] IV&V is mentioned in DO-178B, ISO/IEC 12207 and formalized in IEEE 1012.

At ESA

[edit]

Initially in 2004-2005, a European consortium led by the European Space Agency, and composed of DNV, Critical Software SA, Terma and CODA SciSys plc created the first version of a guide devoted to ISVV, called "ESA Guide for Independent Verification and Validation" with support from other organizations.[10] This guide covers the methodologies applicable to all the software engineering phases in what concerns ISVV.

In 2008 the European Space Agency released a second version, having received inputs from many different European Space ISVV stakeholders.[10]

Methodology

[edit]

ISVV is usually composed of five principal phases, these phases can be executed sequentially or as results of a tailoring process.

Planning

[edit]
  • Planning of ISVV activities
  • System criticality analysis: Identification of critical components through a set of RAMS activities (Value for Money)
  • Selection of the appropriate methods and tools

Requirements verification

[edit]
  • Verification for: completeness, correctness, testability

Design verification

[edit]
  • Design adequacy and conformance to software requirements and interfaces
  • Internal and external consistency
  • Verification of feasibility and maintenance

Code verification

[edit]

Validation

[edit]
  • Identification of unstable components/functionalities
  • Validation focused on error-handling: complementary (not concurrent) validation regarding the one performed by the development team
  • Compliance with software and system requirements
  • Black box testing and White box testing techniques
  • Experience based techniques

Regulatory environment

[edit]

Software often must meet the compliance requirements of legally regulated industries, which is often guided by government agencies[11][12] or industrial administrative authorities. For instance, the FDA requires software versions and patches to be validated.[13]

See also

[edit]

Further reading

[edit]
[edit]

References

[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Software verification and validation

View on Grokipedia
from Grokipedia
Software verification and validation (V&V) refers to the set of systematic processes used to evaluate software products throughout their development lifecycle to ensure they conform to specified requirements and fulfill their intended purpose. Verification specifically involves confirming that the software development outputs at each phase meet the conditions and specifications established at the beginning of that phase, answering the question of whether the product is being built correctly. In contrast, validation assesses whether the final software system satisfies the overall user needs and intended use, determining if the right product has been built. These processes are essential for mitigating risks, detecting defects early, and ensuring software reliability, particularly in safety-critical domains such as , healthcare, and . The V&V processes are integrated across all stages of the software lifecycle, from to deployment and maintenance, as outlined in standards like IEEE Std 1012-2024. Key activities include reviews, inspections, walkthroughs, (such as static code analysis and modeling), and testing (encompassing unit, integration, , and ). These activities are tailored based on software integrity levels, which classify systems according to potential consequences of failure, allowing for proportional rigor in application. For instance, high-integrity software in medical devices requires more extensive V&V than low-risk applications. V&V is guided by international standards that promote consistency and best practices, including IEEE Std 1012-2024 for , and hardware V&V, and ISO/IEC/IEEE 15288:2023 for broader processes. Independent V&V, often performed by a separate team, enhances objectivity and is mandated in regulated industries to build stakeholder confidence. By addressing both technical correctness and user alignment, V&V contributes to higher , reduced costs from rework, and compliance with regulatory requirements.

Fundamentals

Core Definitions

Software verification is the process of evaluating software products or artifacts of a development phase to determine whether they satisfy the conditions imposed at the start of that phase, ensuring conformance to specified requirements and the absence of defects. This evaluation typically involves static techniques such as inspections, which are formal peer reviews of documents or for defects; walkthroughs, where a developer explains the product to colleagues for feedback; and reviews, which are systematic examinations to identify discrepancies from standards or requirements. Software validation is the process of evaluating the software during or at the end of the development process to determine whether it satisfies specified requirements and fulfills its intended use in the target environment, thereby confirming it meets user needs and is fit for purpose. Artifact verification refers to the evaluation of non-executable products, such as requirements specifications, documents, or plans, to check their compliance with applicable standards, consistency, and completeness. In contrast, artifact validation ensures these non-code elements align with the broader project objectives, stakeholder expectations, and the overall intended functionality of the system. serves as a dynamic of verification, focusing on executing the software to uncover defects. The processes for software verification and validation were formalized in IEEE Std 1012-1986, which provided the initial standard for plans and processes for these activities to support uniform application across software projects.

Verification Versus Validation

Verification and validation serve complementary yet distinct roles in ensuring software quality, often summarized by the paradigm of "building the thing right" for verification and "building the right thing" for validation. Verification addresses whether the software product is constructed correctly according to its specifications and design, focusing on internal consistency, adherence to standards, and absence of defects in implementation. This process typically involves activities like code reviews, static analysis, and testing against requirements to confirm that the system meets predefined technical criteria. In contrast, validation evaluates whether the software fulfills its intended purpose in the real-world context, assessing user needs, operational fitness, and overall suitability for the problem it aims to solve. This distinction ensures that verification prevents errors in development while validation confirms alignment with stakeholder expectations. A key framework illustrating this interplay is the of , which visually integrates into the lifecycle. In the , the left descending arm represents progressive development phases—from requirements to design and implementation—where verification activities occur at each corresponding ascending point on the right arm, such as verifying code against design specs and verifying modules against architectural requirements. Validation, depicted on the upper right arm, culminates in system and to ensure the final product addresses the original user needs, forming a "V" shape that emphasizes iterative checks throughout. This model highlights how verification builds confidence in the product's internal correctness, while validation bridges the gap to external applicability, promoting early defect detection and reducing lifecycle risks. Common misconceptions arise when validation is narrowly equated with user acceptance testing (UAT), overlooking its broader scope that includes ongoing evaluations of , performance in target environments, and alignment with evolving needs throughout the project. Treating validation solely as an end-stage activity can lead to overlooked discrepancies between specified requirements and actual user contexts, whereas verification is sometimes misperceived as sufficient alone, ignoring that a perfectly implemented system may still fail to deliver value if it solves the wrong problem. These errors underscore the need for integrated V&V planning to avoid siloed approaches. Misalignment between verification and validation has profound impacts on software quality, often resulting in costly rework, delays, and safety risks when products pass internal checks but fail in deployment. A stark example is the machine incidents between 1985 and 1987, where rigorous verification confirmed the software met design specifications for beam control, but inadequate validation failed to account for real-world operator interactions and hardware-software race conditions, leading to overdoses that caused patient injuries and deaths. Post-incident analyses revealed that enhanced validation, including scenario-based testing of human factors, could have identified these gaps earlier, highlighting how verification alone cannot mitigate context-specific failures and emphasizing the economic and ethical imperative for balanced V&V practices.

Software Testing

Software testing involves the dynamic execution of a or component under specified conditions to evaluate its behavior and uncover defects, serving as a core subset of verification activities within software verification and validation. This focuses on demonstrating that the software meets its specified requirements by observing outputs in response to inputs, thereby helping to identify discrepancies between expected and actual performance. Unlike static analysis techniques, dynamic testing requires running the code in a controlled environment to reveal runtime errors, integration issues, or non-conformities that might not be apparent through inspection alone. Testing occurs across multiple levels to progressively validate the software from individual components to the complete system. Unit testing examines isolated software units, such as functions or modules, to verify their internal logic and functionality. Integration testing assesses the interactions between these units after assembly, ensuring data flow and interface compatibility. System testing evaluates the fully integrated system against overall requirements, including non-functional aspects like performance and security. Acceptance testing confirms that the system satisfies user needs and is ready for deployment, often involving end-users. These levels incorporate black-box approaches, which treat the software as opaque and focus on inputs, outputs, and specifications without internal knowledge, and white-box approaches, which leverage the software's structure, such as control flows, to guide test selection. Recent updates to ISO/IEC/IEEE 29119 include Part 5:2024 on keyword-driven testing. Test case design techniques systematically derive inputs to maximize defect detection efficiency. Equivalence partitioning divides the input domain into classes where each class is expected to exhibit similar behavior, selecting one representative test case per class to reduce redundancy while covering diverse scenarios. complements this by targeting values at the edges of partitions, as defects often cluster near boundaries due to off-by-one errors or limit mishandling. Decision table testing models complex business rules as tables of conditions and actions, generating test cases for all valid and invalid combinations to ensure comprehensive coverage of logical paths. Metrics quantify testing effectiveness and guide improvements. Code coverage measures the extent to which the source code is exercised by tests, with statement coverage tracking the proportion of statements run, branch coverage assessing decision outcomes (e.g., true/false paths in conditionals), and path coverage verifying all possible execution sequences. Defect density calculates the number of confirmed defects per unit of software size, typically defects per thousand lines of code (KLOC), providing an indicator of overall and risk concentration. Software testing has evolved from ad-hoc manual practices in the mid-20th century, where developers informally checked code, to structured automated frameworks that enhance and scale. Pioneering tools like , introduced in 1997 for , enabled developer-driven automation through simple assertions and fixtures. , released in 2004, extended this to web application testing by simulating user interactions across browsers, facilitating end-to-end validation. This shift overlaps with processes by integrating testing into preventive measures for reliable software delivery.

Quality Assurance and Control

Quality assurance (QA) in software engineering focuses on preventive measures to ensure that development processes adhere to established standards and best practices, thereby building quality into the software from the outset. In contrast, quality control (QC) emphasizes detective measures, involving the inspection and evaluation of the final product or deliverables to identify defects and ensure compliance with requirements. This distinction positions QA as an overarching, process-oriented framework that influences all stages of the software life cycle, while QC serves as a targeted, product-oriented activity often conducted through audits and reviews. Verification and validation (V&V) play integral roles within QA by ensuring process compliance and product suitability, respectively. Verification confirms that the software development processes and intermediate products meet specified standards and design requirements, thereby supporting QA's preventive goals through systematic checks like traceability analysis and interface reviews. Validation, on the other hand, verifies that the final software fulfills its intended purpose and user needs, complementing QA by bridging the gap between process adherence and real-world performance, often through activities like prototyping and user evaluations. Together, V&V enhances QA by providing objective, engineering-based assessments that monitor process effectiveness and product quality across the life cycle. Key tools and practices in QA and QC include audits, peer reviews, and process maturity models such as the (CMMI). Audits, such as functional and physical configuration audits, systematically evaluate whether software products align with documentation and requirements, serving as a core QC mechanism to detect discrepancies. Peer reviews, encompassing informal walkthroughs and formal inspections, enable early defect identification by involving team members in structured evaluations of , designs, and documents, thereby reinforcing QA's preventive ethos. CMMI maturity levels, particularly Levels 3 and above, integrate V&V into defined processes; for instance, the Process and Product (PPQA) process area at Level 2 mandates objective evaluations like peer reviews, while higher levels emphasize quantitative management of V&V activities to achieve predictable quality outcomes. Note that CMMI V3.0 was released in 2023 with updated practice areas. Testing represents a primary QC component within this framework, focusing on execution-based defect detection. The integration of QA with V&V gained significant momentum with the introduction of the ISO 9000 series in 1987, which established international guidelines for quality management systems applicable to . This standard emphasized process-oriented through documentation, audits, and continuous improvement, influencing software practices by promoting standardized V&V activities to ensure conformance and . ISO 9000's framework encouraged organizations to embed V&V within broader QA strategies, laying the groundwork for subsequent models like CMMI and fostering a global shift toward proactive in .

Methods and Techniques

Formal Methods

Formal methods in software verification and validation employ mathematically rigorous techniques to specify, develop, analyze, and verify software and hardware systems, ensuring unambiguous and precise descriptions through formal mathematical models. These methods rely on a sound mathematical basis, typically provided by languages, to define system behaviors and properties without ambiguity. For instance, uses and predicate calculus to model system states and operations, enabling the construction of abstract specifications that can be refined step-by-step. Similarly, the (VDM) supports verification of step-wise refinement, including data refinement and operation decomposition, through its VDM-SL, which facilitates the formal description of system invariants and pre/post-conditions. Key techniques in formal methods include , theorem proving, and . exhaustively explores all possible states of a model to verify whether it satisfies specified , often expressed in ; the SPIN tool, for example, uses (LTL) to check concurrent software for like deadlock freedom and liveness. Theorem proving involves interactive or automated generation of mathematical proofs to establish correctness, with tools like Coq enabling the of software through dependent type theory and constructive proofs. approximates program semantics over abstract domains to detect errors such as overflows or dereferences, providing sound over-approximations of concrete behaviors while remaining decidable for practical analysis. Formal specification languages like support lightweight modeling and verification by allowing declarative descriptions of structural constraints and behaviors, which are then analyzed using SAT solvers for and generation. In safety-critical domains, such as and financial systems, reduce ambiguity in requirements and designs, leading to higher assurance levels; the Mondex electronic purse project, for instance, used to specify and refine the system's security properties, including value transfer integrity, resulting in formal proofs of correctness that contributed to its certification under ITSEC Level E6. Despite their strengths, formal methods face significant limitations, including high development costs due to the labor-intensive nature of specification and proof construction, as well as the need for specialized expertise in and formal tools. Scalability issues arise for large, complex systems, where state explosion in or proof complexity can render exhaustive verification impractical, often necessitating approximations or partial application. These challenges limit their widespread adoption beyond high-assurance contexts.

Informal Methods

Informal methods in software verification and validation emphasize human judgment, experience, and empirical observation over mathematical rigor, relying on practical techniques to identify defects and ensure quality. These approaches include static analysis methods such as desk checking, where developers individually code or designs for errors, and collaborative processes like walkthroughs, in which a simulates system execution to uncover issues through discussion. Unlike , informal techniques are accessible and cost-effective for most development projects, prioritizing early defect detection through peer involvement. Static informal methods also encompass code reviews and structured inspections, which systematically examine artifacts like requirements, designs, or against established criteria. A seminal example is the Fagan , developed in the 1970s, which involves planning, preparation, meetings, and follow-up to detect defects in a disciplined yet non-mathematical manner. This uses heuristics and checklists tailored to specific artifacts, such as logic flow or interface consistency, to guide reviewers and achieve high defect detection rates—empirically shown to identify up to 80% of errors early in development. Dynamic informal methods involve executing or simulating software behavior to observe outcomes empirically, including to model responses under various conditions, prototyping to validate user requirements through iterative builds, and where testers freely investigate the application to reveal unanticipated defects. These techniques complement static reviews by focusing on runtime behavior and user interaction, often without predefined scripts, allowing for flexible discovery in evolving projects. In agile environments, informal methods integrate seamlessly with iterative practices, such as testing, where code changes are frequently merged and automatically validated through builds and basic checks to catch integration defects early. Pair programming serves as another key informal verification approach, with two developers collaborating in real-time to review and refine code, reducing major defects by approximately 40% compared to in empirical industry studies. These practices leverage team dynamics and rapid feedback to maintain quality without halting progress. Empirical evidence underscores the effectiveness of informal methods, with studies indicating that peer reviews and inspections detect 60-70% of defects on average across software artifacts, often at lower cost than later-stage testing. For instance, aggregated data from multiple inspections show a median detection rate of around 60%, highlighting their role in preventing escapes to production. Informal methods thus provide a practical complement to formal approaches in hybrid verification strategies, balancing rigor with efficiency for diverse project needs.

Independent Verification and Validation

Historical Development

The fire on January 27, 1967, which resulted in the deaths of astronauts Virgil I. Grissom, Edward H. White II, and during a ground test, prompted to establish enhanced independent oversight mechanisms for safety-critical systems. In response, President directed Administrator to form an independent investigation committee, leading to the creation of the Aerospace Safety Advisory Panel (ASAP) on June 29, 1968, to provide ongoing independent safety reviews across 's programs. These early initiatives emphasized rigorous, detached evaluation processes to prevent oversights in complex engineering, laying foundational concepts for independent verification in emerging software-intensive systems as roles expanded in during the late 1960s. In the , the U.S. Department of Defense (DoD) advanced IV&V through standardized requirements for defense software. The DOD-STD-2167 standard, issued in 1985 and updated as DOD-STD-2167A in 1988, mandated independent activities separate from the developer's to ensure software met mission-critical needs in defense systems. This framework required contractors to demonstrate software correctness, quality, and compliance via independent reviews, influencing subsequent standards like in 1994, which integrated IV&V into broader life cycles for applications. NASA formalized its dedicated IV&V program in 1993, in response to recommendations from the National Research Council (1987) following the 1986 Space Shuttle Challenger accident, to address software reliability issues in safety-critical systems. Funded in 1992 with a $10 million appropriation to , the program became operational in 1993, initially focused on high-risk missions, and began applying IV&V to software for the and other projects, establishing the Independent Verification and Validation Facility in , to conduct impartial analyses. Concurrently, the (ESA) adopted independent software verification and validation (ISVV) practices in the 1990s for space missions, issuing guidelines in 1995 (ESA PSS-05-10) to ensure reliability in onboard software through detached evaluation processes. Following 2000, IV&V evolved to integrate with agile and methodologies, adapting traditional independent reviews to and delivery pipelines while maintaining separation from development teams. This shift enabled earlier defect detection in iterative environments, as seen in DoD and applications for safety-critical software. In the 2020s, AI-assisted IV&V has gained prominence, with increasing use of and for automated code analysis, , and predictive to enhance efficiency in verifying complex systems, such as those in the .

Organizational Applications

The NASA Independent Verification and Validation (IV&V) Facility, based in Fairmont, West Virginia, serves as a dedicated center for analyzing software in high-stakes missions, including the Mars Science Laboratory mission featuring the Curiosity rover. Established to provide evidence-based assurance, the facility conducts risk-based analysis by assessing software artifacts throughout the development lifecycle, prioritizing components with high failure potential through techniques such as static code analysis, dynamic testing, and simulation modeling. This approach has enabled the identification and mitigation of defects in rover software responsible for autonomous navigation and scientific instrument control, ensuring mission reliability in environments where failures could jeopardize multi-billion-dollar investments. The European Space Agency (ESA) applies Independent Software Verification and Validation (ISVV) to onboard software for satellite systems, emphasizing rigorous testing to build confidence in fault-tolerant operations. For projects like the Ariane launchers, ISVV involves independent review of test specifications, execution of non-nominal scenarios, and validation using dedicated facilities to simulate space conditions. Tools such as fault tree analysis are integrated to model potential failure propagations in attitude control and failure detection systems, a practice strengthened following the 1996 Ariane 5 incident to enhance overall software safety and prevent recurrence of guidance errors. In the U.S. Department of Defense (DoD), independent contractors conduct IV&V for weapon systems software, applying tailored analyses to verify compliance with operational requirements and reduce integration risks in networked defense platforms. The (FDA) mandates similar independent validation for software, focusing on and reliability to minimize patient harm. These applications have demonstrated substantial risk reduction, with FDA guidance noting decreased failure rates and fewer recalls through systematic V&V, while DoD reports highlight improved defect detection in mission-critical code. Organizational IV&V delivers key benefits through organizational separation from development teams, fostering objectivity in defect identification and early risk mitigation. Independence is categorized into full (separate entity with no developer influence) and partial (internal but isolated group) levels, where full independence maximizes unbiased assessment and has been linked to higher assurance in safety-critical domains. Metrics such as defect density and risk scores guide prioritization, enabling quantifiable improvements in software quality.

Process and Methodology

Planning and Management

Planning and management in software verification and validation (V&V) establish the foundational framework for ensuring that software systems meet their intended requirements and perform reliably throughout their lifecycle. This phase involves developing comprehensive V&V plans that outline the objectives, methods, and resources needed to systematically verify that the software is built correctly and validate that it fulfills user needs. According to IEEE Std 1012-2024, the V&V plan, often termed the Software Verification and Validation Plan (SVVP), must define the scope by specifying the software items to be verified and validated, along with the applicable life-cycle phases from concept to maintenance. The plan also establishes criteria for V&V activities, including minimum tasks such as , , and test planning, which can be tailored based on project integrity levels derived from risk assessments of consequence and likelihood. Schedules within the SVVP integrate V&V milestones with the overall project timeline, ensuring iterative execution where changes trigger re-planning of affected tasks. A risk-based approach is central to effective V&V planning, prioritizing activities on high-risk components to optimize and mitigate potential failures early. This involves conducting (FMEA), a structured method to identify potential failure modes, assess their severity, occurrence, and detectability, and calculate a priority number (RPN) to guide prioritization. In software contexts, FMEA supports V&V by focusing verification efforts on critical functions, such as safety-related modules in embedded systems, where undetected failures could lead to significant consequences. IEEE Std 1012 reinforces this by mapping V&V rigor to integrity levels, ensuring higher scrutiny for software with greater exposure, such as in or medical applications. Management of V&V requires clear definition of team roles, selection of appropriate tools, and seamless integration with the software development life cycle (SDLC). V&V engineers typically lead the execution of test plans, perform analyses, and report discrepancies, collaborating with developers and specialists to maintain independence while supporting iterative improvements. Tools like requirements traceability matrices (RTMs) are essential for managing linkages between requirements, design elements, and test cases, enabling impact analysis of changes and ensuring complete coverage during planning. Integration with SDLC occurs across phases, with V&V activities embedded from through deployment; for instance, verification reviews align with milestones, and validation tests occur post-integration to confirm system-level compliance. Success in V&V planning is measured through defined metrics, including coverage goals that quantify the extent to which requirements and code are tested—such as achieving 100% or branch coverage thresholds—and exit criteria that signal phase completion, like zero critical defects or met benchmarks. These metrics, often tailored per or IEEE guidelines, provide objective benchmarks for progress reviews and resource adjustments. In agile environments, such as Scrum, traditional V&V planning adapts to iterative cycles by incorporating lightweight, sprint-based V&V activities, where verification occurs continuously through automated testing and validation via acceptance criteria at sprint reviews. This approach maintains risk focus by updating FMEA during backlog refinement and using RTMs to track evolving requirements across sprints, ensuring adaptability without compromising thoroughness.

Requirements and Design Verification

Requirements verification ensures that the specified requirements are complete, consistent, unambiguous, and feasible before proceeding to design and implementation phases. This process involves traceability analysis, which maps requirements to their origins, such as stakeholder needs or higher-level specifications, using tools like a Requirements Traceability Matrix (RTM) to demonstrate bidirectional linkages and prevent . Consistency checks verify that requirements do not contradict each other, often through peer reviews or automated analysis in tools such as Engineering Requirements Management DOORS Next, which supports linking and querying requirements for conflicts. Completeness reviews assess whether all necessary requirements are captured, including functional, non-functional, and performance aspects, typically via checklists derived from standards like IEEE 830-1998. Design verification focuses on confirming that the proposed and artifacts align with the verified requirements and are technically sound. Architectural reviews, conducted formally with multidisciplinary stakeholders, evaluate high-level design decisions for , , and risk, often using structured methods like the (ATAM) to identify potential issues early. Simulation of UML models, such as state machines or sequence diagrams, allows for dynamic analysis of design behavior under various scenarios, enabling detection of timing or interaction flaws without physical ; tools like those based on UPPAAL integrate UML with timed automata for this purpose. Interface compatibility testing examines design elements for seamless integration, using static analysis to ensure protocols and data formats match across modules. Key techniques in these phases include prototyping to assess design feasibility, where rapid mockups or throwaway prototypes simulate user interactions and responses to validate assumptions and uncover gaps in requirements interpretation. Formal reviews, such as structured walkthroughs or inspections, target requirements by involving independent reviewers to probe vague language, pronouns, or implicit assumptions, reducing misinterpretation risks. (MBSE) enhances verification by creating digital twins of the using SysML, which supports integrated modeling of requirements, architecture, and behavior for automated consistency checks and simulation-based verification throughout the lifecycle. A common issue in requirements verification is , often manifesting as incomplete or inconsistent implementations later. Mitigation strategies include mandatory ambiguity checklists during reviews—covering lexical, syntactic, and semantic types—and iterative refinement with prototypes to clarify intent, potentially reducing rework costs that stem from 70-85% of errors traced to early requirements flaws. These approaches, when integrated into planning foundations, ensure pre-implementation artifacts are robust and aligned.

Implementation and Integration Verification

Implementation and integration verification focuses on ensuring that the software's code is correctly implemented and that integrated components interact as intended, primarily through static and dynamic techniques applied during the development and assembly phases. Code verification begins with static analysis, which examines without execution to identify defects such as code smells, potential bugs, and issues. Tools like automate this process by scanning for violations of coding standards and detecting issues like unused variables or overly complex methods, thereby improving code quality early in the implementation phase. Linting tools, such as LCLint, extend this by enforcing specifications and checking for inconsistencies in C programs, reducing errors through rule-based validation. Peer code reviews complement these automated methods by involving developers in manual inspection, which has been shown to effectively catch logical errors and enhance overall code reliability in distributed projects. Integration verification shifts attention to how modules interact, emphasizing interface testing to confirm that exchanges and dependencies function correctly across components. This often employs stubs and mocks to simulate unavailable modules, allowing isolated testing of interfaces without full assembly; for instance, mocking frameworks generate realistic behaviors to validate interactions in unit and integration contexts. Incremental integration strategies, such as top-down (starting from high-level modules and using stubs for lower ones) or bottom-up (beginning with low-level modules and employing drivers for higher ones), facilitate gradual verification and early defect isolation, with empirical studies indicating that top-down approaches often yield more reliable outcomes in complex s. Key defect detection during these phases targets common implementation flaws like buffer overflows and . Static analysis techniques detect buffer overflows by tracing data flows and identifying unsafe memory operations, such as array bounds violations in C code, preventing exploitable vulnerabilities. analysis tools, including those using value-flow tracking, identify unreleased allocations by monitoring heap usage patterns, enabling proactive fixes to avoid runtime performance degradation. To assess code verifiability, metrics like guide testing efforts by quantifying the number of independent paths in a program's . Defined by Thomas McCabe, it is calculated as: V(G)=E−N+2PV(G) = E - N + 2P where EE is the number of edges, NN the number of nodes, and PP the number of connected components (typically 1 for a single graph), helping prioritize modules with high complexity for thorough verification. In the 2020s, (SAST) has become integral to implementation verification, integrating security checks for vulnerabilities like injection flaws directly into code analysis pipelines, as standardized by guidelines to address cybersecurity in modern software development.

System Validation

System validation represents the culminating phase of software verification and validation, where the fully integrated system is evaluated to confirm it fulfills user needs and operates effectively in intended real-world environments. This process ensures the software not only aligns with specified requirements but also delivers value in practical deployment scenarios, mitigating risks of post-release failures. According to IEEE Std 1012-2024, system validation encompasses activities that demonstrate the software's suitability for its operational context, building on prior integration verification outputs. Key validation techniques include user acceptance testing (UAT), operational scenario simulations, and beta testing. UAT involves end-users executing predefined test cases in a production-like environment to verify the system meets business and expectations, serving as the final gate before deployment. Operational scenario simulations replicate real-world usage patterns, such as sequences or environmental stressors, to assess system behavior under dynamic conditions without full . Beta testing extends this by releasing limited versions to external users for feedback on functionality and performance in diverse settings, identifying issues not evident in controlled tests. These techniques collectively ensure the system's fitness for purpose by simulating end-user interactions and edge cases. End-to-end checks during system validation focus on holistic performance, including to evaluate response times and stability under peak usage, assessments to confirm intuitive interfaces and error handling, and compliance with non-functional requirements like reliability and security. frameworks analyze system throughput and scalability to prevent bottlenecks in production. evaluations often employ scenario-based methods to measure user task completion rates and satisfaction against non-functional criteria. Tools such as environment simulators mimic operational contexts for safe testing of complex interactions, while suites automate re-execution of prior tests to detect unintended impacts from final integrations. Success is determined by stakeholder sign-off, where key users and approvers review outcomes against acceptance criteria, culminating in validation reports that document test results, discrepancies resolved, and overall conformance. These reports provide traceability and evidence for deployment approval. A stark illustration of validation's stakes is the 2012 Knight Capital incident, where inadequate system validation of trading software led to a deployment error activating obsolete code, resulting in $460 million in losses within 45 minutes due to uncontrolled erroneous trades across 154 stocks. This failure underscored the need for rigorous end-to-end checks and simulations to avert catastrophic real-world impacts.

Standards and Regulations

Industry Standards

Several key industry standards provide frameworks for software verification and validation (V&V), ensuring consistency, , and quality across the software lifecycle. These standards emphasize processes for confirming that products meet specified requirements (verification) and fulfill intended use in operational environments (validation), often tailored to specific levels or domains. The IEEE Standard for System, Software, and Hardware (IEEE 1012-2024) defines a comprehensive V&V process applicable throughout the lifecycle of systems, software, and hardware, including development, , and reuse of components such as legacy systems or items. It introduces software integrity levels based on consequence and likelihood of failure, which determine the rigor of V&V activities, including documentation requirements like verification plans and reports to support and . This standard aligns V&V with broader system engineering practices, ensuring that selected work products conform to their specifications at each stage. ISO/IEC 25010:2023, part of the Systems and software Requirements and Evaluation (SQuaRE) series, establishes a quality model for software products and systems, defining nine characteristics—functional suitability, efficiency, compatibility, , reliability, , , portability, and —that guide validation efforts. These characteristics, along with sub-characteristics, enable stakeholders to specify and evaluate quality requirements relevant to validation, such as assessing reliability through and recoverability in operational contexts. The model supports both internal product quality evaluation during verification and external quality in use during validation, applicable to a wide range of software-intensive systems. The (CMMI) for Development, version 3.0, integrates V&V into its maturity levels to promote process improvement and predictable outcomes in . At Maturity Level 3 (Defined), organizations establish defined processes for verification, ensuring work products meet specified requirements through peer reviews and testing, while validation at this level confirms the product satisfies user needs in intended environments. Higher levels, such as Level 4 (Quantitatively Managed) and Level 5 (Optimizing), incorporate quantitative analysis and continuous improvement of V&V practices to enhance quality and reduce defects. CMMI appraisals help organizations benchmark their V&V maturity against these levels. In the automotive sector, :2018 addresses for road vehicles, specifying V&V requirements for electrical and electronic systems to mitigate risks from malfunctions. It defines Automotive Safety Integrity Levels (ASIL A-D) based on exposure, severity, and , which dictate the depth of V&V activities, including unit verification, integration testing, and system validation to confirm safety requirements. For software, Part 6 outlines methods like static analysis and for verification, ensuring compliance supports overall vehicle safety goals without mandating specific tools. The ISO/IEC/IEEE 29119 series, initiated in 2013 and updated through 2022, provides an international framework for software testing as a core component of V&V. Part 1 (2022) outlines concepts and terminology, while Part 2 (2021) details test processes for planning, management, monitoring, and control across organizational, project, and technical levels. Subsequent parts cover documentation (Part 3), techniques (Part 4), and advanced methods like (Part 5, 2024), enabling consistent application in diverse projects to verify requirements and validate system behavior. This series promotes interoperability and best practices in testing to achieve reliable software outcomes.

Regulatory Frameworks

Regulatory frameworks for software verification and validation (V&V) impose mandatory compliance requirements in high-stakes industries, where non-adherence can result in severe penalties, including fines, product recalls, or operational shutdowns. These regulations ensure that software systems meet rigorous , reliability, and performance standards through structured V&V processes, often mandating independent oversight and documentation. In sectors like healthcare, , nuclear energy, and , regulators enforce these frameworks to protect public and privacy, distinguishing them from voluntary industry standards by their legal enforceability. In the United States, the (FDA) regulates electronic records and signatures under 21 CFR Part 11, which applies to software systems used in manufacturing and . This regulation requires validation of software to demonstrate that electronic records are trustworthy, reliable, and accurate, equivalent to paper records, including controls for access, audit trails, and signature integrity to prevent unauthorized alterations. Compliance involves lifecycle V&V, such as installation qualification, operational qualification, and performance qualification, with enforcement through FDA inspections that can lead to warning letters or injunctions for deficiencies. For aviation software, the (FAA) mandates adherence to , titled "Software Considerations in Airborne Systems and Equipment ," which outlines V&V objectives tailored to five software levels (A through E) based on failure severity—from Level A for catastrophic risks requiring the highest rigor, including exhaustive testing and , to for no safety impact with minimal objectives. This framework requires planning, development, verification (e.g., reviews, analyses, testing), and validation processes, with independent verification for higher levels, enforced via FAA reviews that can delay aircraft approvals or ground fleets if unmet. These requirements align briefly with broader standards like IEEE 1012 for V&V planning. In the nuclear sector, the U.S. (NRC) enforces independent V&V through Regulatory Guide 1.168 for digital computer software in systems, such as reactor control software, to ensure reliability and prevent failures that could lead to radiological releases. The guide specifies V&V activities including , design reviews, code inspections, testing (unit, integration, system), and , with levels based on significance; non-compliance can trigger license revocation or plant shutdowns during NRC audits. The European Union's Medical Device Regulation (MDR) 2017/745 establishes comprehensive V&V requirements for software in medical devices, classified by risk under Annex VIII (e.g., Class III for high-risk software driving life-sustaining functions). Annex I, Section 17.2 mandates state-of-the-art incorporating , cybersecurity, and V&V throughout the lifecycle, with technical documentation (Annex II) detailing validation evidence from simulated and actual environments; notified bodies verify conformity via audits and testing (Annex IX). Post-market surveillance under Article 83 requires proactive monitoring, incident reporting, and updates to clinical evaluations and risk assessments, with periodic safety update reports for higher classes and penalties laid down by Member States in accordance with Article 113, ensuring they are effective, proportionate, and dissuasive. Additionally, the EU's (GDPR) (Regulation (EU) 2016/679) influences software V&V through Article 25, requiring data protection by design and by default, which mandates integrating privacy features into from inception, including verification of data minimization, pseudonymization, and access controls during development and testing. This implies V&V processes to assess compliance with security measures (Article 32) and conduct data protection impact assessments for high-risk processing (Article 35), with enforcement by data protection authorities imposing fines up to €20 million or 4% of annual global turnover for inadequate privacy validation in software handling .

References

  1. https://standards.ieee.org/ieee/1012/7324/
  2. https://ieeecs-media.computer.org/media/education/sevocab/24765-2017-preview.pdf
  3. https://ieeexplore.ieee.org/document/159342
  4. https://standards.ieee.org/standard/1012-2004.html
  5. https://ieeexplore.ieee.org/document/8055462
  6. https://standards.ieee.org/ieee/1012/1468/
  7. https://www.iso.org/standard/81291.html
  8. https://standards.ieee.org/ieee/29119-2/7498/
  9. https://link.springer.com/article/10.1007/s11219-023-09659-9
  10. https://www.bullseye.com/coverage.html
  11. https://www.geeksforgeeks.org/software-testing/software-testing-defect-density/
  12. https://testrigor.com/blog/the-history-of-test-automation/
  13. https://www.sei.cmu.edu/documents/1533/1987_007_001_15485.pdf
  14. https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication500-165.pdf
  15. https://www.sei.cmu.edu/documents/3379/2004_017_001_24372.pdf
  16. https://cmmiinstitute.com/cmmi
  17. https://acqnotes.com/acqnote/careerfields/iso-9000-quality-management-series
  18. https://www.eng.auburn.edu/~yilmaz/comp7730/FM.pdf
  19. https://users.ece.cmu.edu/~koopman/des_s99/formal_methods/
  20. https://esdalab.ece.uop.gr/pdf/17507934Chapt1_final.pdf
  21. https://staffwww.dcs.shef.ac.uk/people/K.Bogdanov/com6851/commentarybody.pdf
  22. https://www.cs.tufts.edu/comp/150FP/archive/gerard-holzmann/ieee97.pdf
  23. https://www.lri.fr/~paulin/LASER/course-notes.pdf
  24. https://pcousot.github.io/talks/CousotSlides_FEmSys01_1-1.pdf
  25. https://people.csail.mit.edu/dnj/publications/alloy.pdf
  26. https://fm.csl.sri.com/AFM06/papers/2-George.pdf
  27. https://link.springer.com/article/10.1007/s00165-007-0060-5
  28. https://www.kneuper.de/English/Publications/limits-formal-methods.pdf
  29. https://link.springer.com/article/10.1007/BF01211297
  30. https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nbsspecialpublication500-75.pdf
  31. https://ieeexplore.ieee.org/document/5387093
  32. https://dl.acm.org/doi/pdf/10.1145/224401.224456
  33. https://framework.scaledagile.com/continuous-integration
  34. https://dl.acm.org/doi/10.1145/1159733.1159749
  35. https://www.cs.umd.edu/~basili/publications/journals/J81.pdf
  36. https://www.nasa.gov/history/55-years-ago-the-apollo-1-fire-and-its-aftermath/
  37. https://www.nasa.gov/news-release/nasa-ivv-program-celebrates-30th-anniversary/
  38. https://oig.nasa.gov/docs/IG-14-024.pdf
  39. https://www.scribd.com/document/494019509/ESAISVVGuideRev1-0cNov2005
  40. https://www.wvnews.com/news/wvnews/nasa-iv-v-facility-increases-ai-use-to-enhance-software-testing-boost-efficiency/article_f618c3a7-a9cc-491f-a93d-4753715f2b2e.html
  41. https://www.nasa.gov/centers-and-facilities/nesc/understanding-risk-artificial-intelligence-and-improving-software-quality/
  42. https://www.nasa.gov/wp-content/uploads/2016/10/robotics_mission_support_-_forquer.pdf
  43. https://www.nasa.gov/ivv-services/
  44. https://www.nasa.gov/wp-content/uploads/2016/10/585642main_esafswtestingindependentverification.pdf
  45. https://ecss.nl/wp-content/uploads/2023/09/ECSS-Q10-Q20-training-at-ESTEC-2023-Lorenzo.pdf
  46. https://www.esa.int/esapub/bulletin/bulletin123/bulletin123.pdf
  47. https://apps.dtic.mil/sti/tr/pdf/ADA462428.pdf
  48. https://www.fda.gov/media/73141/download
  49. https://www.fda.gov/media/71794/download
  50. https://ndia.dtic.mil/wp-content/uploads/2012/systemtutorial/14604.pdf
  51. https://ntrs.nasa.gov/api/citations/19950020066/downloads/19950020066.pdf
  52. https://nvlpubs.nist.gov/nistpubs/Legacy/FIPS/fipspub132.pdf
  53. https://asq.org/quality-resources/fmea
  54. https://www.jamasoftware.com/requirements-management-guide/meeting-regulatory-compliance-and-industry-standards/fmea/
  55. https://ntrs.nasa.gov/api/citations/19970006981/downloads/19970006981.pdf
  56. https://www.jamasoftware.com/requirements-management-guide/requirements-traceability/how-to-create-and-use-a-requirements-traceability-matrix/
  57. https://visuresolutions.com/alm-guide/verification-and-validation-in-software-testing/
  58. https://swehb.nasa.gov/display/SWEHBVC/7.9%2B-%2BEntrance%2Band%2BExit%2BCriteria
  59. https://link.springer.com/article/10.1007/s11219-022-09593-2
  60. https://www.sei.cmu.edu/documents/2176/2009_004_001_15083.pdf
  61. https://link.springer.com/article/10.1007/s10009-005-0205-x
  62. https://www.geeksforgeeks.org/software-engineering/software-engineering-prototyping-model/
  63. https://www.modernanalyst.com/Resources/Articles/tabid/115/ID/2216/Requirements-Reviews--When-You-Want-Another-Opinion-Part-2.aspx
  64. https://www.sei.cmu.edu/blog/introduction-model-based-systems-engineering-mbse/
  65. https://www.jot.fm/issues/issue_2007_01/column2/
  66. https://www.jamasoftware.com/blog/when-bad-requirements-happen-to-nice-people/
  67. https://www.nasa.gov/reference/5-3-product-verification/
  68. https://ieeexplore.ieee.org/document/10590233/
  69. https://dl.acm.org/doi/10.1145/195274.195297
  70. https://dl.acm.org/doi/10.1145/3131151.3131161
  71. https://dl.acm.org/doi/10.1145/3617171
  72. https://ieeexplore.ieee.org/document/245736/
  73. https://ieeexplore.ieee.org/document/5521566/
  74. https://dl.acm.org/doi/10.1145/1250734.1250789
  75. https://ieeexplore.ieee.org/document/1702388/
  76. https://owasp.org/www-community/Source_Code_Analysis_Tools
  77. https://ieeexplore.ieee.org/document/6204026
  78. https://link.springer.com/article/10.1023/A:1018503800709
  79. https://www.researchgate.net/publication/312559693_A_tutorial_on_the_operational_validation_of_simulation_models
  80. https://ieeexplore.ieee.org/document/7515456/
  81. https://www.researchgate.net/publication/3188490_Scenario-Based_Assessment_of_Nonfunctional_Requirements
  82. https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf
  83. https://standards.ieee.org/ieee/1012/5609/
  84. https://www.iso.org/standard/78176.html
  85. https://cmmiinstitute.com/learning/appraisals/levels
  86. https://www.wibas.com/cmmi/verification-ver-cmmi-dev
  87. https://www.tuvsud.com/en-us/services/functional-safety/iso-26262-automotive
  88. https://www.ecfr.gov/current/title-21/chapter-I/subchapter-A/part-11
  89. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/part-11-electronic-records-electronic-signatures-scope-and-application
  90. https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC_20-115D.pdf
  91. https://www.nrc.gov/docs/ML1307/ML13073A210.pdf
  92. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32017R0745
  93. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679
Add your contribution
Related Hubs
User Avatar
No comments yet.