The central conclusions of this effort are that model-based methods for procurement and systems integration are effective for improving systems engineering outcomes; model-based methods identified many of the risks that were manifested in the lab. Most of the time or capability-loss issues encountered in the lab were related either to software configuration or network configuration. We demonstrated a workable path to successful application of MBSE and ACVIP for major embedded systems integration activities. We observed successful adoption and application of MBSE and ACVIP by FARA performers and successfully applied performer- generated ACVIP artifacts. Galois’s role in OSVD allowed us to contribute model-based risk virtual integration assessments to help reduce risk for physical integration efforts. By applying ACVIP to OSVD, we were able to correctly predict integration risks that were later realized in lab activities (specifically network configuration errors). In this case, we found that the best predictor of problems in physical asset integration was ambiguity rather than incompatibility, in design artifacts. The key remaining challenges for broad realization of the benefits of MBSE and ACVIP are dealing with culture change and scalable deployment of digital engineering environments across organizational boundaries.

Today’s most powerful machine learning approaches are typically designed to train stateless architectures with predefined layers and differentiable activation functions. While these approaches have led to unprecedented successes in areas such as natural language processing and image recognition, the trained models are also susceptible to making mistakes that a human would not. In this paper, we take the view that true intelligence may require the ability of a machine learning model to manage internal state, but that we have not yet discovered the most effective algorithms for training such models. We further postulate that such algorithms might not necessarily be based on gradient descent over a deep architecture, but rather, might work best with an architecture that has discrete activations and few initial topological constraints (such as multiple predefined layers). We present one attempt in our ongoing efforts to design such a training algorithm, applied to an architecture with binary activations and only a single matrix of weights, and show that it is able to form useful representations of natural language text, but is also limited in its ability to leverage large quantities of training data. We then provide ideas for improving the algorithm and for designing other training algorithms for similar architectures. Finally, we discuss potential benefits that could be gained if an effective training algorithm is found, and suggest experiments for evaluating whether these benefits exist in practice.

The 2018 release of the DoD’s Digital Engineering (DE) strategy and the success of applying DE methods in the mechanical and electrical engineering domains motivate application of DE methods in other product development workflows, such as systems and/or software engineering. The expected benefits of this are improved communication and traceability with reduced rework and risk. Organizations have demonstrated advantages of DE methods many times over by using model-based design and analysis methods, such as Finite Element Analysis (FEA) or SPICE (Simulation Program with Integrated Circuit Emphasis), to conduct detailed evaluations earlier in the process (i.e., shifting left). However, other domains such as embedded computing resources for cyber physical systems (CPS) have not yet effectively demonstrated how to incorporate relevant DE methods into their development workflows. Although there is broad support for SysML and there has been significant advancement in specific tools, e.g., MathWorks ®, ANSYS®, and Dassault tool offerings, and standards like Modelica and AADL, the DE benefits to CPS engineering have not been broadly realized. In this paper, we will explore why CPS developers have been slow to embrace DE, how DE methods should be tailored to achieve their stakeholders’ goals, and how to measure the effectiveness of DE-enabled workflows.

In support of the US Army Mission System Architecture Demonstration, Adventium Labs conducted a series of interviews and demonstrations to determine requirements, best practices, and available tool capabilities for building and maintaining an Authoritative Source of Truth (ASoT). An ASoT is a capability that gives definitive answers to queries about a target collection of systems. An ASoT should make information discoverable, enable controlled information sharing, and maintain traceability across time and organizations. The challenges to establishing an ASoT include limited standards adoption by tool vendors, entrenched workflows, and data rights management needs. The systems engineering community can overcome these challenges by keeping ASoT needs at the forefront when planning engineering activities, investing in open and flexible standards for information sharing, and leveraging emerging connectivity tools and model-based systems engineering methods.

In this paper, we demonstrate that the Architecture-Centric Virtual Integration Process (ACVIP) provides value for military aircraft airworthiness qualification. Military aircraft airworthiness criteria describe aviation airworthiness processes and the criteria, standards, and methods of compliance necessary for airworthiness assessment of manned and unmanned military aircraft systems. The U.S. Army Military Airworthiness Certification Criteria (AMACC), for example, includes elements from many existing civilian standards and is used to define airworthiness requirements for existing and new acquisition programs. Software safety of complex systems is assured by compliance with formal development processes and testing of essential elements. The AMACC also allows for verification by analysis to detect defects in the evolving software design. Going forward, the U.S. Department of Defense’s (DoD) Digital Engineering Strategy will improve aircraft requirements, design, and development through model-based engineering. ACVIP provides the foundation needed for effective model-based verification by analysis.

In this Work in Progress report, we describe ongoing research on our DECIMAL project, addressing the problem of modeling computational mechanisms at sufficient fidelity to reason about the execution semantics of programs across abstraction boundaries. The automata-based formalism that we have developed is specifically constructed to support reasoning about timed behavior over compositions of multiple component automata, modeling different parts of the system under study. We show how we use composition to model a wide variety of constructs, including synchronization across abstraction boundaries, communicating asynchronous processes, and specifying programs that can be generalized across different architectures and over localized variations in the program specification.

This document reviews the Architecture Centric Virtual Integration Process (ACVIP) analysis tools developed by Adventium Labs with respect to established Army tool airworthiness certification standards, namely the Army Military Airworthiness Certification Criteria (AMACC) and RTCA/DO-330 “Software Tool Qualification Considerations.”
Airworthiness qualification of a modeling/analysis tool provides a level of assurance that the tool has the necessary integrity to accomplish its specified role on the program. Decisions regarding the airworthiness qualification process are part of the overall airworthiness strategy negotiated between the contractor and the Government. Consequently, how a particular tool is used within the contractor’s design process determines the qualification process for the tool if qualification for the tool is required. For example, a tool may be qualified for DO-330 tool qualification level (TQL) 1 on one program and may not require qualification on a different program.

The following sections help identify potential roles ACVIP tools play in the airworthiness qualification process. The ACVIP tools overall fit within a larger trend towards the application of model-based system engineering (MBSE) techniques on Department of Defense (DoD) embedded systems development programs for the purposes of risk reduction. The full impact of MBSE tools on the airworthiness qualification process to date (circa September 2021) is still relatively unrealized. Potential benefits of MBSE in the qualification process itself (independent of the airworthiness credit required) include:

Improved acquisition strategy due to enhanced communications between organizations in distributed development,
Reduced duplicative effort,
Minimized time for review cycles,
Improved detail to the overall qualification picture,
A standardized formal definition for real time systems of architecture elements, with classifier semantics, properties, execution dynamics, and runtime services.

Audience: This paper is for decision makers at Program Executive Office (PEO) Aviation and in Army Aviation Program Management Offices (PMOs). This paper assumes reader familiarity with the five principles of Modular Open Systems Approach (MOSA) and Architecture Centric Virtual Integration Process (ACVIP).
Takeaway: Readers should come away from this paper understanding how ACVIP provides capabilities to reduce risk for the Army MOSA.

Executive Summary: MOSA is a National Defense Authorization Act (NDAA) mandated systems engineering methodology.   The U.S. Army strategy  for MOSA is refined by the “MOSA Implementation Guide,” “MOSA Reference Framework,” and “Army MOSA Initial Capability Refinement Document (ICRD).” ACVIP is a process for reducing risk in development and procurement of cyber-physical systems. The ACVIP Handbooks define ACVIP. The Army MOSA strategy calls for use of standards for modularization, well-defined interfaces for component integration, integration risk reduction, and verification and validation throughout the lifecycle. ACVIP provides mechanisms for these objectives that have been extensively validated through Army Science and Technology (S&T) activities.

In this study, three experiments examined the impact of stimulus similarity on the benefits of spacing and interleaving for long-term memory. Two laboratory-based experiments (Experiments 1 and 2) and one classroom-based experiment (Experiment 3) were conducted. In Experiment 1, an advantage for interleaving relative to massing stimuli during encoding was observed as a greater proportion of correct responses on a categorization test for birds and paintings. This advantage was significantly greater when the stimuli were similar (e.g., interleaving different bird categories) rather than dissimilar (e.g., interleaving bird and painting categories). In Experiment 2, no advantage of interleaving relative to massing stimuli was observed in either the proportion of correct responses or response times on a categorization test for abstract visual stimuli. In Experiment 3 no significant differences between massed and interleaved study conditions were observed on a categorization test for textual materials. Although the results from this study are preliminary, the pattern of results in Experiment 1 suggests that interleaving may be most beneficial when the interleaved stimuli are similar rather than dissimilar.

CellEnergy is an iOS educational app developed to teach the basics of photosynthesis and cellular respiration for high school life-science courses. Through our many exploratory interviews with biology teachers, photosynthesis was identified as a particularly difficult subject area to engage students with, both because of its abstract nature and the invisible cellular processes it involves.
CellEnergy exemplifies a hybrid approach to learning apps, in which the focus is primarily on learning outcomes with gamelike elements incorporated to make complex processes visible in an engaging and a playful way. The activities in CellEnergy are based on multiple evidence-based learning practices, such as retrieval practice, spaced learning, and immediate feedback. Virtual labs provide inquiry-based learning and reinforcement of science practices in the context of photosynthesis.

Our project culminated in a cluster randomized controlled trial that included more than 600 students in 22 high school biology classrooms. We demonstrated that using CellEnergy resulted in significantly greater learning gains in both photosynthesis concept knowledge and science practices knowledge compared to standard instruction.