Multi-Agent Workflow for Black Box Test Generation for Dynamic Simulation Models

The Functional Mock-up Interface (FMI) specifies the standardized format for model exchange and co-simulation as Functional Mock-up Units (FMUs). An FMU package contains an interface description of the unit, binaries of the model, documentation, and optionally the source code. However, testing FMU-based models is difficult due to limited access to internal implementations and the frequent absence of explicit test oracles. Metamorphic testing is an approach which provides an alternative by defining and evaluating relationships between outputs across multiple executions of a model, but generating valid metamorphic relations (MRs) from specifications is predominantly manual and error-prone.

This thesis presents AgenticMeta, an LLM-powered multi-agent workflow to facilitate specification-based metamorphic testing of FMU simulation models. The workflow takes functional specifications and model interface descriptions as input and uses specialized agents to extract test requirements, generate and refine MRs into Given-WhenThen form, create executable metamorphic test cases, validate them, and execute them against the FMU. The implementation is built in Python using LangGraph to orchestrate the agent workflow, while OpenAI models are used to support language-based reasoning operations.

The approach is evaluated on an FMU Lube Oil Cooling (LOC) system on multiple independent runs. The results show that AgenticMeta can execute the complete workflow reliably and generate meaningful specification linked MRs and test cases. Based on evaluation of experimental results the workflow achieved a requirement coverage of 60.47%, a test-case pass rate of 74.10%, and a mutation score of 0.56. These findings provide feasibility for using multi-agent LLM workflow to automate substantial parts of specification-based metamorphic testing for FMU systems and reduce manual effort in test design, while still leaving room for improvement in coverage, oracle quality, and fault-detection capability.