A new way to find multiple system weaknesses at once
Researchers developed a two-layer method to estimate how many tests are needed to discover various safety violations.
Engineers design safety-critical systems like network protocols. These systems depend on environmental parameters that are often uncertain. A single setting might cause several different safety or performance problems at the same time.
Researchers use parametric Markov decision processes (pMDPs) to model these systems. In a pMDP, the chance of moving between states is a symbolic function of certain parameters. This allows engineers to study how different settings affect the system.
The researchers wanted to know how many random tests are needed to find many different vulnerabilities. They studied how many samples are required before the expected number of undiscovered requirements is small. This is called a random-testing budget.
The solution uses two layers. The first layer is the scenario-approximation layer. It uses a model checker to create a proxy for each requirement. This proxy helps identify conservative vulnerability regions.
The second layer is the testing layer. It treats the search for vulnerabilities like a coupon-collector problem with overlapping types. It uses the size and overlap of regions to decide when to stop testing.
A composition theorem combines the errors from both layers. It bounds the expected non-discovery score by adding the scenario approximation error and the testing risk. This provides an end-to-end bound with one confidence parameter.
The researchers used requirement-wise semantics for their work. This means each requirement is analyzed under its own worst-case scheduler. A system is considered vulnerable if any scheduler can make at least one requirement fail.
The team tested their method on ten benchmark families. These tests included up to monitored requirements. They used empirical surrogates and independent confidence bounds to evaluate their certificates.
Key facts
- The method uses a two-layered approach to find vulnerabilities in pMDPs.
- A composition theorem bounds the total risk from both the approximation and testing layers.
- The approach works with up to requirements in benchmark tests.
- The testing layer uses a stopping certificate based on marginal region masses.