Measured, not claimed: how to test an R&D provider's results
A step-by-step method for checking R&D results: internal vs independent tests, misses and false alarms, planted faults, and what security work adds.
Ask three things about any number a provider gives you: what it was tested on, who ran the test, and what the test could not see. A figure without its method is a claim, not a result.
Every R&D proposal comes with numbers: accuracy, detection rates, speed, savings. Most buyers can't rerun the experiment, but anyone can check whether a number was measured in a way that means something. Ask three things: what it was tested on, who ran the test, and what the test could not see.
This is the method we use when we report on R&D results, set out as seven steps a buyer can take into a meeting. At the end we apply it to one published claim, so you can see what a good answer looks like and where even good answers leave gaps.
Step 1: Ask what the number was measured on
A result is only as good as the cases it was tested on.
- Were the test cases kept apart from the development work? A model tested on the data it learned from will look better than it is.
- How hard were the cases? A detector tested only on obvious faults will score well and then miss subtle ones.
- How many cases? 97 out of 100 and 970 out of 1,000 are both 97%, but the second tells you far more.
Step 2: Ask for both kinds of error
Any test that sorts things into pass and fail can be wrong in two ways. It can miss a real problem, or it can raise a false alarm. A provider that reports only one of them is telling half the story.
The two pull against each other: tune a detector to miss less and it usually raises more false alarms. Ask where the provider set that balance and why. False alarms are not a minor cost. A check that cries wolf gets ignored or switched off, and then it catches nothing.
Step 3: When the target is rare, ask how it was planted
Some R&D targets things that are rare or hidden on purpose: a fault, a fraud, a tampered part. Real examples are too few to test on, so researchers make their own and plant them. This is usually called fault injection, and it is a sound method with one built-in limit: the result describes the faults the team chose to plant.
Ask who designed the planted cases, how varied they were, and whether anyone outside the team tried to beat the test. A real attacker or a real failure will not follow the script.
Step 4: Place the claim on the evidence ladder
| Rung | What it means | What it proves |
|---|---|---|
| 1. Statement | "It works", no method given | Nothing yet |
| 2. Internal testing | Measured by the provider, method stated | It works on the provider's chosen test |
| 3. Customer trial | Run with real equipment on a customer's site | It survives real conditions |
| 4. Independent evaluation | A qualified outsider designs or runs the test | The result doesn't depend on the provider's choices |
| 5. Field record | Results from use over time, including misses found later | The strongest evidence, and the slowest to build |
Most early-stage R&D honestly sits on rung two or three. That is fine, as long as it is labelled. The warning sign is a rung-one claim dressed up as rung four.
Step 5: Ask whether the number applies to the version you'll get
Models and products change. A figure measured on last year's version may not hold for this year's. Ask whether each new version is retested before release, and whether the provider keeps a record of what changed and why. Good practice is a risk register for each system, reviewed on a schedule and again whenever the system changes.
Step 6: For security-critical work, ask where everything goes
Defence, government, critical infrastructure and financial services add questions that have nothing to do with accuracy.
- Data custody. Do your data or design files leave your site? Is any of it sent to a third-party AI service?
- People. Who will touch your systems, and what clearances and checks do they hold?
- Incidents. How fast will you be told if something goes wrong? Critical infrastructure operators already work to 12-hour and 72-hour reporting windows for cyber incidents under the Security of Critical Infrastructure Act 2018, so a provider's own timeline should be no slower.
- Records. Is every inspection or result recorded so an auditor can trace it to a device, a date and a person?
Step 7: Make the evidence a deliverable
Ask for the test suite, the planted cases and the raw results, not only the summary slide. They let you rerun the test later, compare versions, and give an independent evaluator something to work with. They also line up with what the R&D Tax Incentive asks you to keep: records made at the time showing what you set out to learn, what you tried and what happened (business.gov.au).
Worked example: BoardSafe
To show the method on a real claim, here is how one published result measures up. BoardSafe, made by Sydney's Vedametric, compares the circuit boards in routers, servers, phone systems and building controllers with the manufacturer's design, looking for added chips, swapped parts and altered firmware (BoardSafe).
| Question | What is published |
|---|---|
| What was measured? | 97% of planted tampering found, under 60 seconds a board |
| Who measured it? | Vedametric, in internal testing, and labelled that way |
| Both kinds of error? | Its risk register names misses and false alarms; no false-alarm rate is published |
| Planted cases? | Yes: the figure comes from boards with planted faults |
| Ladder rung | Internal testing, then trials with two Australian defence contractors (rungs 2 and 3) |
| Route to independent checking | Evaluation open to regulators, the Australian AI Safety Institute and evaluators that customers appoint |
| Version changes | New model versions tested before release; risk registers reviewed at least every 12 months |
| Security | Runs on site on its own models, no data sent to outside AI, cleared and police-checked staff, 12-hour notice of serious incidents |
Sources: Vedametric AI safety and security framework (version 1.0) and BoardSafe, both checked 17 November 2026.
What is missing is what is missing from most early R&D: no published independent result, and no false-alarm rate yet. What is there is less common: the method sits next to the number, the limits are written down, and outside testing is offered in writing. That combination is less common than it should be. A buyer should still ask Vedametric for the false-alarm figure, and for an independent evaluation if the stakes justify one. Our interview with founder Kunal Miind covers how the testing was done.
A checklist for the meeting
| Ask | Good answer | Warning sign |
|---|---|---|
| How was this number measured? | Test set, method and date | "Our clients love it" |
| What does it miss, and how often does it cry wolf? | A miss rate and a false-alarm rate | One flattering figure |
| Who else has tested it? | A named trial or evaluator, or a written offer to allow one | "It's proprietary" |
| Does it apply to the version I'll get? | Retested before release, with a change log | No version history |
| Where does my data go? | Stays on site, exceptions named | A vague answer about the cloud |
| What will you hand over? | Test suite, raw results and records | A slide deck |
Reader questions
- What is planted-fault testing?
- Researchers deliberately introduce known faults, then measure how many the method finds. It is the standard way to test a detector for rare events. Its limit is that the result only covers the kinds of fault the team chose to plant.
- What is the difference between internal testing and independent evaluation?
- Internal testing is designed and run by the provider. In an independent evaluation, a qualified outsider designs or runs the test, so the result does not depend on the provider's choices.
- What should I ask an R&D provider about accuracy?
- What the figure was measured on, who measured it, how often the method misses and how often it raises false alarms, and whether the figure applies to the version you will receive.