How we check a simulation against datasheets, and what we learned

Why we grade our own work

A simulation is a claim that a computer can stand in for a lab. Before anyone relies on that claim, the computer’s answers should be compared with something it did not produce. For a cooling fluid, the best comparison is a number a manufacturer has measured and published.

We have not mixed a FluxZero fluid yet. So the only honest test we can run today is to point our methods at fluids other people have measured, and to ask how far off we are. This post reports three such tests. One held. Two missed: one on every fluid, the other on all but one prediction. We think the misses are the more useful result.

A boiling solver against an exact answer

Our first test is the cleanest, and its grade is published. A standard textbook problem describes a liquid turning to vapour, with the boundary between them moving over time. It has a closed-form answer, so the correct position of the boundary is known at every moment. The solver and the problem both come from the solver’s authors, not from us.

We ran the solver and compared its boundary position with the exact answer at every recorded time after a short start-up period. The result:

• The worst difference was 1.43%.
• The pass bar was 5%, written down before the comparison was computed.
• Across 6,111 comparison points the typical difference was 1.08%.

That is a good result, and it is worth saying what it covers. The published extract of the grade shows that the problem is one-dimensional and that our own blend runs reuse the same setup with a constant surface tension. So the test shows the solver reproduces this one-dimensional evaporation problem. It says nothing about the flow that surface-tension differences would drive, and nothing about any FluxZero fluid.

The check that could have failed

A useful test is one that can fail. The pass bar was fixed before the comparison was computed, so the test could have failed; the solver output it graded already existed, and the problem is its authors’ own. We say that plainly because a pass only counts if a fail was possible.

Surface tension of three commercial fluids

The second test asked our molecular simulation for the surface tension of three commercial fluids: a hydrofluoro-ether sold for cooling, a perfluorinated liquid sold for cooling and testing, and a hydrofluoro-olefin refrigerant fluid. We compared each with the maker’s published value. The band for a pass was within ±15%, and it was written down before the runs.

All three came out high.

• The hydrofluoro-ether ran 36% above its published value.
• The perfluorinated liquid ran 33% above.
• The hydrofluoro-olefin ran 41% above.

For example, the hydrofluoro-olefin is published at 13.0 millinewtons per metre and our simulation gave 18.4.

What the control told us

The same experiment included a control: pentane, which our pipeline had simulated before; the control checks that the method repeats itself, not that it matches measurement. Pentane also missed its bar under the averaging method chosen in advance. A second averaging method, recorded in the same file, passes the control, yet still leaves all three fluids outside the band.

So the problem is the method, not only the fluids. The graded runs are published, including the control. After the runs we also found that our simulated liquids were between 26% and 59% less dense than the published densities. That is a likely cause of the high surface tension. It is a likely cause, not a proven one, because we found it after the fact and have not yet fixed it and re-run.

Our earlier predictions against datasheets

The third test is broader. We took 9 of our earlier predictions, for fluids we do not own, and graded them against manufacturers’ datasheets. Only 1 fell within its bar. It was the dielectric constant of the perfluorinated liquid: we estimated 2 and the maker publishes 1.75.

One large miss was a hydrofluoro-olefin fuel:

• Our estimator put its dielectric constant at 2.
• The maker publishes 32.

That is a different kind of fluid from the one we assumed. It matters because the dielectric constant is one of the properties, with dielectric strength, that decide whether a liquid can sit in direct contact with electronics.

These grades were computed after the predictions were made, and the bars were chosen after the published values were read. They show how far our methods sit from manufacturer data. They are not tests written down in advance, and we label them that way.

Every row is in the published grades file, with the manufacturer’s value beside our prediction and the method that produced it.

What we learned

Three lessons

First, a solver can be right while the physics we feed it is wrong. The boiling solver matched the exact problem, and our property estimates still missed badly. Those are separate questions, and a single success on one should not be read as success on the other.

Second, a failed control is information. It told us to doubt the method before we doubted any fluid.

Third, publishing the misses is the point. A reader can open each file and see the same numbers we see.