Ground Truth for Simulation

Blog Article
Date
November 16, 2026
+

Ground truth for simulation: tuning physically grounded Digital Twins against the real object

A robot can't learn to handle a cup that only looks like one. In simulation, that cup also needs the right weight, grip and give. We have what most 3D libraries lack: the real object. That is our ground truth. Our 3D scanners digitize the object and creates the physically grounded and SimReady 3D Digital Twin of it. Then we test it, measure how it behaves and tune its Digital Twin until simulation matches reality.

Why looking right isn't enough

A robot learning to grasp, push or catch an object in simulation learns from the physics it is given. If a simulated object slides too easily, bounces too high or deforms too little, the policy learns from behavior that doesn't exist in the real world. That difference between simulated and real behavior is the Real2Sim gap.

Some physical properties can be measured directly: geometry, PBR materials and mass. Others, such as friction, restitution and deformation, are more difficult to capture and measure. They depend on the material, the surface and the contact partner, and they are interpreted differently by each physics engine. A friction value that works well in one simulator can produce noticeably different behavior in another.

The evidence: physically grounded objects train better policies

In 2025, Researchers at Meta Reality Labs and Stanford University used our 3D Digital Twin Data and tested what that grounded data is worth for robotics. Their Digital Twin Catalog (DTC), presented at CVPR 2025, contains 2,000 real-world objects with millimeter-level geometry and PBR materials, digitized by ALLSIDES (then operating as Covision Media).  They trained the same simulated robot policies twice, once on 24 DTC cups and once on 24 cups from Objaverse-XL, then tested both on a cup neither had seen.

Source: Dong et al., CVPR 2025

With the same policy and training budget, precise pushing within 3 cm succeeded 47% more often with DTC objects. Results were also far more consistent: the spread between training runs shrank from ±6.0% to ±1.2%. Grasping improved on average, though with three runs the error bars overlap. The authors note these are simulation results and that transfer to real robots remains an open challenge.

What does this mean for us? When objects are measured from reality, policies learn from more faithful data. The next step is making sure the physics is just as grounded as the geometry.

Measuring behavior on the real object

Most libraries of 3D assets are built from files, not from objects. Ours starts with the physical object on a shelf, which means we can test it. A small set of controlled experiments, all recorded on video, gives us ground truth for the properties that matter most in contact-rich tasks:

> Tilt-table test → friction. Tilting a surface until the object starts to slide lets us determine its friction coefficient.  

> Drop test → restitution. Dropping the object from a known height and measuring its rebound lets us determine  how much of its bounce it keeps; as a coefficient of restitution for rigid objects, or as stiffness and damping for deformable ones.

> Press test → deformation. A force gauge presses into the object, and we compare how far it deforms under the same load in reality and in simulation.

Task videos → behavior in context. Beyond standard tests, we record objects in real tasks, such as screwing in a light bulb or bending a cable, to capture behavior no single test covers. We then run the same experiment in simulation and adjust the parameters of the assets until the simulated result matches the measurement.

Side by side: real vs. simulated

The two examples below compare a real ball with its Digital Twin in NVIDIA Isaac Sim 6.1 (PhysX), simulated as an FEM deformable body.

Drop test. The ball is released from 0.96 m and recorded alongside the simulation, aligned to the moment of first impact and slowed down 8x. Matching the timing and height of each rebound tunes the asset's stiffness and damping.

Press test. A force gauge presses into the real ball, up to roughly 42 N, while a high-frame-rate camera records it. In simulation, a probe applies the same load as the gauge reading, and we compare how far it travels into the ball (up to around 20 mm). Matching the force–deformation curve tunes how the asset deforms, so it gives way under contact the way the real object does.

Tuned for a simulator, and for a task

Because physics engines model contact and deformation differently, and every simulation has to fit a compute budget, we tune assets for a specific simulator rather than assuming one set of values fits everywhere. The examples here target Isaac Sim with PhysX; the same approach applies to other supported simulators such as MuJoCo or Newton.

Tasks differ too, and each depends on different physical properties. Tuning against the behavior that matters for a given task improves accuracy.

What we're building

This work feeds into three things:

  1. A benchmark dataset for validation and training. Paired real-world measurements and videos alongside their simulated counterparts, so physical accuracy can be measured rather than assumed, and used as training signal.
  1. Task-specific tuning on request. Assets fine-tuned for a customer's simulator and task, for example a grasping setup in Isaac Sim.
  1. Better physical accuracy across our library. What we learn from tested objects improves the physical properties of our SimReady Digital Twins at scale.

In our pipeline, this sits in the Physical Property Enrichment stage: geometry, PBR materials and mass are measured from reality; friction, restitution and other physical properties are predicted and, depending on the asset, validated against real measurements.

From reality to simulation

Simulation is only as useful as its match to the real world. Having the real objects lets us check that match and improve it. We'll share updated results as the benchmark grows.

Working on a manipulation task where contact behavior matters? Get in touch at allsides.tech to talk about tuned, physically grounded Digital Twins for your simulator.

Sources

ABOUT ALLSIDES

ALLSIDES builds the 3D data infrastructure for Physical AI. Its automated 3D Digital Twin Factory turns real-world objects into physically grounded, SimReady 3D Digital Twins with measured geometry, PBR materials and physical properties, delivered through the ALLSIDES 3D Data Platform. ALLSIDES data supports robotics, simulation, World Models, generative 3D AI and digital commerce. The company is headquartered in Bressanone, Italy, with offices in New York and San Francisco. More at allsides.tech.

‍