Ongoing research · Work in progress
Studying how dense human–object contact trajectories decompose into grasp-level structure, temporal evolution, and realization variation.
Observations from ongoing work, not a finished method or published result.
Motivation
One way to generate hand motion goes through contact: object trajectory → contact → hand motion.
In the two-stage generator we studied (BimArt, CVPR 2025; our port to TACO), replacing the recorded contact with the contact stage’s prediction raises the mean hand error on all four TACO test splits: 37.1 → 58.6, 48.8 → 86.5, 63.4 → 104.9 and 50.9 → 79.8 mm.
These results suggest that contact-stage errors can substantially affect downstream hand generation.
One model family; one training run of our TACO port; split 1 also served for model selection. The recorded-contact input is an oracle, measured from the recorded hands; with it the hand is still 37 to 63 mm off, so contact is a major source of error, not the only one. We quote no percentage share: the two stages’ contributions do not add. No confidence interval exists for the gap.
What information does a temporal contact generator actually need to model?
Observation 1
What varies across a human contact trajectory?
Without grasp labels, “grasp realization” is proxied by take, object or subject identity and the first contact map.
Within a take (one recording), the spatial pattern changes about as much as between two takes on TACO, slightly less on ARCTIC and clearly less on OakInk2, and less than across object meshes or instances. The contact amount differs about twice as much within a TACO take, which includes approach and release, as between takes; not within ARCTIC’s long multi-episode takes (1.10) or OakInk2’s interaction-only segments (0.60). The pattern does change over time: the statement is relative.
p(C1:T | G, τ) ≈ p(s0 | G) · p(C1:T | s0, G, τ)
Choose an initial contact realization (first contact map s0), then model its evolution (C: contact maps; G: object geometry; τ: object motion). On TACO, a learned model that evolves the true first map clearly beats holding it (error 3.48 → 1.82); on ARCTIC and OakInk2 the difference is not distinguishable from zero, although contact still changes there.
Ratios are means over groups; intervals exist per group only. On TACO an object mesh is confounded with the people who used it. ARCTIC’s third bar is relative to two takes of the same subject (1.01 against the level of its other bars); OakInk2’s covers fewer than half of its groups. Holding and evolving both start from the true first map: diagnostic conditions, not generators. Held → evolved: ARCTIC 2.21 → 2.14 (paired difference −0.07, 95% interval −0.26 to +0.11); OakInk2 1.26 → 1.24 (−0.02, −0.11 to +0.06). One model family, one training run; error units differ between datasets.
Interpretation
Which contact a sequence starts from and how it evolves are different problems; how much the second adds differs with the interaction regime.
The evolution itself is not all of one kind.
Observation 2
Does every visible contact reconfiguration change what the grasp can do?
A friction-cone grasp model gives the force and torque a grasp can apply along 76 directions: its modelled wrench capability, a proxy for functional grasp capability. Retention: the share of the later grasp’s capability that the earlier one already had.
Median retention
TACO 0.85 119 events
ARCTIC 0.89 26 events; 95% interval 0.76 to 0.96
Median cosine of the capability profiles
TACO 0.97
ARCTIC 0.96
On TACO contact-change size says little about capability change (rank correlation 0.19); on ARCTIC (0.55) 5 of the 11 larger changes gain more than 25%.
Capability comes from a simplified single-hand grasp model: one unit of force per touching hand part, linearised friction, no object mass or force limits; values are relative, not physical forces. Events: persistent spatial reconfigurations in a maintained grasp; medians over small classes. Larger changes: at or above the training median. Retention is one-sided (a grasp that shrank still scores near one), and the cosine ignores overall scale. A stricter exact-direction measure gives lower medians: 0.66 (TACO), 0.75 (ARCTIC).
Interpretation
Dense contact trajectories mix structural change with realization variation that, in the typical event, leaves the modelled wrench capability nearly unchanged.
Can the structural part be isolated?
Observation 3
Can the functionally meaningful part of dense contact be represented more compactly?
Each frame is described at eight levels; fixed small models recover its modelled wrench profile (itself computed from contact positions and directions) and predict the compact quantities and that profile 1, 4 and 8 frames ahead. A ranking of inputs, not absolute prediction.
Wrench-profile error (relative L1): 0.35 → 0.18 (TACO), 0.51 → 0.18 (ARCTIC) with coarse position and direction.
Wrench error, adding spread and patch count
TACO −10%
ARCTIC −7%
Future-structure error, adding a coarse hand pose
TACO −2% averaged over horizons
ARCTIC no reduction
From the compact state alone, a small fitted model leaves 11% (TACO) and 26% (ARCTIC) of the contact-map variance unexplained.
That remainder persists from frame to frame; the predictors we tested capture little of its change.
The compact level is called R2 (48 numbers per frame). These are comparisons between inputs under fixed small models, on the levels we tested (nothing between 12 and 48 numbers was tried); they are not statements of absolute predictability or information content, and the compact level is close to, not at, the lowest error. Dense inputs score worse under these fixed models; that is no statement about the information they hold. The dense inputs remain better at predicting the dense future itself. The residual is what one particular reconstruction model leaves. TACO and ARCTIC only.
Interpretation
Dense contact reads as a structured contact state plus realization detail.
How should such a structured state evolve over time?
Current research direction
Can human contact dynamics be modelled as the evolution of a compact structural state, while dense contact detail is treated as realization variation?
We are currently investigating how such structural states should evolve over time and how realization detail should be preserved.
This is a direction: no method and no result is claimed.
| TACO | ARCTIC | OakInk2 | |
|---|---|---|---|
| Motivation | ● | – | – |
| Observation 1 | ● | ● | ● |
| Observations 2 and 3 | ● | ● | – |
TACO shows the stronger effect of temporal evolution; on ARCTIC the predictors we tested rarely beat carrying the current contact forward, although contact does change there. We read this as contact dynamics depending on the interaction regime (the test sets also differ in composition), not as a failure of either dataset.
Data: TACO, ARCTIC, OakInk2 (their own licences). Prior work: BimArt; hands: MANO (Romero et al., 2017). Rendered images only; no data is redistributed.
Contact: 01unghui@gmail.com · github.com/ungungnam