Motion Manifold Flow Primitives for Task-Conditioned Trajectory Generation under Complex Task-Motion Dependencies

Under review

Yonghyeon Lee1, Byeongho Lee2, Seungyeon Kim2, Frank C. Park2
1Korea Institute For Advanced Study, 2Seoul National University
Paper Code (Coming soon)

TL;DR: MMFP can generate motions from text inputs, capturing complex text dependencies in motion distributions while simultaneously addressing the challenges posed by high-dimensional trajectory data and small dataset sizes.

Example 1: SE(3) Pouring Motion

Can I have a drink,
whatever you've got?
Give me some wine,
please.
May I have some water
from the very left?

Example 2: 7-DoF Robot Waving Motion

Wave your hand.
Look to the very left and
wave your hand.
Look to the front and
wave your hand in a small gesture.

Real Robot Experiments

Give me some wine.
Give me some water and pour it from the very left.
Turn to the front and wave your hand in a very big gesture.
Turn to the very right and wave your hand.

Abstract

Developing text-based robot trajectory generation models is made particularly difficult by the small dataset size, high dimensionality of the trajectory space, and the inherent complexity of the text-conditional motion distribution. Recent manifold learning-based methods have partially addressed the dimensionality and dataset size issues, but struggle with the complex text-conditional distribution. In this paper we propose a text-based trajectory generation model that attempts to address all three challenges while relying on only a handful of demonstration trajectory data. Our key idea is to leverage recent flow-based models capable of capturing complex conditional distributions, not directly in the high-dimensional trajectory space, but rather in the low-dimensional latent coordinate space of the motion manifold, with deliberately designed regularization terms to ensure smoothness of motions and robustness to text variations. We show that our Motion Manifold Flow Primitive (MMFP) framework can accurately generate qualitatively distinct motions for a wide range of text inputs, significantly outperforming existing methods.

Summary

1 Dataset: SE(3) Pouring Trajectories

2 Three Key Challenges

3 Curse of Dimensionality

! A naive application of existing algorithms directly in the high-dimensional trajectory space
significantly fails, given only a handful of demonstration data.

4 Previous Works: Manifold-based Models

! The conditional autoencoder architecture, where the decoder takes both the latent variable \(z\) and text \(\tau\) as inputs,
struggles to capture complex text dependencies in motion distributions while addressing issues related to dimensionality and dataset size.

References:
1. Task-Conditioned Variational Autoencoders for Learning Movement Primitives (Noseworthy et al., CoRL 2019)
2. Equivariant Motion Manifold Primitives (Lee et al., CoRL 2023)

5 Solution: Motion Manifold Flow Primitives

* Language-Guided Trajectory Generation with MMFP and Sentece-BERT

Citation


      @article{lee2024motion,
        title={Motion Manifold Flow Primitives for Language-Guided Trajectory Generation},
        author={Lee, Yonghyeon and Lee, Byeongho and Kim, Seungyeon and Park, Frank C},
        journal={arXiv preprint arXiv:2407.19681},
        year={2024}
      }