DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding

Jungbin Cho1*, Junwan Kim1*, Jisoo Kim1, Minseo Kim1, Mingu Kang2, Sungeun Hong2, Tae-Hyun Oh13, Youngjae Yu13†
1Yonsei University, 2Sungkyunkwan University, 3POSTECH

 

*Equal Contribution. †Corresponding author.
Paper arXiv Video Code
Description of the image

Abstract

Human motion, inherently continuous and dynamic, presents significant challenges for generative models. Despite their dominance, discrete quantization methods, such as VQ-VAEs, suffer from inherent limitations, including restricted expressiveness and frame-wise noise artifacts. Continuous approaches, while producing smoother and more natural motions, often falter due to high-dimensional complexity and limited training data. To resolve this discord between discrete and continuous representations, we introduce DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding, a novel method that decodes discrete motion tokens into continuous motion through rectified flow. By employing an iterative refinement process in the continuous space, DisCoRD captures fine-grained dynamics and ensures smoother and more natural motions. Compatible with any discrete-based framework, our method enhances naturalness without compromising faithfulness to the conditioning signals. Extensive evaluations demonstrate that DisCoRD achieves state-of-the-art performance, with FID of 0.032 on HumanML3D and 0.169 on KIT-ML. These results solidify DisCoRD as a robust solution for bridging the divide between discrete efficiency and continuous realism.

Concept

Discrete quantization methods encode multiple motions into a single quantized representation. While existing methods deterministically decode from this quantized representation, DisCoRD iteratively decodes the discrete latent in acontinuous space to recover the inherent continuity and dynamism of motion. To assess the gap between reconstructed and real motion, prior work primarily used FID as the metric. Here, we additionally propose symmetric Jerk Percentage Error (sJPE) to evaluate the differences in naturalness between reconstructed and real motion.

Architecture

An overview of DisCoRD. During the Training stage, we leverage a pretrained quantizer to first obtain discrete representations (tokens) of motion. These tokens are then projected into continuous features C, which are concatenated with noisy motion Xt. This concatenated feature is used to train a vector field v. During the Inference stage, we use a pretrained token prediction model based on the pretrained quantizer to first generate tokens from the given control signal. These generated tokens are then projected into continuous features Ĉ, concatenated with Gaussian noise X0 ~ N(0, I), and iteratively decoded through the learned vector field vθ into motion X̂1.

Demo Video

BibTeX

@misc{cho2024discorddiscretetokenscontinuous,
      title={DisCoRD: Discrete Tokens to Continuous Motion via Rectified Flow Decoding}, 
      author={Jungbin Cho and Junwan Kim and Jisoo Kim and Minseo Kim and Mingu Kang and Sungeun Hong and Tae-Hyun Oh and Youngjae Yu},
      year={2024},
      eprint={2411.19527},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.19527}, 
}