Learning style of gestures to synthesize dance from music

Abstract

Synchronization between audio and body movements is a promising research field as it has vast applications in sign language generation, dance creation, robotics, animation and so on. Synthesis of dance from music is a very challenging task because the characteristics of swift and sequential movements of major parts of the body are required to be finely preserved using a small set of audio data. This thesis proposes a novel architecture to learn dances of different styles including Ballet, Rumba, Cha-Cha, Tango and Waltz from music. In particular, a deep learning architecture comprising convolutional neural network (CNN), long short-term memory (LSTM) and mixture density network (MDN) are used to generate rhythmic movements of stick diagram from music. Then generative adversarial network (GAN) is employed to synthesize realistic dance from the rhythmic movements of the stick diagram. Experiments on the proposed and existing models show that the proposed algorithm outperforms the existing method in terms of commonly used performance indices such as- root mean square percentage error (RMSPE), symmetric mean absolute percentage error (SMAPE), organ movement speed error (OMSE). Thus, the proposed method can have a great influence in learning the gaps between action of a trainer and that of the trainee in various applications such as choreography in movies, athletic movements and medical surgery.