Page 15 - My FlipBook
P. 15
Brochure 2020
generation methods. Our proposed model consists of a we incorporate rhythm tracking from the audio model
bowing model for the right hand, a fingering (position) and an emotion predictor model to control those aspects
model for the left hand, and a musical emotion (expression) of body motion. These same principles can be applied to
model for the upper body. The bowing model has been other kinds of stringed instruments. We are still tackling
designed with an audio-based attack detection network, the problem of generating body movements solely from
whereas the ngering model computes left-hand position audio content, but there are many possibilities for future
from music pitch. From this information, patterns for the development.
generated skeleton can be determined. In terms of music
emotion recognition, since periodic head tilt and upper
body motion tend to follow the rhythm and music type,
Figure 2 : The skeleton generation system for violin performance.
For real-time synchronization, our proposed system program director to design responsive events based on
incorporates three elements, i.e., a music tracker, a music that information. We have applied this system to music
detector, and a position estimator. The music tracker visualization, automatic accompaniment/ensemble, and
includes online dynamic time-warping (ODTW) algorithms generation of automatic body movements for a virtual
working across multiple threads. Each thread uses ODTW musician.
to estimate the current performance speed of the live
music performance. Estimated values across threads are Our system has been utilized for several live performances,
averaged to obtain a stable and accurate estimate of including the Sound and Sense concert (in cooperation with
performance speed. Relative tempo values are obtained the Pace Culture and Education Foundation, performed
by comparing the live performance with a reference in the National Concert Hall), the opening ceremony of
performance recording. The function of the music detector the NTHU AI Orchestra (in collaboration with the NTHU AI
is to automatically detect when the music starts, meaning Orchestra), Whispers in the Night (in collaboration with
that there is no need to manually launch the real-time
synchronization mechanism. Finally, since music exhibits utist Sophia Lin, performed in the Weiwuying Auditorium),
many repetitive segments, our position estimation and Sound and Shape (in collaboration with Koko Lab. Inc.,
mechanism allows us to simultaneously track the positions performed at Wetland Venue). These concerts were held not
that the musician is currently playing. Combining these only to test our technology, but also to facilitate in-depth
three elements, we can immediately align the position of conversation among music producers, performers, and
a live performance to a reference recording, allowing a music technology developers, with the view to introducing
new-age music technology to the multimedia industry.
13
generation methods. Our proposed model consists of a we incorporate rhythm tracking from the audio model
bowing model for the right hand, a fingering (position) and an emotion predictor model to control those aspects
model for the left hand, and a musical emotion (expression) of body motion. These same principles can be applied to
model for the upper body. The bowing model has been other kinds of stringed instruments. We are still tackling
designed with an audio-based attack detection network, the problem of generating body movements solely from
whereas the ngering model computes left-hand position audio content, but there are many possibilities for future
from music pitch. From this information, patterns for the development.
generated skeleton can be determined. In terms of music
emotion recognition, since periodic head tilt and upper
body motion tend to follow the rhythm and music type,
Figure 2 : The skeleton generation system for violin performance.
For real-time synchronization, our proposed system program director to design responsive events based on
incorporates three elements, i.e., a music tracker, a music that information. We have applied this system to music
detector, and a position estimator. The music tracker visualization, automatic accompaniment/ensemble, and
includes online dynamic time-warping (ODTW) algorithms generation of automatic body movements for a virtual
working across multiple threads. Each thread uses ODTW musician.
to estimate the current performance speed of the live
music performance. Estimated values across threads are Our system has been utilized for several live performances,
averaged to obtain a stable and accurate estimate of including the Sound and Sense concert (in cooperation with
performance speed. Relative tempo values are obtained the Pace Culture and Education Foundation, performed
by comparing the live performance with a reference in the National Concert Hall), the opening ceremony of
performance recording. The function of the music detector the NTHU AI Orchestra (in collaboration with the NTHU AI
is to automatically detect when the music starts, meaning Orchestra), Whispers in the Night (in collaboration with
that there is no need to manually launch the real-time
synchronization mechanism. Finally, since music exhibits utist Sophia Lin, performed in the Weiwuying Auditorium),
many repetitive segments, our position estimation and Sound and Shape (in collaboration with Koko Lab. Inc.,
mechanism allows us to simultaneously track the positions performed at Wetland Venue). These concerts were held not
that the musician is currently playing. Combining these only to test our technology, but also to facilitate in-depth
three elements, we can immediately align the position of conversation among music producers, performers, and
a live performance to a reference recording, allowing a music technology developers, with the view to introducing
new-age music technology to the multimedia industry.
13
