Multi-Modal Perception of Music and Music Instruments

Sound and vibration perception are always coupled in live music experience. Just think of a rock concert or hearing (and feeling) a church organ sitting on a wooden pew. Even in a concert hall vibrations above the perception threshold can be measured. However, those are usually not perceived separately, because they are integrated with the other senses into one multimodal concert experience.

© Sebastian Merchel

The air-borne sound causes seat vibrations, e.g., in the Semperoper opera house in the picture, or excites the skin surface directly. For some instruments (e.g. an organ) structure-borne sound is transmitted directly from the instrument to the listener. 

© Sebastian Merchel

Room impulse responses can be measured binaurally using a dummy head and for all directions individually using a spherical microphone array. This data is used to reproduce the opera house acoustically via wave field synthesis in the Multi-Modal Measurement Lab for further perceptual experiments.

As can be seen in the picture below: the frequency ranges of auditory and tactile perception overlap. Additionally the fundamental frequency ranges are illustrated for some instruments, which can excite sound and vibrations.

© Sebastian Merchel

If concert recordings are played back with multimedia hi-fi systems at home, the vibratory information is missing in the majority of cases. This is due to low reproduction levels or to the limited frequency range of conventional loudspeakers.

The audio signal on todays DVDs contains an additional channel for low frequency effects (LFE), which is intended for reproduction using a subwoofer. The generation of tactile components is still very restricted. An enhancement of such a systems might be possible using an electrodynamical shaker which generates whole body or seat vibration.

Different systems implementing this approach have been developed and evaluated in cooperation with car manufacturers and supliers.

© Sebastian Merchel

For the generation of the vibrotactile signal, signal processing algorithms are analyzed using exemplary audio signals. Different parameters during this process have to be examined in relation to their perceptual consequences using psychophysical experiments.

In general: Vibrations can have a positiv effect on the perception of music. In addition, they can increase the perceived loudness. 

© Sebastian Merchel

Further, haptic feedback proved to be important for playing a music instrument. As an example, an electric violin equiped with a shaker under the chin rest and a freeDSP, as shown in the picture, was used to investigate such effects.

In a current research project, the vibrations on a violin are specifically influenced by shakers in order to change the perception of the instrument during playing. The aim is to investigate whether certain frequency components in the vibration signal can enhance or suppress certain perceptual attributes (e.g. responsiveness, power, clarity). To achieve this goal, a 'virtual prototype' based on an electric violin will be used, which can modify the sound as well as the vibrations of the instrument as desired.

Contact