JISE


  [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]


Journal of Information Science and Engineering, Vol. 40 No. 3, pp. 581-594


Visualization Method for Sound Information in Videos


YASUFUMI TAKAMA+, IKUYA SASAKATA
AND HIROKI SHIBATA
Graduate School of System Design
Tokyo Metropolitan University
Hino, Tokyo 191-0065, Japan
E-mail: {ytakama;hshibata}@tmu.ac.jp


This paper proposes a visualization method of sound information in videos. There is a situation in our daily lives where we want to watch a video with the sound off, such as when traveling by public transport and watching a few videos all at once. One of the most common alternatives to audio for representing sound information in a video is a subtitle. However, subtitles are not suitable for representing nonverbal information of sound, such as speakers’ emotions and the atmosphere of BGM. As another alternative to represent the impression represented by a sound, this paper aims to visualize the sound information in videos by combining different representations for various sound elements. Employing the representations used in comics, the proposed visualization method uses speech bubbles for conveying the emotions of speakers, frame borders for the atmosphere of BGM, and concentration lines for a sudden noise. The effectiveness of the proposed method is shown by comparing it with a subtitle-based method with user experiments. The proposed method is designed considering the balance between simplicity and informativeness. The validity of the design concept is investigated by comparing the proposed method with its modified version that is designed based on the findings from the experiment.


Keywords: sound information, visualization, video, human interface, sound visualization

  Retrieve PDF document (JISE_202403_10.pdf)