Wed 2 Feb, 2011: my lyrics-to-audio alignment seminar at QM
I’ve recently arrived in London and I’m looking forward to showing you the lyrics-to-audio alignment work I did with Hiromasa Fujihara and Masataka Goto at AIST in Japan. My talk will give a short introduction to the underlying technology and I will present the Song Prompter application that builds on it.
The seminar is entitled “Lyrics-to-Audio Alignment: Methods of Integrating Textual Chord Labels and an Application” and will take place on Wednesday, 2nd February, 2011, at 3pm, in room 105 in the Electronic Engineering building, Queen Mary University of London, Mile End Road, London E1 4NS.
I’ll also be happy to briefly talk about the NNLS Chroma/Chordino Vamp plugins I recently released with Chris Cannam.
Aligning lyrics to audio has a wide range of applications such as the automatic generation karaoke of scores, song-browsing by lyrics, and the generation of audio thumbnails. Existing methods are restricted to using only lyrics and match them to phoneme features extracted from the audio (usually mel-frequency cepstral coefficients). Our novel idea is to integrate the textual chord information provided in the paired chords-lyrics format known from song books and Internet sites into the inference procedure. We propose two novel methods that implement this idea: firstly, assuming that all chords of a song are known, we extend a hidden Markov model (HMM) framework by including chord changes in the Markov chain and an additional audio feature (chroma) in the emission vector; secondly, for the more realistic case in which some chord information is missing, we present a method that recovers the missing chord information by exploiting repetition in the song. We conducted experiments with five changing parameters and show that with accuracies of 87.5% and 76.0%, respectively, both methods perform better than the baseline with statistical significance. We will demonstrate Song Prompter, a software system that acts as a performance assistant by showing horizontally scrolling lyrics and chords in a graphical user interface, together with an audio accompaniment consisting of bass and MIDI drums. The application shows that the automatic alignment is accurate enough to be used in a musical performance.