Home » Seen and Liked

Ground Truth Session at the Dagstuhl Seminar and beyond

11 Februar 2011 1,862 views 4 Comments

I liked the session on ground truth at Dagstuhl. There are always philosophic issues of course, but we had a really nice practical introduction to ground truth from Ichiro Fujinaga (Music Technology at McGill) through a video showing an actual real-time example of a segmentation annotation (part of SALAMI, I believe). Sadly, the video is not available online, since the charming girl performing the segmentation didn’t like the idea of being distributed world-wide. Ichiro also reported of many issues their practical quest encountered that lead directly to more (sometimes philosophical) questions and issues, e.g.

  • can we one annotation style fit all kinds of music? (it seems not)
  • how many annotators do we need? – McGill used two per piece, Geoffroy Peeters suggested an odd number to filter mistakes
  • you might need a reviewer to do the filtering
  • some tasks are doable by mildly attentive musicians, some only by hardcore experts
  • paying people more money is not always a good incentive — there seems to be a limit of how much people can annotate per day without going insane

These are all important points, I think. On a side note: the last one is very much in line with a psychological finding I’ve recently been made aware of through a video of a TED talk by Daniel Pink (highly recommended): for difficult, non-mechanical work, money is not a good incentive.

We also had a fervent talk by Kazuyoshi Yoshii advocating a new kind of ground truth, which may need further exploration. I find it hard to summarise here because I did not quite understand, but Yoshii (or Kaz, as he’s known in the West) is a clever guy, and I shall ask him about it again.

Here’s some more of my own thoughts (possibly linked to Yoshii’s): So, engineering needs ground truth. Or does it? I can imagine that die-hard ground truth will fall out of fashion in the next few years, since we’ve “done that”, and the limitations exposed through the ground truth based MIREX tasks may lead people to concentrate more on purely data-driven research that seems to be becoming all the rage now (cf Columbia University’s Million Song Dataset or the likes of Last.fm).

In the meantime, let’s generate some good ground truth!

  • Let’s have comparable formats and annotation strategies in order to enable collaboration
  • Let’s all include annotations in our next research grant proposal (I did)!


  • Matt McV said:

    very nice!

    Interesting to see that money isn’t a good motivation for ground truths. Was there any update on the McGill dataset? Best,


  • Thierry BM said:

    Very interesting!
    but you seem rather pessimistic about die-hard ground truth, have we really “done that”? How big/large have we tried, and is it enough to give up?
    And isn’t there a third way, algorithm-aided annotation, with computers and humans in the same loop? For instance, you can find many critics of the Echo Nest “sections”, but they are still reasonable, could they help a human annotate faster (since he would not start from scratch)?

  • Matthias Mauch (author) said:

    Hi Thierry,

    First of all: congratulations to the Million Song Dataset! Promises to be a fabulous resource.

    I’m not pessimistic about ground truth itself at all. And I could not agree more that computer-human interaction for ground truth generation is a way forward. I guess I’ll have to say more about that soon, especially if my grant application goes through — I propose to do more or less exactly that.

    The only way in which I may be a bit pessimistic is that people might blindly data-mine and stop caring about actual ground truth data, or good modelling. So I hope that great efforts such as yours will broaden peoples minds, not narrow them.


  • Matthias Mauch (author) said:

    Hi Matt, thanks for the praise … not sure about any updates from McGill’s side, I think they said they’ve got nearly where they wanted to be, so I assume that ISMIR will reveal more.