Report of the EuroITV 2010 Conference in Tampere, Finland

The 8th EuroITV Conference was held in Tampere, Finland on 9-11 June 2010. EuroITV is the leading conference on interactive television and video. Hosts of the conference were the Tampere University of Technology, the Helsinki University of Technology and the Tampere University of Applied Sciences.

This year’s theme “” created a space for a very broad range of topics, not only focussing on the television itself, but also on television content on the Internet, the mobile phone and crossovers between television and various other (social) media.

The first day of the conference was dedicated to tutorials, mainly about designing interactive television applications, and workshops about user experiences, the future of television and integrated, ambient television applications. The second and third day were filled with poster presentations, demonstrations and various sessions with a broad range of topics like personalization and recommendation systems, interactive applications and convergence and cross-media. These two days also included two keynotes by Marcos Gonzalez-Flower, Global Head of Media Consulting at Siemens UK and Jyri Huopaniemi, Director of the Nokia Research Center.

The Netherlands Institute for Sound and Vision has made a substantial contribution to the conference with a short paper presentation about expert search by broadcast professionals, two demonstrations about television and the semantic web, and browsing archival video with advanced video retrieval and last but not least, the Golden Award (first prize) for Waisda? in the Grand Challenge Competition.

Some conference highlights:

Methods for User Studies of Interactive (TV) Technologies

This full day workshop was chaired by Dr. Paul Marrow from BT Innovative & Design, Dr. Lydia Meesters from Eindhoven University of Technology and Prof. Marianna Obrist from the ICT&S Center of the University Salzburg. As the title of the workshop already suggests, this workshop wasn’t just about user studies methodologies for television technologies, but also for interactive technologies in general. Measuring user experiences for any interactive application is a very complex form of research, which requires a very pragmatic and ‘open’ research attitude. This workshop focussed on the many approaches that are used for measuring different levels of user experience. Approximately 20 people joined this workshop, which are all researchers working for various companies and universities, and doing research on interactive applications.

The morning programme contained presentations from the workshop participants about the various approaches and practices in researching user experience. In the afternoon session, participants were asked to pose a number of research questions and problems. After clustering these into main themes, the participants split up in three groups, where one theme was discussed in more detail. The results of the discussions were documented in a poster and presented to the other groups. The main insight of this day was that user experience has to be measured by combining various research methods, both quantitative and qualitative, and that the various stages within the product development require different forms of testing with different user groups.

Session ITV Applications for Communities

This session was scheduled to contain four presentations: two on mobile video sharing and two on digital archiving. Unfortunately, the speaker of the presentation about user inspired digital archiving of cultural heritage wasn’t able to attend, so the focus was mainly drawn towards video on mobile devices.

Mobile Video Sharing: Documentation Tools for Working Communities – Antti Koivisto

Antti Koivisto from Tampere University of Technology presented the development and the infrastructure of MoViE, a mobile application that allows users to create short clips of their environment and upload these to a website where they can share, edit and remix the content Together with his colleagues from Tampere university, he did a first test with the prototype during a festival and although users liked the application a lot, there were some technical issues like a failing Internet connection that made the use of the prototype difficult at times. The test results were used to improve the MoViE application and they are currently working on a new version.

Troubleshooting and Creating Joined Experience: Festival Personnel Engaging in Mobile Video – Anna Haverinen

This talk by Anna Haverinen from the University of Turku reported the results of the first MoViE tests from a user perspective. As a virtual anthropologist, she analysed the kind of content the test subjects created during the test. Eight festival employees acted as test subjects and were asked to shoot and upload videos during the festival. She discovered a great difference between the remarks that people made about the application and the actual use of it. For instance, some people for instance indicated that a commenting tool was very helpful and easy, without actually using it in practice.

Expert Search for Radio and Television: a Case Study amongst Dutch Broadcast Professionals – Wietske van den Heuvel

This talk was given by Wietske van den Heuvel from Sound and Vision. The presentation highlighted the results from a study in which eight broadcast professionals from various broadcasting companies were observed while performing searches with the online search system of Sound and Vision. The research focussed on the question if the task affects the professionals’ search strategy. Three kinds of factors were researched: difference in searching for radio or television content, difference in length (shots versus entire programs) and the influence of time restraints. Results from the study show there is no difference between the search strategies that are used during the performance of the tasks. People always use the same strategy, despite their information need.

The EuroITV 2010 Grand Challenge Competition

The Grand Challenge Competition honours the creation of interactive video content, applications, and services that enhance the television and video viewing experience. The entries are judged by an international jury of interactive media experts and the best entry is awarded with a prize of € 3,000. Three entries were nominated before the start of the conference:

  1. Smart Video Buddy: an application that analyses video streams and automatically detects semantic concepts, which are used to create links to related content.
  2. Smeet: one of the fastest growing online 3D world and chat community for entertainment and social viewing on the Internet.
  3. Waisda?: a video labeling game by Sound and Vision and KRO Broadcasting. The labels or ‘tags’added by Waisda? players can be used as time-related metadata, which improves the findability of the content.

The awards ceremony took place during the first keynote session on June 10, where Waisda? was named as the Grand Challenge winner. The nominated applications were also demonstrated during the rest of the conference day.


The programme included various interesting demonstrations, two of which were given by Johan Oomen from Sound and Vision. The first demonstration concerned Video Active and EUscreen . Both projects provide free online access to Europe’s television heritage. Video Active ended in August 2009 and EUscreen started as its follow-up project in October 2009. Both projects are funded within the eContentplus programme of the European Commision. EUscreen will contain over 30,000 items and is one of the main audiovisual aggregators for Europeana. The EUscreen portal will be online in January 2011. The second demonstration showed the project Hollands Glorie op Pinkpop, which ended in February 2010. The project is a cooperation between Sound and Vision, Videodock, the University of Amsterdam and the University of Twente. By using automatic concept detection and speech recognition, videos of pop concerts and interviews with the musicians are searchable on individual shot level. Videos from the pop festival Pinkpop were used for this public demo website. Users could also give feedback on the level of retrieval and indicate if the concept detection was right or wrong. This feedback will be used to improve the concept detection technology.

Further reading

Read the full conference programme here
Conference report by CTOi Consulting

Scientific Publications (May 2010)

Researchers from the R&D department of Sound and Vision have recently contributed to the following papers:

Oomen, Johan and Belice Baltussen, Lotte and Limonard, Sander and van Ees, Annelies (2010) Emerging Practices in the Cultural Heritage Domain – Engaging Users in a Shared Information Space. In: Proceedings of the WebSci10: Extending the Frontiers of Society On-Line, April 26-27th, 2010, Raleigh, NC: US.

B. Huurnink, C.G.M. Snoek, M. de Rijke, and A.W.M. Smeulders. Today’s and Tomorrow’s Retrieval Practice in the Audiovisual Archive. In: ACM International Conference on Image and Video Retrieval 2010 (CIVR 2010). ACM, July 2010. (to appear, best paper candidate).

Oomen, Johan. Audiovisual Heritage and Participatory Culture. In: JTS2010, DIGITAL CHALLENGES AND DIGITAL OPPORTUNITIES IN AUDIOVISUAL ARCHIVING May 2 – 5, 2010 Oslo, Norway.

Celluloid Remix – Lessons learned

In this document Annelies Termeer of EYE Film Institute Netherlands describes step by step the development and the approach of the online video remix competition Celluloid Remix, organized in 2009 by EYE (then Filmmuseum), Knowledgeland and Images for the Future. This document aims to inform archives or museums wishing to plan a similar project about the do’s and don’ts.

See also and

Background Celluloid Remix

As part of the large-scale digitization project Images for the Future, the Filmmuseum restores and digitizes a large part of its film collection. An important – already digitized – subcollection of the Filmmuseum consists of Dutch Early Cinema.

Early cinema was to a certain extent a semi-finished product. The film was often accompanied by someone who provided explanation, and by a musician or a small orchestra playing background music. It also occurred that the one who held the show cut the film, and then created his/her own compilation with material from another source (sometimes his/her own footage). Here, an analogy is found between the practice of early film screening and the contemporary remix practice.

To experiment with the possibilities of digitized film material the Filmmuseum, in collaboration with Images for the Future and Knowledgeland, organized Celluloid Remix: an online remix contest using footage from the collection.

Starting points

  • Provide 21 (public domain) fragments from the Filmmuseum’s collection.
  • Theme: “Modern Times”.
  • Duration of the contest: 15 April until 1 September 2009.
  • Publication under Creative Commons license, with a choice between two licenses:
  • Creative Commons Attribution Non-Commercial Share Alike license (CC-BY-NC-SA) and Creative Commons Attribution (CC-BY) license
  • Every contestant uses his/her own remix software. We provided tips and suggestions for useful programmes.
  • Maximum length entries: five minutes.
  • Four prizes available:
  • First prize: a remix package of €1,000.- to spend as desired on hardware or software (for instance Adobe Premiere Pro CS4 or Final Cut Studio 2); Gouden Kalf ticket to the Dutch Film Festival; one year free admission to the Filmmuseum.
    Second prize: Flip MinoHD camera; Gouden Kalf ticket to the Dutch Film Festival; one year free admission to the Filmmuseum.
    Third and fourth prize: a Resolume Avenue 3 VJ software package.


  • Establishing and increasing the name of the Images for the Future project among young people; demonstrating that an important part of our cultural heritage is being disclosed, which is relevant for the contemporary, personal experience of culture.
  • Inspiring other heritage institutions to disclose their collections on external platforms to a young and broad audience.
  • Introducing a new target group (young creatives) to the (collection of the) Filmmuseum.
  • Drawing attention to the Dutch Early Cinema collection.
  • Inspiring other heritage institutions to create an environment in which end users can reuse the heritage.
  • Exchanging knowledge regarding copyright issues and open content models for the heritage sector.
  • Showing higher education institutes in the field of film culture, film production and film art that the Images for the Future project saves and discloses an important part of our cultural heritage for use and reuse.

Project organization. Duration: December 2008 until September 2009

  1. From January onwards – choosing footage based on theme (Modern Times) and availability (footage without copyright; public domain).
  2. Selecting online access platform: On this platform, various video formats can be downloaded and uploaded. We provided Celluloid Remix footage in .mov as well as in .ogg format.
  3. Launching website
  4. 15 April 2009 – launch at BeamLab, in Pakhuis de Zwijger (Amsterdam).
  5. Generating publicity (press release, e-mail, Facebook, Hyves, Twitter).
  6. Approaching schools and academies for cooperation: participation in the contest and possible adaptation in course programme.
  7. June/July/August – organizing workshops at Willem de Kooning Academy / Crosslab, VJ Academy, Hogeschool Rotterdam (HRO) Crossmedia, BeamLab Summer School and the Netherlands Media Art Institute Summer School (NIMK). Under supervision of among others Eboman, Ruud Lanfermeijer and Jaromil those people interested could start working on the footage.
  8. 1 September – entry deadline.
  9. 8 September – jury deliberation and shortlist compilation.
  10. 25 September – final event during Dutch Film Festival. Shortlist entries are shown and prizes are awarded to four winners.

Ambassador and cooperative partners
In an early stage we requested the renowned Dutch sample artist Eboman to make the first remix of the available footage. At the same time, this remix was used as promotion film for the competition. In addition, we asked him to be the ambassador of Celluloid Remix. This improved the competition’s status and image.

By cooperating with various parties (BeamLab, VJ Academy, Netherlands Media Art Institute, Dutch Film Festival, Willem de Kooning Academy) we tried to reach interested groups and increase the support for the contest.


  • Total entries: 54 (more than expected: we assumed 30-40 entries).
  • Source of entries: the contestants were mainly attracted online. 13 out of the 54 entries came from workshop participants, the rest came through online word of mouth. Social networks: 23 contestants were members of the Celluloid Remix Facebook and Hyves group. However, there was minimal online discussion or conversation between contestants.
  • Timing: most entries were submitted at the last moment. The total amount of entries halted at 14/15 for quite a while, the rest came in during the last two days.
  • Style: narrative/documentary character (29) featuring a personal quest, next came VJ (13), animation/art (8), humour (3), promotion/advertising (1).
  • Atmosphere/ contestants’ reactions:
    • Motivation to compete: fun to play with the old material. Profile: amateurs and semi-professionals, people following creative courses. Enthusiasm/dedication: Two contestants submitted three entries, one contestant submitted two. Several people participated in both workshops.
    • Appreciation of the Filmmuseum‘s new look and review of the events: Jata Haan (Movement): “Digitization projects like Images for the Future and websites supporting the Creative Commons license, like Flickr and The Freesound Project, have established a culture of reusable and more accessible digital media. In due time these cooperation projects should gain popularity, receiving the support they deserve.”
    • Enthusiasm/dedication: Two contestants submitted three entries, one contestant submitted two. Several people participated in both workshops. o    Appreciation of the Filmmuseum‘s new look and review of the events: Jata Haan (Movement): “Digitization projects like Images for the Future and websites supporting the Creative Commons license, like Flickr and The Freesound Project, have established a culture of reusable and more accessible digital media. In due time these cooperation projects should gain popularity, receiving the support they deserve.”

In general

  • Celluloid Remix had a positive effect on the image of the Filmmuseum.
  • Celluloid Remix created a name and goodwill among, and connection with a new, young target group.
  • Eboman’s role as ambassador was important.
  • Cooperation of various partners went well – positive effect on reach.
  • The workshops led by professionals were very much appreciated, and were also necessary; many users needed tips and support.
  • The competition was now held locally (in the Netherlands). Could next editions be held internationally?

Points of interest

  • Allot enough time to select footage and theme.
  • Create the right balance between informing and stimulating the audience.
  • In practice, was not the best solution to facilitate Celluloid Remix. For the next edition, we will search for other platforms.
  • A five-minute remix turned out to be too long: decrease the maximum duration of entries to four minutes.
  • Various contestants asked for higher resolution footage: next time it should be available. Now the maximum width was 480 pixels.
  • We could have made more of the partnership with Dutch Film Festival: more attention in, and more connection with the festival programme. A different screening platform could be more suitable.
  • Focus on one target group instead of several.
  • As for the cooperation with the academies, the timing of Celluloid Remix 2009 could have been better – they were very busy working on the final presentations and exams in May/June. Initiate cooperation with academies earlier.

Waisda? Video Labeling Game: Evaluation Report

The Waisda? (which translates to What’s that?) video labeling game was launched in May 2009. It invites users to tag what they see and hear and receive points for a tag if it matches a tag that their opponent has entered. Waisda? is the world’s first operational video labelling game. The underlying assumption is that tags are most probably valid if there’s mutual agreement. Over 2,000 people played the project and within six months, over 340k tags have been added to over 600 items from the archive. Initial findings have been published earlier, when the pilot period was still running. This evaluation report (PDF download, in Dutch), includes a quantitative and qualitative analysis of the tags, as well as a usability study of the game environment and a study into the incentives that apply to people playing the game. The evaluation report is written by Lotte Belice Baltussen, in collaboration with Maarten Brinkerink and Johan Oomen of the Netherlands Institute for Sound and Vision R&D Department. Researchers at the VU University Amsterdam, Business Web & Media Section, also provided crucial input. The VU University Amsterdam carries out this research in light of their involvement in the PrestoPRIME European research project.

The evaluation report provides evidence that crowdsourcing video annotation in a serious, social game setting can indeed enhance retrieval of video in archives. It features success factors organizations need to take into account in setting up services that aim to actively engage their audiences online. The main conclusions are listed below:

Read the rest of this entry »

Video Labeling Game Waisda?: Preliminary results and ongoing research

Ten months ago the entry “The Wisdom of the Crowds in the Audiovisual Archive Domain” was posted on this research blog. In it, the interest of the Images of the Future consortium in creating social and open archives was discussed. Due to the large scale digitisation of archival materials the opportunities of offering public access to these materials has increased dramatically. One of the ways in which archives can provide access is by creating opportunities for social tagging. This allows people to annotate archival materials with their own key terms (tags). This is not only beneficial for the original tagger, who can use their own tags to find these self-annotated materials more easily in the future, but also for other users that are searching through the user-annotated collection with similar search terms. The tags that are added by users might overlap with the metadata that is produced by experts, or generate new terms and consequently new ways of looking at and finding archival materials. This might bridge the semantic gap[1] between the vocabulary that annotation experts use and the ways in which the general public refers to and interprets (audio-visual) information.

The debate on whether tagging and other crowdsourcing possibilities will actually contribute to the accessibility of archives, or that it will just cause chaos and make finding materials more complicated and murky is still in full swing.[2] There have been some pilot projects on tagging and other crowdsourced metadata which generated some interesting and encouraging data. Notably, partners of the Flickr: The Commons project used this popular photo sharing website to make their collections more accessible, and for collecting annotations by the public. The Nationaal Archief (the Dutch national archive, and one of the Images for the Future partners) and Spaarnestad Photo were part of this project. The results were promising[3], but more hard data is needed to show in what way tagging can be beneficial to archives. Thus, the result of the Images for the Future consortium’s interest in user generated metadata was the development of tagging game through which moving images can be annotated by the public. Through this game, a dataset that will be gathered which can be used to answer some questions raised in this ongoing and topical debate.

Waisda? What’s that?

The online video labeling game Waisda? (which translates to What’s that?), is a project that is an initiative managed by the Netherlands Institute for Sound and Vision in close collaboration with the Dutch public broadcaster KRO (Catholic Radio Broadcasting). The game was developed by internet agency Q42. When the six month pilot project ends in December, the VU University Amsterdam will research various possibilities for implementing these user-generated  tags, and will develop new versions of the game with improved and extended game design and interface options.

Waisda? was launched in May and allows players to annotate Polygoon newsreel journals and KRO programmes such as Boer zoekt Vrouw (Farmer Wants a Wife), Spoorloos (Find my Family) and Memories. Recently the archive of Barend en Van Dorp, a popular Dutch talk show (broadcast from 1990 to 2006), was added to Waisda?

The basis of the game is simple. Players go to the Waisda? website and are presented with a selection of four different episodes of the programmes mentioned above. They can choose any of these programmes to start tagging. The programmes do not start from the beginning, but are played sequentially on the website. Therefore the players drop in at the point that video happens to be at. This means they never have to wait for a game to start, but can start tagging straight whenever they want. Players are asked to tag what they see and hear and receive points for a tag if it matches a tag that their opponent has typed in. The reasoning behind this is that a tag is probably valid if at least two people agree on it. This is the same assumption that is made in the case of the Games with a Purposethat were developed by Luis von Ahn, now professor of Computer Science at Carnegie Mellon University.[4] His ESP game demonstrator, in which two players add tags to a picture, was so successful that Google has licensed it under the name ‘Google Image Labeler’[5].

Right now the Netherlands Institute for Sound and Vision is working on a preliminary evaluation of Waisda? This involves both performing more research on how the game itself can be improved and analysing the crowdsourced tags that were added by the players in the last months. Since Waisda? was launched in May 2009, well over five hundred videos were tagged by a total of almost 2,300 unique players. There are 150 registered players, most of whom return frequently to play the game. So far over 14,000 unique tags have been added via Waisda? and this number still rises every day. In the end, the dataset generated by the people that play Waisda? will be used for in-depth research that will result in recommendations for the improvement of the accessibility of, and search functionalities for, audiovisual archives with crowdsourced metadata.

Tag analysis and research

The tags that were added so far were compared to the terms in the GTAA thesaurus the Netherlands Institute for Sound and Vision uses to classify the audiovisual materials in their archive, and almost 15 % of the tags provided a perfect match. This may not seem like a big number at first glance, but the GTAA contains only very specific terms like person names, genres and topics, and it was therefore not expected that many tags would match with this professional thesaurus. The tags were also compared to another database called Cornetto which contains the bulk of all official Dutch words, and another 45 % of the tags matched. There is some overlap between the Sound and Vision thesaurus and Cornetto, but still well over half the tags added via Waisda? are definitely usable based upon this first simple quantitative analysis.

This does not imply that the other half is not. There are, for instance, tags that contain spelling or typing errors but point to relevant tags. After analysing a representative and random sample of the tags a little under 10 % of them turned out to contain an error. It is expected that this percentage will eventually be lower, since there are players that enter their erroneous tag correctly after realising their mistake, in order to still receive points.

There are also tags that consist of more than one word and that are not recognised as correct terms in the Sound and Vision thesaurus or Cornetto. For example, the tag ‘illegitimate children’ does not appear in the GTAA thesaurus or the Cornettovocabulary, but the individual words do appear in Cornetto. Thus, by separating the tags that consist of multiple terms and that do not match either thesaurus they can still prove to be very useful.

Another area that requires additional research are tags that appear in multiple categories of the thesaurus and Cornetto. The tag ‘link’ means ‘dangerous’ or ‘connection’ in Dutch, among other things, and the term is therefore ambiguous. To find out which meaning the tagger intended, one solution would be to analyse the tags that were added to that video in proximity to ‘link’. If ‘scary’ and ‘exciting’ were added besides ‘link’ it is possible to semantically determine that in this case the meaning ‘dangerous’ is the most plausible. Ideally, semantic software can be used to make these determinations automatically.

These and other topics are analysed in a follow-up research project that is executed in close collaboration with the VU University Amsterdam. This project is part of the European PrestoPRIME programme, in which various partners are collaborating to “research and develop practical solutions for the long-term preservation of digital media objects, programmes and collections.”[6] The university’s research will take three years, and will result in in-depth advice on how to process and implement user generated content such as the Waisda? tags, as well as new implementations for game and interface design, which will be discussed later on in this article.

Read the rest of this entry »

Taking Pictures To The Public

Evaluation Report Nationaal Archief & Spaarnestad Photo on Flickr The Commons

On October 21, 2008 the Nationaal Archief and Spaarnestad Photo were the first Dutch heritage institutions to place a small selection of their photos on Flickr The Commons, the ‘archives section’ in the popular online photo community Flickr. This initiative is part of a series of pilot projects that are being developed within Images for the Future in order to conduct research on involving the broad audience in photographic collections and metadata generation. Within two weeks, the photo stream of the Nationaal Archief had over 400,000 page views and 400 comments. These large numbers were caused by the extensive amount of attention the media dedicated to the initiative–resulting in, among other things, articles in national newspapers De Volkskrant and Het Parool, radio reporting by the Wereldomroep and, most spectacular of all, a prime time news item in NOS Journaal (causing page views to rise up to 100,000 in one night).

The pilot was decided to have a duration of 6 months (October 2008 – April 2009), followed by an evaluation. Now, after this period, the Nationaal Archief Flickr account contains almost 800 photos, had more than 1 million page views and has almost 2000 comments and over 6800 tags added.

In the evaluation report evaluatie-nationaal-archief-op-flickr-commons.pdf  these results are reviewed. Are the initial goals achieved? What can be said about the quality of the audience’s comments and tags? In the conclusion, we seek to answer the question of what added value Flickr can have in making photographic collections more accessible to a broad audience, and in enrichment of these collections by the end user.

Read the full report here (in Dutch, 3.7 Mb): evaluatie-nationaal-archief-op-flickr-commons.pdf  (please open in Adobe Reader)

The National Archive and Spaarnestad Photo release new photos on The Commons on Flickr.

It has been a busy week here at the National Archive. On the 18th of December the National Archive and Spaarnestad Photo launched new photographs on The Commons on Flickr. This time the photographs related to various subjects. Firstly,  as part of the National Archive’s Afscheid van Indië project (www.afscheidvanIndië.nl), The National Archive published some photographs of the Dutch East Indies on Flickr. Secondly, the National Archive also published a number of photographs by the famous Dutch photographer Willem van de Poll, which can also be viewed on The Commons. Then, getting in to the Christmas spirit, our partner, Spaarnestad Photo published some photos with a Christmas theme.

The National Archive has now been on-line for almost two months and so far it has generated about 650 000 views, about 1400 tags and 250 comments.

The initiative has caused quite some commotion in both the Dutch and international archival community. Last week I gave a talk in front of a critical audience of Photo journalists at a conference of the European Federation of Journalists (EFJ). The title of the conference was: “Photo Journalists: an endangered species in Europe? Development of an European sustainable quality agenda for photo journalism.” Photo journalists from all over Europe gathered to discuss their profession, and what they see as its possible decline.

Although they were a critical audience, it was still very interesting to hear the photographer’s point of view on initiatives like The Commons on Flickr and big digitalization projects like “Images for the Future”. Although, there was a general agreement about the fact that digitalization should be done to preserve historical photo collections. There was much less agreement about how to handle the copyright issues. It is clear that  solutions in the course of general licensing need to be found.

What was striking to me, was that photographers and copyright holders, who were present, were not up-to-date with the general licensing methods and Open Content initiatives like Creative Commons. The big learning for me was that it is good to bear in mind that archives and heritage institutions can benefit from maintaining a regular dialogue with photographers and copyright holders. This dialogue will allow both parties to inform each other about these sorts of initiatives and enable them to work together in finding solutions for the copyright issues, encountered by big digitalization projects.

If you would like to find out more about my talk you can find my presentation at SlideShare:

In the meantime you can still see all our photographs at

Maaike Toonen

Copyright specialist, National Archive “Beelden voor de Toekomst”

The Wisdom of the Crowds in the Audiovisual Archive Domain

BCK – social tagging by pulguita (CC-BY-SA)

Our consortium partner the Dutch National Archives recently joined Flickr: The Commons. Within Flickr: The Commons, leading archives across the globe upload items from their collections to Flickr and invite visitors to add tags and comments to it. This has been a major success: in six weeks the 500 photo’s of the National Archives have been viewed 600.000 times and 1200 tags have been added. Putting material out in the open like the Dutch National Archive did at Flickr raises questions. Are general users qualified enough to complete or even replace the annotations made by archivists? Who is responsible for the outcome of the annotation process? How do we motivate users to annotate the material? These questions partly remain unanswered. This article tries to shed some light on possible following directions.

Users nowadays create their own content. In a recent lecture at the University of Toronto, David Weinberger sees the Web 2.0 as a radical change in thinking about information. There is no longer one truth provided by one source, but instead there’s an ecosystem of truths provided by many. This ecosystem is constantly changing and evolving. Content becomes similar to connection and metadata becomes data.[1] Users are no longer passive visitors of websites but active creators. They give new meaning to existing information by linking different sources and using personal preferences to arrange information.

Social Archives?
The change in the perception of the Internet as a medium has a lot of opportunities for Images for the Future. Digitalizing the archival material takes a lot of time, money and manpower. The digitised material has to be accessible for users. This doesn’t only include the presentation of the material through different services but also the metadata, in order to make it searchable and to add context. The creation of metadata is a very labor-intensive process and not very efficient when its solely done by archivists. Data mining technology might be a solution. Also, archives are exploring how they can put the ‘wisdom of the crowds’ into use.[2]

One of the practices of social software the consortium is interested in is social tagging. Tagging allows users to label different forms of content and can also be used by other people to search content. There are various ways to show tags, like for instance a tag cloud where font size indicates the number of times the tag is used.

One of the tasks of an audiovisual archive is to arrange the information and to embed it in a context. Because of social software, the role of an archive is changing. Annemieke de Jong from Sound and Vision describes these changes in the article Users, Producers & Other Tags. Instead of producing the metadata, documentalists and archivists increasingly classify and correct the metadata produced by others. A part of the metadata arise during digitalization, other metadata can be created by outside experts, crowdsourcing (an open call to an undefined group of people) and social tagging. According to De Jong annotation by users can save time and money. “Free tagging by the general public could be of enormous help in making our collections accessible, on clip level and from multiple viewpoints.”[3]

Experts vs. the General User
There is a tension between the traditional annotation system and the social tagging system. Although the phrase social suggests a community spirit among users, most of them are driven by self-interest.[4] Social taggers use tags primarily to save information that is relevant for their own purpose and add every kind of tag they like. This creates a folksonomy, a free-form system of tags modified by many users. Archivists on the other hand are experts who use metadata from a thesaurus, or a closed vocabulary. The main goal is to make the information accessible for others, not for themselves.  A thesaurus is usually thoroughly designed by a small group of people.

Folksonomies are based on the principal of the ’wisdom of the crowd’. If a lot of people share the same opinion it must be a correct one. Quality is defined by the majority and not by expertise. Wikipedia is build on this assumption. Articles contain general information and there is a lack of value of expert opinion. To Weinberger, wisdom of the crowds is more credential than the wisdom of one, if the process of the creation of the content is visible to users.[5] Not everybody involved in Web 2.0 shares this opinion. One of the founders of Wikipedia, Larry Sanger, wanted more possibilities for expert contribution. He therefore started a new open encyclopedia, Citizendium, where experts have more authority.[6] This example shows that everyone does not always support the wisdom of the crowds. Social tagging challenges the notion of quality and the value of expert opinion. De Jong states that social tagging should not be a substitute for the annotation by experts. Instead there should be an exchange of information between the two systems.[7]

The different nature of the two systems makes it hard to combine them. Sound and Vision is involved in several projects that research the possibilities to combine both systems. The institute participates in the consortium MultimediaN. One of the projects is Spiegle, an alternative search engine for Google. Spiegle combines various levels of metadata like the user profile, the platform and the features of a collection. Unlike Google, this search engine provides very narrow results, which makes it easier for users to find the right content. Also, Sound and Vision participated in the project Total Content Recommendation with the Telematica Institute in Enschede. The project explored the possibilities of social tagging of audiovisual content and the incentives of users to tag the content. Sound and Vision also participates in PrestoPrRIME, a project in the 7th Framework programme of the EU for the research and development of long-term preservation of new media, which will start in January 2009. One of the goals of this project is to establish interoperability between various databases by enabling information exchange between different systems of metadata. Different forms of annotation are connected with each other, creating a semantic system of tags. Semantic tagging makes the annotation process much more efficient in the long term and eventually bridges the gap between different annotation systems.

Motivation of Users
The success of social software has two reasons. It creates weak ties between users and it operates in an open and free environment. Because of the openness of the software, users can choose the level of participation in a community. [8] Users have to be motivated to participate very actively.  During the Total Content Recommendation project Lex van Velsen and Mark Melenhorst of the Telematica Institute  have done research after the incentives of users to tag video content. Following the classification of Cameron Marlow, research scientist at Facebook, they define six different incentives:

  1. Future retrieval
  2. Contribution and sharing
  3. Attract attention
  4. Play and competition
  5. Self presentation
  6. Opinion expression[9]

The authors tested these motives with two groups. Although the groups were very small, the authors conclude users didn’t tag for play and competition and for self-presentation. The tagging sites that were tested didn’t have a game element so it was very likely users didn’t use those sites for play and competition.

Someone who has done a lot of research about crowdsourcing and games is Luis von Ahn, a professor of Computer Science at Carnegie Mellon University. His research is based on the assumption that “computers still don’t possess the basic conceptual intelligence op perceptual capabilities that humans take for granted.”[10] Computers aren’t able to solve problems that are relatively easy solved by most humans. Von Ahn calls this human computation. The human brain can be treated as a processor in a distributed system that can perform a small part of a massive computation. To become part of this massive computation, they do require incentives to solve these kinds of problems, like a game.[11]

An example is the ESP Game. In this game, two random players see the same image they need to label. Goal of the game is to use the same label as your partner. Von Ahn also developed a game called Peekaboom to determine the place of an object within an image. Other games he developed are Verbosity, a game for collecting commonsense facts and Phetch a game for collecting image descriptions for visual impaired.[12] Other researchers have developped games to collect metadata for audio content, like the Listen Game,Tag-a-Tune and MajorMiner. The Listen Game is based on a list of tags players can use to annotate the material. The other two games are similar to the ESP Game: the added tags are compared with those from a database or a different player. In both games players are free to use any tag they like.[13] There are also a few tagging initatives based on video content. Yahoo’s Video Tag Game is based on the same principle as the ESP Game. Players earn points by adding similar tags.[14] This game, developed by Yahoo research, is still in an exploration stadium. The game VideoTag, developed by Stacey Greenaway – as an Msc research project – Is already operational as a single-player game where players collect points adding tags. Some tags are pitfalls (tags that are too obvious) that lower the players score. All these tagging games are examples to generate metadata in a playfull way. Von Ahn sees great possibilities for these kind of games. However, a game designed to solve a problem should produce the right solution and be fun at the same time.[15]

Future Archives
The future archive has to be an open archive to survive the transition from Web 1.0 to Web 2.0 and beyond where users are able to use and label content for their own purpose. Archivists still have a role in evaluating and contextualizing the metadata created by general users. In order to stimulate the creation of different forms of metadata archives could develop creative concepts like games, to encourage users to create new metadata. Until now a lot of research has been done on gathering metadata from general users. Quantity seems more important than quality. Most of the games are designed after the ESP Game. Like von Ahn stated, tagging games should provide the right solution. Further research should focus on the right solution. But what is the right solution? Is it the metadata created by experts, or is it the wisdom of the crowd. And if we know the right solution, how do we control the tagging process to get it? Are players able to provide the right solution or is it necessary for archives to check the metadata that is produced? If that’s the case is a tagging game profitable enough? Further research should focus on these questions in order to gain more insight in the possibilities social tagging has for archives. Sound and Vision will release a video tagging environment early 2009 in collaboration with the Free University Amsterdam and the broadcaster KRO in order to answer some of the questions raised above.


  • Ahn, L. von.  “Games with a Purpose,” Computer (Vol. 39:6, June 2006). pp. 92-94. URL
  • Jong, A. de. “Users, producers & other tags. Trends and developments in metadata creation.” Lecture at the FIAT/IFTA conference (October 2007) URL
  • Mechant, P. “Culture ‘2.0’: Social and Cultural Exploration through the use of Folksonomies and Weak Cooperation.” Cultuur 2.0. (Amsterdam: Virtueel Platform, 2007). URL
  • Lusenet, Y de. Geven en nemen. Archiefinstellingen en het sociale web. (Den Haag: Taskforce archieven, 2008).
  • Siorpaes, K. & Hepp, M. “Games with a purpose for the semantic web.” Intelligent Systems (Vol 23:3, May 2008) pp. 50-60. URL
  • Surowiecki, J. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. (New York: Random House, 2004).
  • Turnbull, D. e.a. “Five approaches to collecting tags for music.” ISMIR Conference (2008). pp.225-230. URL
  • Velsen, L. van & Melenhorst, M. “User Motives for Tagging Video Content.” (2008). URL
  • Wartena, C & Brussee, R. “Instanced-based mapping between thesauri and folksonomies.” Proceedings of the 7th International Semantic Web Conference (ISWC’08) (2008). URL
  • Weinberger. D. “Knowledge at the End of the Information Age.” Bertha Bassam Lecture at the University of Toronto (2008). URL
  • Zwol, R. van. e.a. “Video Tag Game.” 17th International World Wide Web Conference (WWW developer track) (ACM Press, 2008).

[1] Weinberger, D. (2008)
[2] Surowiecki, D. (2004)

[3] Jong, A, de. (2007).

[4] Velsen, L. van & Melenhorst, M. (2008) p.2.

[5] Weinberger, D. (2008)

[6] Lusenet, Y. de. (2008) p.19-20.

[7] Jong, A, de (2007).

[8] Mechant, P. (2007) p. 24.

[9] Velsen, L. van & Melenhorst, M. (2008) p.2.

[10] Ahn, L. von. (2006) p.96.

[11] Ibid. p.96.

[12] Siorpas, K. & Hepp, M. (2008) p. 51.

[13] Turnbull, D. e.a. (2008) p. 227.

[14] Zwol, R. van. e.a. (2008) p.1-2.

[15] Ahn, L. von. (2006) p.96-98.

Library of Congress releases report on Flickr pilot

After 9 months The Library of Congress (LoC) released a detailed report on their Flickr pilot. In January 2008 the LoC and Flickr launched Flickr Commons. They uploaded a few thousand historical photos which have drawn more than 10 million views, 7,166 comments and more than 67,000 tags, according to the new report from the project team. The project had an unexpected impact:

“The pilot spurred many positive yet unexpected outcomes—especially Flickr members’ willingness to devote great effort to photo-related detective work and their level of engagement with historical images. Further, Flickr members have often drawn on personal histories to connect with the pictures, including memories of farming practices, grandparents’ lives, women’s roles in World War II, and the changing landscape of local neighborhoods”


Photo: Library of Congress, Germany Schaefer, Washington AL (baseball), 1911.

If you want to read in more detail how the LoC organised and experienced the pilot, you can download the whole Flickr report LoC here.

German Federal Archive publishes photos on Wikipedia under Creative Commons license

On December 6th, the German Federal Archive and the online encyclopedia Wikipedia announced their cooperation in making publicly available 100,000 digitized images under Creative Commons licence (CC-BY-SA) in exchange for linking the photos to Wikipedia’s Persondata. A big step for opening up public content and data.

The commons

In September 2007 the German Federal Archive already made 113,000 images available on their own online digital archive. In total the Federal Archives keeps approximately 11 million still pictures, aerial photographs and posters from modern German history. The cooperation with Wikipedia is the next big step for the German Federal Archive in opening up the archive, as the vice president of the German Federal Archive Dr. Angelika Menne-Haritz said during the press conference.

Photo: Raimond Spekking, Creative Commons Attribution ShareAlike 3.0.

The photos are not of the highest resolution, about 800 pixels on the longest side. But, this is an enormous addition to the commons. According to Wikimedia, the repository of free content images, sound and other multimedia files on Wikipedia, the donation by the German Federal Archive of 100,000 images is the single largest one to Wikimedia Commons so far. This is even more than the archival project Flickr Commons makes available now in cooperation with 16 archival partners around the world.

Click here for the image gallery:


Photo: Mitglieder des Deutschen Reichstag, German Federal Archive (1889). Author: Braatz, Julius. Creative Commons Attribution ShareAlike 3.0.

Creative Commons License

The images by the German Federal Archive are licensed Creative Commons Attribution ShareAlike 3.0 Germany License (CC-BY-SA). This means that you are free to share and remix the images under the condition that you give attribution and spread this with a similar or compatible license. The Federal Archive can do this because they own sufficient rights on the images to be able to grant this kind of license. To use such a free license for archival material is really exciting. Few archives work with Creative Commons licences. One of the rare examples is the McCord Museum and the Brabants Historisch Informatiecentrum. And, the archival project Flickr Commons works with “no known copyright restrictions”.


The other part of the cooperation between the German Federal Archive and Wikipedia is a tool for linking people from a list compiled by the Federal Archive to the German Wikipedia Persondata and to the person authority file of the German National Library. Something German Wikipedia has already been doing since 2005. Around 27% of 100,000 photos is already done. The expectation is that because the cooperation is now public, the tempo will speed up. Moreover, the users will add new information to the images. You can find the To Do list here.


Though projectleader Creative Commons Germany, Markus says that this is only a small revolution for German notions, this could very well set an example for other archives to make their content publicly available and therefore grow bigger. It will be very interesting to see where we can find the photos and in which (rich) context. Because that will make a strong argument for archives to experiment with this.