The Wisdom of the Crowds in the Audiovisual Archive Domain

Thursday, December 18th, 2008

BCK – social tagging by pulguita (CC-BY-SA)

Our consortium partner the Dutch National Archives recently joined Flickr: The Commons. Within Flickr: The Commons, leading archives across the globe upload items from their collections to Flickr and invite visitors to add tags and comments to it. This has been a major success: in six weeks the 500 photo’s of the National Archives have been viewed 600.000 times and 1200 tags have been added. Putting material out in the open like the Dutch National Archive did at Flickr raises questions. Are general users qualified enough to complete or even replace the annotations made by archivists? Who is responsible for the outcome of the annotation process? How do we motivate users to annotate the material? These questions partly remain unanswered. This article tries to shed some light on possible following directions.

Users nowadays create their own content. In a recent lecture at the University of Toronto, David Weinberger sees the Web 2.0 as a radical change in thinking about information. There is no longer one truth provided by one source, but instead there’s an ecosystem of truths provided by many. This ecosystem is constantly changing and evolving. Content becomes similar to connection and metadata becomes data.[1] Users are no longer passive visitors of websites but active creators. They give new meaning to existing information by linking different sources and using personal preferences to arrange information.

Social Archives?
The change in the perception of the Internet as a medium has a lot of opportunities for Images for the Future. Digitalizing the archival material takes a lot of time, money and manpower. The digitised material has to be accessible for users. This doesn’t only include the presentation of the material through different services but also the metadata, in order to make it searchable and to add context. The creation of metadata is a very labor-intensive process and not very efficient when its solely done by archivists. Data mining technology might be a solution. Also, archives are exploring how they can put the ‘wisdom of the crowds’ into use.[2]

One of the practices of social software the consortium is interested in is social tagging. Tagging allows users to label different forms of content and can also be used by other people to search content. There are various ways to show tags, like for instance a tag cloud where font size indicates the number of times the tag is used.

One of the tasks of an audiovisual archive is to arrange the information and to embed it in a context. Because of social software, the role of an archive is changing. Annemieke de Jong from Sound and Vision describes these changes in the article Users, Producers & Other Tags. Instead of producing the metadata, documentalists and archivists increasingly classify and correct the metadata produced by others. A part of the metadata arise during digitalization, other metadata can be created by outside experts, crowdsourcing (an open call to an undefined group of people) and social tagging. According to De Jong annotation by users can save time and money. “Free tagging by the general public could be of enormous help in making our collections accessible, on clip level and from multiple viewpoints.”[3]

Experts vs. the General User
There is a tension between the traditional annotation system and the social tagging system. Although the phrase social suggests a community spirit among users, most of them are driven by self-interest.[4] Social taggers use tags primarily to save information that is relevant for their own purpose and add every kind of tag they like. This creates a folksonomy, a free-form system of tags modified by many users. Archivists on the other hand are experts who use metadata from a thesaurus, or a closed vocabulary. The main goal is to make the information accessible for others, not for themselves.  A thesaurus is usually thoroughly designed by a small group of people.

Folksonomies are based on the principal of the ’wisdom of the crowd’. If a lot of people share the same opinion it must be a correct one. Quality is defined by the majority and not by expertise. Wikipedia is build on this assumption. Articles contain general information and there is a lack of value of expert opinion. To Weinberger, wisdom of the crowds is more credential than the wisdom of one, if the process of the creation of the content is visible to users.[5] Not everybody involved in Web 2.0 shares this opinion. One of the founders of Wikipedia, Larry Sanger, wanted more possibilities for expert contribution. He therefore started a new open encyclopedia, Citizendium, where experts have more authority.[6] This example shows that everyone does not always support the wisdom of the crowds. Social tagging challenges the notion of quality and the value of expert opinion. De Jong states that social tagging should not be a substitute for the annotation by experts. Instead there should be an exchange of information between the two systems.[7]

The different nature of the two systems makes it hard to combine them. Sound and Vision is involved in several projects that research the possibilities to combine both systems. The institute participates in the consortium MultimediaN. One of the projects is Spiegle, an alternative search engine for Google. Spiegle combines various levels of metadata like the user profile, the platform and the features of a collection. Unlike Google, this search engine provides very narrow results, which makes it easier for users to find the right content. Also, Sound and Vision participated in the project Total Content Recommendation with the Telematica Institute in Enschede. The project explored the possibilities of social tagging of audiovisual content and the incentives of users to tag the content. Sound and Vision also participates in PrestoPrRIME, a project in the 7th Framework programme of the EU for the research and development of long-term preservation of new media, which will start in January 2009. One of the goals of this project is to establish interoperability between various databases by enabling information exchange between different systems of metadata. Different forms of annotation are connected with each other, creating a semantic system of tags. Semantic tagging makes the annotation process much more efficient in the long term and eventually bridges the gap between different annotation systems.

Motivation of Users
The success of social software has two reasons. It creates weak ties between users and it operates in an open and free environment. Because of the openness of the software, users can choose the level of participation in a community. [8] Users have to be motivated to participate very actively.  During the Total Content Recommendation project Lex van Velsen and Mark Melenhorst of the Telematica Institute  have done research after the incentives of users to tag video content. Following the classification of Cameron Marlow, research scientist at Facebook, they define six different incentives:

  1. Future retrieval
  2. Contribution and sharing
  3. Attract attention
  4. Play and competition
  5. Self presentation
  6. Opinion expression[9]

The authors tested these motives with two groups. Although the groups were very small, the authors conclude users didn’t tag for play and competition and for self-presentation. The tagging sites that were tested didn’t have a game element so it was very likely users didn’t use those sites for play and competition.

Someone who has done a lot of research about crowdsourcing and games is Luis von Ahn, a professor of Computer Science at Carnegie Mellon University. His research is based on the assumption that “computers still don’t possess the basic conceptual intelligence op perceptual capabilities that humans take for granted.”[10] Computers aren’t able to solve problems that are relatively easy solved by most humans. Von Ahn calls this human computation. The human brain can be treated as a processor in a distributed system that can perform a small part of a massive computation. To become part of this massive computation, they do require incentives to solve these kinds of problems, like a game.[11]

An example is the ESP Game. In this game, two random players see the same image they need to label. Goal of the game is to use the same label as your partner. Von Ahn also developed a game called Peekaboom to determine the place of an object within an image. Other games he developed are Verbosity, a game for collecting commonsense facts and Phetch a game for collecting image descriptions for visual impaired.[12] Other researchers have developped games to collect metadata for audio content, like the Listen Game,Tag-a-Tune and MajorMiner. The Listen Game is based on a list of tags players can use to annotate the material. The other two games are similar to the ESP Game: the added tags are compared with those from a database or a different player. In both games players are free to use any tag they like.[13] There are also a few tagging initatives based on video content. Yahoo’s Video Tag Game is based on the same principle as the ESP Game. Players earn points by adding similar tags.[14] This game, developed by Yahoo research, is still in an exploration stadium. The game VideoTag, developed by Stacey Greenaway – as an Msc research project – Is already operational as a single-player game where players collect points adding tags. Some tags are pitfalls (tags that are too obvious) that lower the players score. All these tagging games are examples to generate metadata in a playfull way. Von Ahn sees great possibilities for these kind of games. However, a game designed to solve a problem should produce the right solution and be fun at the same time.[15]

Future Archives
The future archive has to be an open archive to survive the transition from Web 1.0 to Web 2.0 and beyond where users are able to use and label content for their own purpose. Archivists still have a role in evaluating and contextualizing the metadata created by general users. In order to stimulate the creation of different forms of metadata archives could develop creative concepts like games, to encourage users to create new metadata. Until now a lot of research has been done on gathering metadata from general users. Quantity seems more important than quality. Most of the games are designed after the ESP Game. Like von Ahn stated, tagging games should provide the right solution. Further research should focus on the right solution. But what is the right solution? Is it the metadata created by experts, or is it the wisdom of the crowd. And if we know the right solution, how do we control the tagging process to get it? Are players able to provide the right solution or is it necessary for archives to check the metadata that is produced? If that’s the case is a tagging game profitable enough? Further research should focus on these questions in order to gain more insight in the possibilities social tagging has for archives. Sound and Vision will release a video tagging environment early 2009 in collaboration with the Free University Amsterdam and the broadcaster KRO in order to answer some of the questions raised above.


  • Ahn, L. von.  “Games with a Purpose,” Computer (Vol. 39:6, June 2006). pp. 92-94. URL
  • Jong, A. de. “Users, producers & other tags. Trends and developments in metadata creation.” Lecture at the FIAT/IFTA conference (October 2007) URL
  • Mechant, P. “Culture ‘2.0’: Social and Cultural Exploration through the use of Folksonomies and Weak Cooperation.” Cultuur 2.0. (Amsterdam: Virtueel Platform, 2007). URL
  • Lusenet, Y de. Geven en nemen. Archiefinstellingen en het sociale web. (Den Haag: Taskforce archieven, 2008).
  • Siorpaes, K. & Hepp, M. “Games with a purpose for the semantic web.” Intelligent Systems (Vol 23:3, May 2008) pp. 50-60. URL
  • Surowiecki, J. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. (New York: Random House, 2004).
  • Turnbull, D. e.a. “Five approaches to collecting tags for music.” ISMIR Conference (2008). pp.225-230. URL
  • Velsen, L. van & Melenhorst, M. “User Motives for Tagging Video Content.” (2008). URL
  • Wartena, C & Brussee, R. “Instanced-based mapping between thesauri and folksonomies.” Proceedings of the 7th International Semantic Web Conference (ISWC’08) (2008). URL
  • Weinberger. D. “Knowledge at the End of the Information Age.” Bertha Bassam Lecture at the University of Toronto (2008). URL
  • Zwol, R. van. e.a. “Video Tag Game.” 17th International World Wide Web Conference (WWW developer track) (ACM Press, 2008).

[1] Weinberger, D. (2008)
[2] Surowiecki, D. (2004)

[3] Jong, A, de. (2007).

[4] Velsen, L. van & Melenhorst, M. (2008) p.2.

[5] Weinberger, D. (2008)

[6] Lusenet, Y. de. (2008) p.19-20.

[7] Jong, A, de (2007).

[8] Mechant, P. (2007) p. 24.

[9] Velsen, L. van & Melenhorst, M. (2008) p.2.

[10] Ahn, L. von. (2006) p.96.

[11] Ibid. p.96.

[12] Siorpas, K. & Hepp, M. (2008) p. 51.

[13] Turnbull, D. e.a. (2008) p. 227.

[14] Zwol, R. van. e.a. (2008) p.1-2.

[15] Ahn, L. von. (2006) p.96-98.


Dutch create more online content than the European average

Tuesday, November 27th, 2007

Tipped by a post on the weblog I discovered that research conducted by the European Interactive Advertising Association (EIAA) this November showed that the Dutch not only spend more time online and less on watching television, but also that we watch more online video and behave more like prosumers on the internet than the average European.

Some conclusions on media use in the Netherlands
EIAA conducted research in Europe and interviewed 500 people in the Netherlands. A few interesting conclusions:

  • internet penetration in Holland is the biggest in Europe; 81% uses internet weekley, 80% via broadband
  • the Dutch surf 5,6 days average a week
  • a quarter we can find on fora
  • 40% of the Dutch using internet visit social networking sites
  • 26% of the 16-24 year olds spend more time on surfing the internet that watching television
  • since 2006 the amount of 55 plus people and women using the internet rose by 11%
  • the Dutch blog: the amount of people blogging rose by 30% since 2006

Interestingly but not very new, the research shows that there is a clear shift from the traditional media consumption of watching television to watching online in the Netherlands. The amount of Dutch internet users that watch at least once a month online television programmes, films or videoclips rose between 2006 with 133%. 42% watches less TV, 20% listens less radio and 23% reads less often a paper. The shift is a fact.

Building personal archives
Not only do the Dutch ‘passively’ watch video on the internet, the research also concludes that the Dutch are the most active prosumers of Europe. 40% of the Dutch creates online content, whereas the European average is 18%. However this research doesn’t show whether this development is growing fast, you can guess that this will be big in the near future. Camera’s are getting cheaper every day, it becomes regular to have a mobile phone with a camera in it and easy to put it somewhere on the web.

So, everybody can create content when, where and where ever he or she likes. And we do! A collegue of mine just had a baby. In one month you could find more audiovisual content on the web about his life than about mine, and I already exist for 26 years. Research conducted by Ruigrok Netpanel for the Next Web about Web 2.0 in the Netherlands showed that now already 50% of the Dutch people share their photo’s online; and 16% does this with video content.

Personal versus institutional archives?
It seems like there is a tension between the core bussiness of Images for the Future, actually digitizing our audiovisual heritage of the past, and the broad social movement we see in the growth of User Generated Content and thus personal archives. How does this interfere? Or, how can this succesfully come together? How can we unite the world of the archives and the world of the private collections? In Holland there is a project which is trying to open up personal archives; audiovisual material right out of your parents dusty ceiling, this project is called Images of our Past. Why not combine theses personal stories with the official stories which are nationally archived?

Some institutions are already experimenting with this idea. A few Dutch broadcasters already announced plans on working with user generated content platforms. SBS will create it’s own platform like the German ProSieben Sat1 does with Myvideo; NPO will work together with popular social networking site Hyves. These are interesting cases which will show what can be possible and what will be desirible.