Within the context of the Images for the Future project the Netherlands Institute for Sound and Vision is – as mentioned before on this weblog – exploring potential of mobile and location-based access to digital heritage. The goal is to explore how digital heritage can be combined with mobile technology and location awareness to offer an augmented reality (AR) to different types of end users. In this context AR functions by augmenting the physical reality of end users with contextual information using digital means.

As part of this exploration Sound and Vision partnered with the V2_ Institute for the Unstable Media to initiate a multidisciplinary research project (later named ClosAR). The goal of the collaboration was explore new approaches to bring the audiovisual (AV) archive closer to its audience using innovative methods of presentation and interaction, such as augmented reality. The ClosAR group is comprised of five members with a background in new media studies, art science, industrial design engineering, and virtual games.

In their final report (you can download a PDF version here) Bas Bergervoet, Kate Cunningham, Aestha van Dam, Shauna Jin and Connie Yeh propose seven concept directions as possible starting points for Sound and Vision and V2_, with references to the background research and intermediate process:

The classic archive is static: a tomb where “official” documents, media, and information collect dust. Digital technology opens opportunities for information access and presentation, and most notably, participation. As the archive only exists to be accessed, the relationship between itself and its audience is quite important. Augmented reality (AR) brings dynamic information into the physical world. While Beeld en Geluid seeks to improve its visibility outside of its “Experience” and on- site archive search through mobile access, we explored the off-site extreme. How does moving the context of the archive open up possibilities for new interactions? What is the minimum amount of technology needed to achieve this?

The evaluation report provides evidence that crowdsourcing video annotation in a serious, social game setting can indeed enhance retrieval of video in archives. It features success factors organizations need to take into account in setting up services that aim to actively engage their audiences online. The main conclusions are listed below:

Popularity

Waisda? managed to attract a large audience. Since its launch on the 19^th of May 2009 Waisda? was consulted by 9.198 unique visitors and gathered 340.551 tags describing 604 items, added by a total of 2.296 players. The website was consulted 12.297 times. 3,61 pages were visited on average. Only 38% of the visitors didn’t look further than the home page. Average game sessions lasted for 6 minutes and 45 seconds. On the 3th of November 2009) 42.068 unique tags have been added. The total amount of tags added by players is 340.551, of which 40,3% (137.421 tags) consists of matching tags (tags added by two more players within a time frame of 10 seconds).

The majority of players (1.051, or 45,8%) added between one to ten tags. A smaller number of players (810, 35,3%) added between ten and a hundred tags, and less then half of that number (372, 16,2%) added between a hundred and a thousand tags. Only a few players added more than a thousand tags (63, 2,7%), but together were responsible for adding the largest number of al contributed tags. The longest session lasted about three hours, in which one player added 3.329 tags. This indicates that a project like Waisda? shouldn’t only aim for a wide audience, but should also find a way to specifically target these ‘super taggers’.

More than 70% of the traffic on the website was generated through referrals by external websites. The three main referring websites also resulted in the lowest bounce rate, suggesting that visitors that arrive at the website through an external link are more specifically interested in the content and the project than direct visitors. Also, increases in the number of registered players are strongly related to promotional activity. Lastly there is a clear relation between the most popular and heavily tagged content, and the efforts by the Dutch public broadcaster – and project partner – KRO to promote playing Waisda? with this content through their – very popular – programme website. These results show the importance of extensive external promotion of a project like this, aimed at relevant target groups. This implies that Waisda? should target these audiences through existing and popular channels related to the content available within the game, or communities interested in tagging and innovative projects within the cultural heritage sector. For a continuation of the project, and to have video labelling as a standards service, it is important to actively collaborate with websites from broadcasters delivering content for Waisda?. For example by posting an article to notify potential players, an explicit call to action or a contest.

Earlier research (see “Literature” below) has shown that altruism is an important motivation for playing Waisda?. Therefore the ‘about’ section of the website should emphasize the benefits of player activity to the (public) accessibility of the content and further research on tagging. The current research on tagging shows that taggers that are explicitly invited to help an institution by tagging, are notably more active. To further promote Waisda? a strategy that targets these altruistic players should be developed. Besides that, players should be given a sense of the impact of their activity by experimenting with ways to demonstrate the usefulness of the tags for searching through the content.

Apart from altruism, the evaluation showed that the video content itself has also proven to be a motivational factor for players to play the game. The most popular channel on Waisda? contains a popular Dutch reality show with a weekly viewing audience of millions. To attract a broad constituency of users, it is important to expand the diversity of the content available on Waisda?, and experiment with different types of content. Research has shown a particular interest from users in popular talk shows reflecting on recent events, programmes aimed at children and historical footage. Also, it is important to keep the content fresh. For example, at the moment there are already 29 items that contain over 2.000 tags.

Although Waisda? can be played in solitude (against so-called bots), user research has shown that the vast majority of players prefer playing against others. This shows the importance of a substantial and active community of players. Next to the abovementioned promotion on external websites, organizing a contest on Waisda? has shown how handing out prizes can motivate players. It is certainly worthwhile to further experiment with this. Apart from that, social media can also play an important role to position Waisda? as a serious social game, i.e. linking to existing social networks such as Facebook, Twitter and Hyves. The integration with Twitter is already carried out within the pilot.

Game and Interface Design

Studying the latest literature on social tagging has shown that most of the decisions that were made during the design of Waisda? correspond with recommendations from renowned experts and results from related studies. However, there are also some suggestions for improvements. User research has shown that people experience the website as being clear and synoptic. The majority is also positive about the layout of the website. Still, a few things need more explicit clarification. For instance a small portion of visitors do not get the goal of the website instantly. To further reduce the bounce rate of the game and motivate players to actually play Waisda?, the homepage needs to be adjusted to better communicate the goal of the game in a single glance.

Most taggers are fairly inexperienced with the temporal aspect of the moving images tagged within Waisda?. Players should be made aware of the fact they can add multiple tags over the course of the whole video. It is also important to point players to the fact that they can still add a tag after the subject has disappeared from the image because they weren’t done typing (the system keeps track of this ‘delay’ during the entry of a tag). It turned out some players felt the scoring mechanism to lack transparency. This should be explained more clearly, for example by providing a short instruction video or an example game session. The textual pages on the website do answer most questions players might initially have about Waisda?, but the navigation structure could be made more straightforward.

User research (carried out through an extensive questionnaire and the execution of focus groups) has also shown that the meaning of the high scores on the homepage is not clear. This should be clarified, for example by renaming the high scores. Players would also like to see their actual score next to their ranking within the high scores. Usability testing has shown that the position of the tag input field needs emphasizing, to prevent players from overlooking it. Finally, the user research show players appreciate the overlay tips that appear while playing the game, although they sometimes overlook them and like them to be emphasised more.

Literature studies, user research and practical experience with Waisda? have shown that both intrinsic and extrinsic factors play a role in the motivation of players. The recent literature also supports the initial concept of the Waisda? project, that assumes that a game setting is a good way to motivate people to tag (audiovisual) archive material. This shows it is important to make sure that the game design also motivates players that are not particularly interested in tagging per se, or feel that in general tagging is too much of an effort. Besides that, it is crucial to provide a good game design and game play, so the altruistic players also enjoy Waisda?. Ideally, the intrinsic and extrinsic factors come together in the game and interface design.

User research provided concrete recommendations to improve the game design. The statistics on the personal profiles of the players were appreciated, and players would like to see additions. For instance, users pointed out it would motivate them to see the high scores of players directly above and beneath them, the material they contributed tags to and more detailed statistics on how their scoring history. Apart from this, players also voiced a need for feedback on the activity of their opponents during a game. Also, they would like to see different levels of difficulty introduced to the game. When playing in solitude, players should be able to still receive points in retrospect when another player matches their tags in a later session. Furthermore, players constantly need to be encouraged to keep adding tags, and not only focus on watching the video. Because a majority of the players mentioned they want to work towards a goal, one last thing to think about is for example to introduce time limits. A next version of the game should consider all the abovementioned improvements.

Tag Analysis and Further Research

Analyses of the most recent database dump of tags shows that 5,8% of the tags match with the terms in the GTAA thesaurus the Netherlands Institute for Sound and Vision uses to classify their collection. Apart from this, 23,6% corresponds with Cornetto, a linguistic database that contains the bulk of all official Dutch words. Since only a small number of tags is present in both databases (1.135, or 2,7%), it can already be assumed that almost a third of the tags is an existing and correct word.

A professional senior cataloguer (employee of the Netherlands Institute for Sound and Vision) has judged the tags added to two episodes on their usefulness. The selected episodes were the best-tagged episode (from the popular Dutch reality show mentioned before, with 19.322 tags added to it) and an episode that was tagged with an averaged number of tags (a documentary series about a former news correspondent situated in the U.S. returning to the Netherlands with 738 tags).

• Looking at the best-tagged episode 45% of the tags were deemed useful with 27,45% having a low and 11,76% having a high accuracy.
• The averagely tagged episode contained 72,69% tags deemed useful with 26,39% having a low and 19,44% having a high accuracy.

The senior cataloguer noted that in general the useful tags describe the material in a different way than keywords that catalogues add do. Firstly because the tags focus on describing what is seen and heard within a programme, while the professional metadata for audiovisual content focuses on the subjects that a programme refers to. Apart from that, the tags also describe instances from a programme, instead of a logical segmented part and or entire episode. The fact that the crowdsourced tags for the audiovisual material differ from professional metadata is no surprise, and possibly even an indication that the tags contribute to bridging the semantic gap (the gap between professional descriptions, based on a closed vocabulary, and the free search terms used by potential audiences to find the material). However, further research on the usefulness of the tags is needed, for example by conducting search experiments with different types of end users.

To describe the episode as a whole, only two tags from the top 20 most added tags of the averagely tagged episode proved to be useful. For the best-tagged episode none of the top 20 tags were deemed useful to describe the complete episode. Tags added to the documentary series episode were notably more often useful than tags added to the reality show. They were more defining and specific. The reality show contained more general tags and lacked specificity. These findings contradict the assumption that the more a tag is added to an episode, the higher the usefulness of this tag is to the audiovisual archive. The content seems to influence the specificity of the tags that are entered. It is also striking that in the case of the reality show more tags correspond with the GTAA or Cornetto database, but still the tags added to the documentary series episode were deemed more useful by the professional senior cataloguer. This suggests that when a programme contains a multitude of specific items or topics, this might result in more specific and useful tags. The way content seems to influence the way it is tagged demands further research. Since the metadata for audiovisual collections mostly only describe collections on an item level, time-based metadata like tags can result in an important progress in servicing media professionals looking for specific fragments. It is therefore important to further develop this research to discover how and to what degree the tags can be used within the professional metadata in the catalogue of the Netherlands Institute for Sound and Vision.

Within the PrestoPRIME European research project the Web & Media Section of the VU University Amsterdam will investigate how additional technology (such as the use of vocabularies and auto-completion) can help to increase the quality of the contributions. Their research will focus on topics like ambiguity, synonyms, narrower and broader terms, typos, tag overflow, profanity, and will also develop more advanced methods to analyse the (usefulness of) the tags. Also, more research will be carried out to examine the possible implications of the tags in relation to the professional metadata in the catalogue of the Netherlands Institute for Sound and Vision. The VU will also look at possible improvements of the Waisda? Game play and interface design. The Netherlands Institute for Sound and Vision also participates within PrestoPRIME and will hence be closely involved. Currently, the future of Waisda? is examined. Later this year, an improved version (taking on board the results of the evaluation as well as additional functionalities proposed by the VU) will be launched.

For more information on Waisda?, please contact Maarten Brinkerink.

Literature

Ahn, Luis von en Laura Dabbish. “Labeling Images with a Computer Game.” In CHI ’04: Proceedings of the 2004 conference on Human factors in computing systems, p. 319 – 326. New York: ACM Press, 2004.

Ahn, Luis von en Laura Dabbish. “Designing Games with a Purpose.” Communications of the ACM, vol. 51, no. 8 (augustus 2008): p. 58–67. http://www.cs.cmu.edu/~biglou/GWAP_CACM.pdf. Geraadpleegd op 12 oktober 2009.

Ahn, Luis von, Ruoran Liu en Manuel Blum. “Peekaboom: a game for locating objects in images.” In CHI ‘06: Proceedings of the SIGCHI conference on Human Factors in computing systems: p. 55 – 64. New York: ACM Press, 2006.

Brinkerink, Maarten, Johan Oomen en Just Vervaart. Plan van Aanpak. Video Labeling Game, versie 1.0 5 januari 2009. Nederlands Instituut voor Beeld en Geluid en KRO: 2009.

Brinkerink, Maarten en Just Vervaart. Promotieplan Waisda?, versie 4 april 2009. Nederlands Instituut voor Beeld en Geluid en KRO: 2009.

Chan, Sebastian. “Tagging and Searching – Serendipity and Museum Collection Databases.” In Museums and the Web 2007: Proceedings, Toronto: Archives & Museum Informatics, eindredactie door Jennifer Trant en David Bearman. Gepubliceerd 1 maart 2007. http://www.archimuse.com/mw2007/papers/chan/chan.html. Geraadpleegd op 9 december, 2009.

Frank, Anders. “Balancing Three Different Foci in the Design of Serious Games: Engagement, Training Objective and Context.” In Conference Proceedings DiGRA 2007, p. 567 – 574. http://www.digra.org/dl/db/07312.29037.pdf. Tokyo: DiGRA, 2007. Geraadpleegd op 22 september 2009.

Guy, Marieke en Emma Tonkin. “Folksonomies: Tidying up Tags?” D-Lib Magazine, vol. 12, no.1 (januari 2006), http://www.dlib.org/dlib/january06/guy/01guy.html. Geraadpleegd op 17 oktober 2009.

Huurnink, Bouke, Laura Hollink, Wietske van den Heuvel en Maarten de Rijke. “The Search Behavior of Media Professionals at an Audiovisual Archive: A Transaction Log Analysis.” Verwachte publicatie: 2010. Versie 14 december 2009.

Jörgensen, Corinne. “Image Access, the Semantic Gap, and Social Tagging as a Paradigm Shift.” In Proceedings 18th Workshop of the American Society for Information Science and Technology Special Interest Group in Classification Research, eindredactie door Joan Lussky. http://dlist.sir.arizona.edu/2064/. Geraadpleegd op 17 oktober 2009.

Keen, Andrew. The Cult of the Amateur: How Today’s Internet Is Killing Our Culture and Assaulting Our Economy. London: Nicholas Brealey Publishing, 2007

Kennerly, David Ethan. “Fun is Fine. Toward a Philosophy of Game Design.” David Ethan Kennerly. http://finegamedesign.com/fun_is_fine.html (gepubliceerd op 22 juni 2003). Geraadpleegd op 2 oktober 2009.

Koster, Raph. “The Core of Fun.” Keynote op The O’Reilly Emerging Technology Conference 2007, San Diego, California, Verenigde Staten, 28 maart 2007. Volledige video op: http://conferences.oreillynet.com/cs/et2007/view/e_sess/13595.

Leason, Tiffany. “Steve: The Art Museum Social Tagging Project: A Report on the Tag Contributor Experience.” Paper gepresenteerd op Museums and the Web 2009 Conference, Indianapolis, Indiana, Verenigde Staten, 15-18 april 2009. http://www.archimuse.com/mw2009/papers/leason/leason.html#. Geraadpleegd op 7 oktober 2009.

Malone, Thomas M. “Heuristics for Designing Enjoyable User Interfaces: Lessons from Computer Games.” In Proceedings of the Conference on Human Factors in Computing Systems 1982, p. 63 – 68. New York: ACM Press, 1982.

Marlow, Cameron, Mor Naaman, Danah Boyd, en Marc Davis. “Ht06, Tagging Paper, Taxonomy, Flickr, Academic Article, To Read.” In HYPERTEXT ’06: Proceedings of the Seventeenth Conference on Hypertext and Hypermedia, p. 31 – 40. New York: ACM Press, 2006.

Melenhorst, Mark, Jan Grootveld, Mark van Setten en Mettina Veenstra. “Tag-based information Retrieval for Educational Videos.” EBU Technical Review, Q2 (2008). http://tech.ebu.ch/docs/techreview/trev_2008-Q2_social-tagging.pdf. Geraadpleegd op 16 oktober 2009.

Moortgat, Judith. “Taking Pictures to the Public. Evaluatieverslag Nationaal Archief & Spaarnestad Photo op Flickr The Commons.” Nationaal Archief, 8 juni 2009, versie 1.0. http://www.nationaalarchief.nl/images/3_16370.pdf. Geraadpleegd op 15 september 2009.

Peterson, Elaine. “Beneath the Metadata: Some Philosophical Problems with Folksonomy.” D-Lib Magazine, vol. 12, no.11 (november 2006), http://www.dlib.org/dlib/november06/peterson/11peterson.html. Geraadpleegd op 17 oktober 2009.

Robertson, Stephen, Milan Vojnovic, en Ingmar Weber. “Rethinking the ESP Game.” In CHI EA ’09: Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, p. 3937 – 3942. New York: ACM Press, 2009.

Ryan, Richard M. en Edward L. Deci. “Intrinsic and extrinsic motivations: Classic Definitions and New Directions.” Contemporary Educational Psychology, vol. 25, issue 1 (januari 2000): p. 54 – 67. http://www.psych.rochester.edu/SDT/documents/2000_RyanDeci_IntExtDefs.pdf. Geraadpleegd op 12 oktober 2009.

Seftel, Josh en Susan K. Lewis. “Profile: Luis von Ahn. Games with a Purpose.” NOVA, Science Now. Interview gehouden op 6 november 2008. http://www.pbs.org/wgbh/nova/sciencenow/0401/04-gwap-nf.html. Geraadpleegd op 30 november 2009.

Smeulders, Arnold, Marcel Worring, Simone Santini, Amarnath Gupta en Ramesh Jain. “Content Based Image Retrieval at the End of the Early Years.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no 12 (2000): p. 1349 – 1380. http://www.science.uva.nl/research/publications/2000/SmeuldersTPAMI2000. Geraadpleegd op 15 oktober 2009

Springer, Michelle, Beth Dulabahn, Phil Michel, Barbara Natanson, David Reser, David Woodward, en Helena Zinkham. “For the Common Good: The Library of Congress Flickr Pilot Project. Report Summary.” Library of Congress, versie 30 oktober 2008. http://www.loc.gov/rr/print/flickr_report_final_summary.pdf. Geraadpleegd op 15 november 2009.

Springer, Michelle, Beth Dulabahn, Phil Michel, Barbara Natanson, David Reser, David Woodward, en Helena Zinkham. “For the Common Good: The Library of Congress Flickr Pilot Project.” Library of Congress, versie 30 oktober 2008. http://www.loc.gov/rr/print/flickr_report_final.pdf. Geraadpleegd op 15 november 2009.

steve.museum. “Guide to Term Review: Instructions for steve Term Reviewers.” steve.museum, versie 2 januari 2008. http://steve.museum/index.php?option=com_docman&task=doc_download&gid=16. Geraadpleegd op 17 december 2009.

Trant, Jennifer. Tagging, Folksonomy and Art Museums: Results of steve.museum’s Research – 31 January 2009. steve.museum / Archives & Museum Informatics: 2009. http://conference.archimuse.com/files/trantSteveResearchReport2008.pdf. Geraadpleegd op 10 oktober 2009.

Velsen, Lex van en Mark Melenhorst. “Incorporating user motivations to design for video tagging.” Interacting with Computers, vol.21 no.3 (juli 2009), p. 221 – 232.

Vossen, Piek, Isa Maks, Roxane Segers en Hennie van der Vliet. “Integrating lexical units, synsets and ontology in the Cornetto Database.” In Proceedings of LREC-2008, Marrakech: 2008. http://www2.let.vu.nl/oz/cltl/cornetto/docs/vossen-lrec08.pdf.

Wikipedia. “Game design.” Wikipedia. http://en.wikipedia.org/wiki/Game_Design. Geraadpleegd op 30 september 2009.

The commons

In September 2007 the German Federal Archive already made 113,000 images available on their own online digital archive. In total the Federal Archives keeps approximately 11 million still pictures, aerial photographs and posters from modern German history. The cooperation with Wikipedia is the next big step for the German Federal Archive in opening up the archive, as the vice president of the German Federal Archive Dr. Angelika Menne-Haritz said during the press conference.

Photo: Raimond Spekking, Creative Commons Attribution ShareAlike 3.0.

The photos are not of the highest resolution, about 800 pixels on the longest side. But, this is an enormous addition to the commons. According to Wikimedia, the repository of free content images, sound and other multimedia files on Wikipedia, the donation by the German Federal Archive of 100,000 images is the single largest one to Wikimedia Commons so far. This is even more than the archival project Flickr Commons makes available now in cooperation with 16 archival partners around the world.

Click here for the image gallery: http://www.bild.bundesarchiv.de/

Photo: Mitglieder des Deutschen Reichstag, German Federal Archive (1889). Author: Braatz, Julius. Creative Commons Attribution ShareAlike 3.0.

Creative Commons License

The images by the German Federal Archive are licensed Creative Commons Attribution ShareAlike 3.0 Germany License (CC-BY-SA). This means that you are free to share and remix the images under the condition that you give attribution and spread this with a similar or compatible license. The Federal Archive can do this because they own sufficient rights on the images to be able to grant this kind of license. To use such a free license for archival material is really exciting. Few archives work with Creative Commons licences. One of the rare examples is the McCord Museum and the Brabants Historisch Informatiecentrum. And, the archival project Flickr Commons works with “no known copyright restrictions”.

Persondata

The other part of the cooperation between the German Federal Archive and Wikipedia is a tool for linking people from a list compiled by the Federal Archive to the German Wikipedia Persondata and to the person authority file of the German National Library. Something German Wikipedia has already been doing since 2005. Around 27% of 100,000 photos is already done. The expectation is that because the cooperation is now public, the tempo will speed up. Moreover, the users will add new information to the images. You can find the To Do list here.

Conclusion

Though projectleader Creative Commons Germany, Markus says that this is only a small revolution for German notions, this could very well set an example for other archives to make their content publicly available and therefore grow bigger. It will be very interesting to see where we can find the photos and in which (rich) context. Because that will make a strong argument for archives to experiment with this.

“These images can become part of the artistic process, in a society which wants to recycle its heritage, which likes to reuse old images in order to realise something new”

There are interesting initiatives going on to give better access the material. Film Archives Online for example gives free access to catalogue information of film archives from all over Europe, via a multi-lingual web portal. At the Moving Images Archive you can view around 2000 films from the Prelinger Archives. But, do these facilitate young filmmakers in reusing the archives enough? What do young filmmakers expect from the archives? How can cultural heritage contribute to their work and how would they like to have access? If you have an opinion on this topic, please comment.

Out of some 14 million prints, photographs and other visual materials at the Library of Congress, more than 3000 photos from two of the most popular collections are being made available on the new Flickr page. Including only images for which no copyright restrictions are known to exist.

The Library of Congress is not the first putting their archive on Flickr to gain more visibility and accessibility. A lot of prominant museums already did, like the Brooklyn Museum, the Philadelphia Museum of Art, the Powerhouse Museum and the Smithsonian American Art Museum & the Renwick Museum. They all put their collection in a place where people actually spend more time than on the website of the museum itself.

The Commons
So, would this grand announcement be just a clever marketing trick to get the attention of the internet savvy youth? Maybe. But The Library of Congres doesn’t want to be just another collection on Flickr. They officially partnered with Flickr to move the Flickr community to tag the photo’s and baptised the pilot “The Commons”.

Their goal is to increase exposure to the collections and to facilitate the collection of general knowledge with the hope that this information can feed back into the catalogues, making them richer and easier to search. With this pilot the Library of Congress embraces the “power of the web”, as we can read on their blog:

We’re also very excited that, as part of this pilot, Flickr has created a new publication model for publicly held photographic collections called “The Commons.” Flickr hopes—as do we—that the project will eventually capture the imagination and involvement of other public institutions, as well. From the Library’s perspective, this pilot project is a statement about the power of the Web and user communities to help people better acquire information, knowledge and—most importantly—wisdom.

I hope that this pilot will get the attention of other institutions and encourage them to explore the rules of the web more profoundly. If you take a look at the collection already tagged it is impressive to see that some objects already have 25 tags. Somebody got payed to do this?

For the press release of the Library of Congress pilot on Flickr click here.