Jon Olav H. Eikenes

Connecting motional form to interface actions in web browsing: investigating through motion sketching

This article has been published in the academic journal FORMakademisk, Vol 3, Nr 1, 2010. This web-version is slightly different from the original: a Prezi-presentation and three videos are here embedded in the text, and a table of contents has been added. Download the original PDF from FORMakademisk.

Contents:

Abstract
Introduction
Framework & research method
Case context & concepts
Design experiments and design methods
Analysis of motion sketches
Conclusions
References
Notes

Abstract

It is now possible to include complex visual movement in screen interfaces, including those that enable web browsing on different media devices. This article investigates the potential for employing movement in web browsing – or more specifically, how motional form may be connected to interface actions. The investigation is carried out through design experimentation. Techniques of ‘motion sketching’ have been developed and utilized in a practice-based research project. The resulting motion sketches are analysed as realizations of complex mediation – by drawing on social semiotics and the concept of action from Leont’ev. The article argues that motional form is made meaningful through connotations and experiential metaphors, and suggests ten provisional principles for how motional form may be used in web browsing. This challenges notions of form and function in current interface design and how social semiotic theory may be produced.

Keywords: movement, web browsing, sketching, design experimentation, social semiotics.

Introduction

Digital artifacts permeate our daily life in work and play. We live in an ‘interfaces culture‘ (Johnson 1997) where a range of activities are mediated by screens-based interfaces such as mobile phones, laptops and gaming platforms. These interfaces mediate a range of individual and social activities that take place in ‘information space’ (Benyon 2001), including web browsing, reading, gaming and interpersonal communication. Interfaces are also part of the larger media landscape in which visual movement and computer-generated animation is widely made use of, for example in the ‘animated worlds’ (Buchan 2006) of feature films, TV commercials and gaming.

Designers create expressive interfaces that move in response to user actions. The screens present high-resolution and colourful imagery, video and three-dimensional simulation, combined with real-time generated movement in the interface (Eikenes & Morrison 2010; Skjulstad & Morrison 2005). Movement has a promising potential for communication in interaction design (Petersen & Nielsen 2002), with its ability to attract and lead attention, represent process and change, and through its kinetic aesthetic qualities. Attention in design research has been given to the design of Tangible User Interfaces (TUI) (e.g. Hummels, et al. 2007), but there is still a need to investigate the potentials and shaping of dynamic screen interfaces. This includes the need for a critical vocabulary for describing as well as analyzing movement in the interface. In this article I describe movement as motional form to bring attention to the shaping of movement through design.

Along with the development of dynamic interfaces, the web has become available on the same devices through web browsers.[1] The browser is today the main place where people access digital information as well as write documents and e-mails, and as such it offers an important service to its users. Wright (2009) argues that the browser may eventually replace and undertake the role of the operating system.

Information on the web is constantly changing and moving. Similarly, our activities mediated by web browsers are dynamic in character; we navigate information spaces through multiple and parallel traversals and produce content ourselves. In addition, we increasingly control digital artifacts, such as the Nintendo Wii and the Apple iPhone, through physical movement and gestures. As computers and digital devices are given stronger graphical capabilities, there is a great potential for including motional forms that previously have been reserved for computer games and animation. Motional form has a communicative potential that is poorly understood and underused in the context of interface design in general, and specifically in web browsing. Knowledge of motional form, including a vocabulary for describing it, is important for education, theory and practice of interface design.

The aim of this article is to investigate how motion in the interface may be employed in web browsing. To do so, it has been necessary to develop appropriate techniques for sketching movement. Further, there is a need to investigate how designers may communicate ideas of motion to diverse participants in a project team, and various decision makers involved in software design and development. I introduce motion sketching as a way of investigating how motional form may be connected to activities mediated by interfaces.

The study reported on in this article is carried out from within a practice-based research project in collaboration with Opera Software, a world-leading web browser company. The article presents design experiments that investigate how motional form may be incorporated in the design of an online service provided by this company. The service allows people to collect, organise, share and explore information and media content across multiple technological platforms and devices. Based on these experiments I propose ten provisional principles for how motional form may be used in web browsing.

The practice of the author is based within interaction design and interactive media design. The article adopts a socio-cultural approach to design, bringing attention to the mediating and communicative role of screen interfaces in diverse social and cultural contexts.

The following sections will present the theoretical framework and situate the study as practice-based. The context of web browsing and movement in the interface will be discussed. I then move on to frame the design experiments and design methods, before presenting and analysing the experiments.

Framework & research method

Professional and leisure activities are increasingly enabled and supported by digital technology, mediated by an interface between the human and the digital. This 'human-computer interface’is commonly seen as a layer or set of contact points between a user and a computer system, which allow people to control and understand digital artifacts (Jensen 2005).

The mediating role of the interface is more complex than that of a medium for linear communication between two agents – a user and a system. Bødker & Andersen (2005) point to the double role of the interface in which complex mediation takes place in terms of instrumental mediation and semiotic mediation. Through the interface we do things, like send e-mails or explore the Internet. Here, the interface functions as a tool or instrument in human activity. In addition, the interface is a semiotic artefact made meaningful through the use of signs. However, the relation between tool and sign is complex and not clear-cut.

A socio-cultural view on design acknowledges that interfaces are cultural artefacts that are situated in a social, cultural and historical context (Morrison forthcoming 2010).[2] Such a view draws on Vygotsky (1978), who argued that tool systems as well as sign systems such as language are socially and culturally produced over time. As a consequence, human development is rooted in society and culture. Tools as well as signs have a mediating function between the individual and its environment in human activity and development.

Leont'ev (1978) expanded on the notion of mediation by categorising the internal structure of activities. He provides a three-level model of activity, compromising activities as oriented towards certain motives, actions driven by conscious goals, and automatic operations conditioned by the tools of the action at hand. Following this, I will refer to actions that a user performs through an interface as interface actions. According to Leont’ev, humans carry out actions through a series of operations to realise objective results in the world. Actions are therefore goal-oriented, e.g. opening an application or saving a document.[3]

The interface is a semiotic artefact in which meaning is inscribed through the employment of signs. The field of social semiotics allows us to see that the meanings of artefacts are socially constructed, continuously being negotiated and redefined (van Leeuwen 2005). Social semiotics draws on traditional semiotics as well as systemic-functional linguistics (Halliday 1978). In this approach, signs, or rather semiotic resources, are used to make meaning. Semiotic resources are not fixed, but their potential meanings change as they are used over time.[4] Social semiotics has been applied to a range of semiotic modes and media to understand how they communicate, such as images (Kress & van Leeuwen 2006/1996), typography (van Leeuwen 2006), speech and music (van Leeuwen 1999). It has not been much applied to movement.[5] This article will make use of the concepts of experiential metaphor, connotation and salience from social semiotics (van Leeuwen 2005). These concepts will be explained in detail when applied in the analysis.

For interface and interaction design there is a need to combine the approaches of instrumental mediation and semiotic mediation for which Vygotsky laid the foundations (Morrison forthcoming 2010). This article will investigate how semiotic mediation in terms of motional form may be connected to interface actions in activities of web browsing.

Research through design experimentation

To investigate the possibility of connecting motional form to interface actions in web browsing involves the exploration of potentials that not yet have been realised. Therefore it has been necessary to generate interface actions as well as motional forms through design. This research is therefore situated between the creative and practical on one hand, and the critical and analytical on the other.

Traditional media research has largely analysed media artefacts after they have been designed and put into the world. As Liestøl (2003) argues, the humanities may gain from moving from analysis to also include synthesis – to reveal future potentials. van Leeuwen (1999, p. 10) argues that social semiotics also needs to be extended from analysis of existing texts to expand and unlock semiotic resources, like experimental film-makers did in Soviet Russia in the 1920s. Design experimentation may integrate analysis and synthesis by generating features that have not yet been explored, embed these in artifacts, and at the same time build new analytical concepts.

Being an interaction designer allows me to use my own practice for exploring new mediational possibilities. As a designer-researcher I learn and reflect through designing (reflection-in-action)and on designing (reflection-on-action) (Schön 1983). This is an integrated, generative and dynamic process that oscillates between design and ‘learning through doing’ on one hand, and reflection and analysis on the other. Such an approach contrasts and supplements mainstream studies that focus on testing or analysing existing interfaces, for example in Human-Computer Interaction and media studies. The result of this process is presented in the analysis.
I have worked in collaboration with other designer-researchers as well as a commercial business partner within the context of a practice-based research project called RECORD.[6] This article is based upon a project within RECORD, carried out between the author and Opera Software, a leading web browser company.[7] The project was not directly connected to any of Opera's existing development projects, and was therefore open-ended. The author carried out the design experiments, and we met several times in meetings and workshops to discuss the project as it progressed. Two people from Opera were involved, including an interaction designer.
The design methods and techniques, which also are part of the research method, will be introduced later. First it is necessary to look at the context of web browsing, and especially its current use of motional form.

Case context & concepts

The web browser interface

A web browser provides the possibility of accessing web pages, images, videos and other digital content on the web. Opera is arguably one of the leading browser companies in the world. Their vision is “to deliver the best Internet experience on any device,” and Opera browsers are available for a range of platforms such as computers, mobile phones and PDAs, TVs, and game consoles.[8]

Web browsers were originally developed in the early 1990s to view and edit hypertext documents on the Internet (Berners-Lee & Fischetti 1999). Today, the browser is the interface through which the web is mediated from the users’ point of view. The browser holds a special position, being a platform and interface for web mediation and itself being a software application. Websites are also interfaces, where menus, buttons, maps, indexes and hyperlinks are used as navigational tools (Gloor 1997). However, traditional interface conventions related to navigation and control of websites are different and sometimes in conflict with those of operating systems and their software applications (Müller-Prove 2002).[9] Wright (2009) suggests that the web and the operating system may eventually converge; the web browser may become the new operating system as applications that used to run on the desktop are moving to the web. As applications move to the web, so does also information move from private computers to web servers and social media platforms, popularly referred to as ‘cloud computing’. As a result, large amounts of information and media are available online, and the need for users to archive and manage information found while browsing is even greater than before (Schraefel & Zhu 2001).[10]

Motional form and web browsing

Movement is intrinsic to human life, and is addressed from a range of disciplines as diverse as dance, physics and anthropology. This article focuses on digitally produced movement presented on screens.[11]

In recent years, screen-based movement has become fundamental to visual culture, such as in computer-animated movies (Buchan 2006), virtual reality interfaces, motion graphics (Manovich 2007) and gaming (A. Wood 2007). Design research has focused on human movement and gestures (Hummels, et al. 2007; Loke, et al. 2007), but screen-based movement is still in need of further research.

I will use the term motional form to denote designed changes in visual appearance over time. By using the term ‘form’ I want to bring attention to the fact that motion has to be shaped and designed, much like physical materials are shaped into products. I will focus on motional form that is presented through the screens of digital artifacts. Such forms are mostly composed using programming or animation tools, as opposed to camera-recorded movement (live action). This understanding of movement draws mainly on the tradition of animation, and may include representation of two-dimensional as well as three-dimensional form. Here, form is defined broadly, to denote both ‘arrangement of parts’ and ‘what is directly given to the senses’ (Tatarkiewicz 1980). There are immense possibilities for motional form, leading to vast possibilities for aesthetic expressions and meanings, especially when combined with interactivity and various modes of communication such as images, sound, typography and colour (Jeamsinkul & Poggenpohl 2002). However, the potential is not yet widely adopted in interface design in general, and specifically in the context of web browsing.

From its purely textual origin (Berners-Lee & Fischetti 1999), the web has become multimodal, by the use of different semiotic modes such as images, videos, and sound. It has developed from “a primitive text-based tool into a sophisticated design medium” (Engholm 2007, p. 230). Movement and animation is increasingly employed on the web (Eikenes & Morrison 2010; Skjulstad & Morrison 2005). There are a range of different technologies for achieving this, such as Adobe Flash and Java.[12] Simple animation is also possible to achieve using CSS,[13] and web standards are continuously developing to enable more dynamic website compositions (Wright 2009). Web browsers, on the other hand, have traditionally not included much movement beyond simple page scrolling.[14]

This article will use concepts that Eikenes & Morrison (2010) developed by investigating the intersections of navigation and animation, or navimation, in screen based interfaces. They introduced and applied three concepts to a set of dynamic interfaces: temporal navigation denoting how navigation at a micro level may be continuous when intertwined with motional form; spatial manipulation denoting how movement may create and manipulate the sensation of space in the interface; motional transformation denoting how a visual element may gradually change over time through motion. Eikenes (2009) introduced two new concepts when applying navimation to the context of social media: indexical compositing denoting how a dynamic composition may allude to a cultural setting; virtual kinetics referring to how motional form may create the sensation that graphic elements possess magnetic potential or are located in an environment with gravity. I will use and extend previous work by connecting movement to actions in a specific interface context, and thereby integrate views of semiotic and instrumental mediation.

Design experiments and design methods

The project was carried out in collaboration between Opera Software and AHO, framed within the larger research project RECORD. Several people were involved in discussing the project, with the author designing and producing the experiments. The overall aim was to investigate how motional form could be used in activities of web browsing, such as finding and sharing information. Based on earlier research on navimation and social media (Eikenes 2009), we wanted to investigate the possibility of combining social activities such as sharing information with motional form in the browser interface. We also recognised the need for better tools to individually archive and manage information in the browser (Schraefel & Zhu 2001). These activities are important in web browsing, and therefore it might be argued that the browser should support such activities directly in its interface.

We found it useful to introduce a new social media service that Opera could provide their users. By envisioning a service the experiments could be situated within a defined context, and provide convincing examples of how motional forms could map onto such a service. Further, the service could easily be deconstructed into a set of possible interface actions, providing a structure for generating and arranging a number of motion sketches.

Two parallel design activities emerged during the project: first, to envision a new service for Opera, and second, to map motional form onto interface actions taking place within that service. As this article focuses on mapping motional form to interface actions, the service will only be presented briefly in the following section.

Towards a set of interface actions: Opera Media Collections

With the rise of social media services, cloud computing, and visually dynamic interfaces, it is now possible for browsers to become rich dynamic sites for managing and sharing information. Here, the notion of a collection becomes relevant as a device for managing information individually and socially. We envisioned a new service called Opera Media Collections (OMC), that would allow people to collect, organize, browse, and share specific media items such as images, videos and music, in addition to webpages (Figure 1). The service should be integrated in the Opera web browser to provide easy access during activities of web browsing. It represents a shift from text-based private bookmarking systems towards social collections that are dynamic, visual, and integrated in the web browser.[15]

Opera Media Collections

Figure 1. Diagram showing the underlying system of Opera Media Collections. Users add media items to their collections (1), and may subscribe to and share collections. These collections and media items may then be explored in dedicated interface environments on different platforms (2). (Click to see larger image.)

Users could create a set of collections, into which media items were added while surfing the web through a browser on a PC or mobile device.[16] These collections could be private or public, allowing users to subscribe to other people’s collections, including those provided by professional content providers. Within the web browser, a new and dedicated interface environment could allow people to explore and organize their media items and collections.

Taking the OMC service as a starting point, we identified a set of possible interface actions, such as adding a media item or opening a media collection. The full list of actions is presented in Table 1 below.

Interface actions

Table 1: List of some of the interface actions that could be available within the Opera Media Collections service.

Design experimentation through sketching

According to Gedenryd (1998), sketching is fundamental to the creative processes of design professions such as architecture, graphic design and industrial design.[17] The notions of prototype and sketch are sometimes used interchangeably in interaction design.[18] I follow Buxton's (2007) distinction, in which the sketch is tentative and close to what designers do - to explore ideas, question, and provoke. Prototypes on the other hand, are specific and used to test, refine, and give answers. Buxton argues that the activity of sketching could be extended to other forms than just pencil on paper, as sketches of interaction and experience "…need to be able to capture the essence of design concepts around transitions, dynamics, feel, phrasing, and all the other unique attributes of interactive systems" (2007, p. 136).

The first step towards finding out how motional form could be used in web browsing was achieved by identifying a set of interface actions as given above. Now, the challenge was to investigate how motional form could be connected to these actions. We wanted to explore widely the potential of movement, and produce a range of diverse motional forms rather than to design one coherent environment. It was necessary to find ways to quickly sketch movement as to produce many alternative forms. It was also necessary to find appropriate ways of presenting and communicate these forms. This lead to the question: in what ways may motion be sketched and represented temporally?

‘Motion sketching’

As this investigation is carried out through design experimentation, the sketching techniques become an important component of the overall research method. I propose motion sketching as a general term to bring attention to how motional form may be investigated, produced and communicated. Different strategies may be taken for sketching motion (Vertelney 1989), including drawn representations on paper.[19] Such techniques of motion sketching are quick and efficient, but lack the temporal aspect of motion. I will now describe four techniques that include the temporal dimension.

Motion may be sketched temporally by making physical representations of interface elements, for example of paper, and physically move these elements around in real time. This performance may be recorded using a video camera – what we might call sketching through video. The sketch is then saved for later playback, and the designer can control how and from where the performance is recorded.[20]

It has previously been suggested that interface design may benefit from techniques and features of animation[21] (Baecker & Small 1990). ‘Stop motion’ is a specific technique used in professional animation practice. It makes depicted objects appear to move on their own by starting and stopping the camera, and moving the physical objects in between each recording. Applied here, we may call this sketching through stop motion.
Today, animation is often produced using computer software, the result ranging from TV commercials to music videos and feature films. According to Manovich (2007) software such as Adobe After Effects has had a broad effect on moving image production by creating a new visual language based on the ability to remix all kinds of media content. A range of software tools such as Autodesk Maya, Adobe Flash, and even Microsoft PowerPoint may be used for sketching through computer animation.

Finally, if the designer has sufficient programming skills, motion sketching may be carried out through writing code, in software environments such as Adobe Flash or Processing.[22] This we may call motion sketching through code. The advantage of this technique is the possibility of making motion sketches ‘interactive’ - that they react and change in response to user actions. However, this technique demands programming skills, and may for some limit the creative and open process of sketching.

The different techniques of motion sketching affect what kinds of ideas and motional forms can be realised. For example, motion sketching through video and stop motion are quick and easy to make, but is confined by physical limitations such as gravity and camera perspective. Computer animation is less restricted and provides possibilities for more precise and sophisticated movement, but demands animation skills and professional software, using digital tools that also affect the resulting sketches.

The techniques can be combined in multiple ways, and may include a representation of the user and his actions, e.g. a mouse cursor or fingers touching the screen. There might be other techniques for sketching motion than those described here, and there is a potential for introducing new digital tools to facilitate such sketching.

If the main objective of the sketch is to explore motional form, it may be useful to simplify the sketch graphically to focus on the quality of the movement itself. However, it is important to be aware that the reading of motional form may change drastically when put into different contexts, combined with other modes of communication and made interactive.

Connecting motional form to interface action through motion sketching

After identifying a set of interface actions and finding techniques for sketching motion, it was possible to explore directly how motional form could be connected to web browsing. More than 30 short videos (ranging from 3 to 20 seconds) were produced using the sketching techniques presented above. Short time and few resources made us confine our experimentation to a few interface actions, of which the three most suitable will be presented later.

For organizing and presenting the motion sketches, it was helpful to lay them out in a two-dimensional grid according to the respective interface actions they connected to. The presentation was made with the online tool Prezi,[23] which provides a canvas for composing video, text, images and graphics (Figure 3). The interface actions are laid out along the horizontal axis at the top, while all the motion sketches are placed underneath each corresponding action.


Figure 3: Interface actions are laid out along the horizontal axis at the top, while all the motion sketches are placed underneath each corresponding action. (This is the Prezi presentation - click to explore.)

Visually, this scheme shows the potential for literally and conceptually 'zooming in' and concentrating on a specific area, or translating a motional form from one column to another. The grid is therefore not only a presentation tool, but also a design tool that opens up new possibilities and itself demonstrates the potential of motional form. If to design and implement a specific service for actual use, a strategy could be to choose one or several sketches and from that build up a specific ‘motion palette’ or ‘motion genre’. This could then be applied to several interface actions and platforms such as mobile devices or TVs.

The experiments allowed us to investigate and explore by design how motional form may be used to realise interface actions in web browsing. In addition, the sketches show the potential of motional form through design.[24] The next section will present and analyse some of these sketches.

Analysis of motion sketches

The aim of this section is to analyze how interface actions may be realised through motional form. What kind of resources and potentials for making meaning start to emerge if motional form is connected to interface actions? What kind of roles or ‘functions’ may motional form play in interface actions? The experiments are deliberately diverse in order to explore a wide range of possibilities, and therefore the analysis focuses on different aspects from experiment to experiment. Some of these aspects will be discussed after the analysis. An additional aim here is to develop a vocabulary for describing motional forms.

I will now go through some of the motion sketches produced for three interface actions: (1) add a media item to a media collection, (2) browse between several media items, and (3) open a collection to access the contained media items.

Interface action 1: add a media item to a media collection

A central feature of Opera Media Collections (OMC) is to allow people to gather and assemble media items from websites into collections. This implies a concrete action in which a user adds a media item - such as an image – to a collection, preferably without leaving the website. The aim of the following experiments was to investigate how this action of adding a media item may be supported by the use of visual movement. Video 1 presents five experiments, all of which were sketched through computer animation.


Video 1: Interface action: saving an image to a media collection. 5 experiments, named 1.1 - 1.5. (Click to play).

There are two events connected to the action of assigning a media item to a collection: first, to view the collections that are available, and second, to add the media item to the selected collection. Motional form may be used for both events. Video 1 presents five experiments that show different ways of assigning an image to a media collection.

In Experiment 1.1, a list of collections emerge on top of the image, realised by using techniques of fading and twisting. The list appears on top of the media item, as if emerging out from the image itself. This motional form therefore indicates a connection between the list and the media item – indicating the link that is about to be made from the media item to a collection. In social semiotic terms, this linking may be described as a conjunction. Further, assigning the image to a collection by selecting an item on the list is followed by a quick light flash, indicating and affirming that the action of assigning the image to a collection has been realised. The light flash might be described as a motional transformation (Eikenes & Morrison 2010).

Experiments 1.2 to 1.5 take a different strategy for adding a media item to a specific collection, by allowing the user to move the media item on the screen. When clicking on the image, a copy is made from the original image. The copy is scaled down and moves to visually connect to the mouse pointer through scaling and moving. This motional form indicates that the control of the media item has been given to the user, thereby transferring agency from the interface to the user. This invites the users to control and continuously move around the copy of the image, thereby creating new movement that unfolds directly in response to user action. Such continuous control may be conceptualised as temporal navigation (Eikenes & Morrison 2010). This connection of motional form and interface action may also be understood as an experiential metaphor (van Leeuwen 2005, p. 29). Metaphor is a widely used concept in semiotics; according to Lakoff & Johnson (1980, p. 5), “the essence of metaphor is understanding and experiencing one kind of thing in terms of another.” An experiential metaphor has its basis in concrete, physical experience, such as grabbing and moving objects in the physical world. The temporal navigation in the interface may then become meaningful through its similarity to the physical experience. However, with time the metaphor may turn into a convention and the reference to the external world becomes less important.

The available collections are presented differently in Experiments 1.2 to 1.5. For example, in Experiment 1.2, a tab is provided in the bottom of the screen. When the image is placed on top of it, the tab expands. This motional form introduces a new element, a collection in which the item may be placed, through a motional form that alludes to how drawers open in our physical world. However, this ‘drawer’ appear from the bottom of the screen, as if controlled by a mechanical force. Such motion may be described as an instance of virtual kinetics (Eikenes 2009). When the item has been placed in the collection, the drawer closes by contracting, as to affirm that the action has been completed. A similar motional form appears in Experiment 1.4, only this time the drawer enters from the left. This indicates a different location of the collections. In Experiment 1.3, the whole website retracts by being scaled down. This motion reveals information – in this case a set of underlying collections, and thereby provides a sense of layering and depth in the interface. In Experiment 1.5, a panel ejects from the left, pushing the website correspondingly to the right. This movement presents a set of collections while partly hiding the website, through a motional form that suggests a shift of focus – prompting the users to attend to the panel and make a decision as to where to store the media item.

These five experiments have suggested different ways in which motional form may connect to the interface action of adding a media item to a media collection. For example, fade and twist are employed to indicate a connection between elements, while a motional transformation is used to affirm that an action has been realised. Further, agency is transferred from the interface to the user through scaling and movement. A new element is introduced through the motional form of expansion, and an action is affirmed through contraction. In comparing Experiment 1.3 and 1.4, it becomes apparent that motional form connected to an interface action may play a role in spatially arranging elements in the interface. Finally, information is revealed through scaling, and movement is used to suggest a shift of focus.

Interface action 2: browse between several media items

As a result of collecting large amounts of media items and webpages there is a need for users to browse between different media items within collections. Video 2 presents four experiments on different ways of browsing between media items. These were all sketched using video and stop motion.


Video 2: Interface action: browse between several media items. 4 experiments, named 2.1, 2.2, 2.3, and 2.4. (Click to play.)

In Experiment 2.1, the camera is placed above the centre of a circle onto which images are placed. For producing this sketch, the images were placed on a sheet of paper and rotated in front of the still-standing camera. However, in the final movie, the camera is perceived as rotating in a manner that in film language is described as a ‘pan’. The arrangement of the camera and the media items, combined with the motional form of the pan, places the user in a three-dimensional spatial environment. It could be argued that the meaning potential of the pan has a basis in our experience of turning our head from one side to another, inviting us to investigate and compare the images that are sequenced in the circle. Consequently, the pan becomes a metaphor for our experience of looking. The motional form is also similar to that of the carousel in an amusement park – suggesting connotations of play and amusement, or dizziness. Here, connotation refers to the abstract ideas and values that are expressed through what is represented, as opposed to denotation, which refers to the concrete things that are represented on the screen (van Leeuwen 2005).[25]

In Experiment 2.2, images are presented as hanging from the ceiling in a virtual room (the motion sketch was produced upside down, from paper images standing on a flat surface). The virtual camera moves horizontally, which in film language is described as ‘track’, and in and out, which is described as ‘dolly’. The track movement suggests that we are watching or inspecting similar items that are placed along a line – the ‘front row’. The dolly movement on the other hand, may indicate a shift from foreground to background, as if suggesting a hierarchical order of media items. It may also suggest ‘approximation’ or ‘immersion’ – that we now are about to get closer to the media content. The camera moves only in two directions. The orientation of the camera view does not change, and we always look up to the images, as if they are looking down on us. This is similar to how a low, vertical angle in images may be said to make a depicted item “look imposing and awesome” (Kress & van Leeuwen 2006/1996, p. 140). Thus, the movement of the camera constructs a relation of power between the user and the media items through the viewpoint constructed by the camera. The combination of the grid-like setup of images and the rigid camera movement further suggests values of ‘mechanical structure’ and ‘order’.

In Experiment 2.3, the images are laid out in a grid pattern on a two-dimensional surface. Browsing the images is achieved by moving a row (horizontally) or column (vertically). How the user moves the rows and columns is not specified in the sketch; it could be achieved by using fingers touching a screen, a pointing device or a keyboard. As a row or column is moved, the media items change their individual location and the composition is thereby re-arranged. Such movement mimics physical and computer-based games, and may therefore allude to meanings such as ‘playfulness’, inviting users to regroup and re-order the items. On the other hand, if used in an inappropriate context it might allude to ‘chaos’ or ‘disorder’, despite the rigid grid arrangement.

In Experiment 2.4, the context is more specific and defined than in the other experiments; the sketch includes a representation of how the user controls the interface action through gestures. The media item, which in this case is a webpage, moves in from the right and settles in the middle of the screen. The item is introduced through a simple motional form: horizontal movement from right to left. This indicates that there may be more items located outside the screenspace on the right side. To browse to the next item, the user touches the screen and pushes the webpage left. The webpage moves simultaneously out of the screen, indicating that the page is located at the left side of the screenspace, and may be retrieved from there later.

These five experiments suggest some ways in which motional form may connect to the interface action of browsing between several media items. The movement of the camera places the user in a three-dimensional environment, suggesting a metaphor for looking. The position and movement of the camera constructs a relation of power between the user and the media content, in a similar way to how images may create a relation of power between depicted persons and the viewer (Kress & van Leeuwen 2006/1996). Further, browsing is enabled by re-arranging images through the motional forms of horizontal and vertical movement. Finally, an item is introduced and later located through horizontal movement.

Interface action 3: open collection to access the contained media items

As a consequence of organizing media items in collections, there is a need to open these to see the media items they contain. Video 3 presents nine ways in which motion may be used for the interface action of opening a collection. The first 8 experiments are sketched using computer animation, while the last one is sketched using stop motion.


Video 3: Interface action: open collection to access the containing media items. 9 experiments, named 3.1-3.9. (Click to play.)

Experiments 3.1-3.7 explore different ways of opening a collection from a list of collections (Video 3). In Experiment 3.1, a collection is opened by resizing the visual frame representing the collection as if it is being expanded or stretched. At the same time, the title of the collection fades out, and the containing items fade in. As a result, the representation of the collection transforms and becomes the open collection. This instance of motional transformation may therefore signify something flexible, elastic or organic, despite the rather abrupt motional form.

Experiments 3.2-3.7 suggest different ways in which motional form may reveal a collection. In Experiment 3.2, the open collection expands from underneath the representation of the collection. The linear and downward direction of the movement indicates a sense of weight and gravity, i.e. an instance of virtual kinetics. This sense of weight might further allude to ‘heaviness’ or ‘release’. Similarly, when the collection closes, it does so through a contraction, as if being pulled up by kinetic forces, squeezed, collected and contained.

In Experiment 3.3, the content of the collection is revealedthrough a curl, combined with a camera zoom. The ‘collection sheet’ curls as if possessing materiality and elasticity. Further, the bending combined with the camera zoom may allude to uncovering something that was previously covert; something secret, exciting or even intimate. There seems to be a potential to develop this idea further by exploring different kinds of materialities combined with motion.

In Experiment 3.4, the representation of the collection flips around a horizontal centre axis, and at the same time transforms into the open collection. This motional transformation, which we might call a centre flip, indicates that the content of the collection is located at the backside of the collection item. At the same time, the shape changes during the flip, as if the interface performs magic or invites to a game of hide-and-seek. As the collection flips, the other collections move correspondingly, reacting to the kinetic force of the flipping collection. In terms of cohesion, there are several motional forms that appear simultaneously. The movement of the other collections are subordinate to the flip; they have lower status and contributes less to the action than the flip (Martinec 1998, p. 165).

Experiment 3.5 also presents a flip. Here, however, the collection springs forward along an invisible curved path, settling closer to the interface camera and the viewer. This motional form, which we might call a spring flip, may suggest a friendly gesture as if ‘reaching out’ in order to show or present something. On the other hand, the motion appears to be more offensive or aggressive than the flip in the previous experiment; it demands attention from the viewer. This may be understood in terms of salience, which in semiotics refers to the degree an element in a composition attracts the viewer’s attention (Kress & van Leeuwen 2006/1996). Here, a high degree of salience is achiever through motional form.

In Experiment 3.6, the list of collections seems to be presented on a slightly curved surface or large cylinder. When selecting a collection, the cylinder rotates, and the open collection expands from the selected collection, as if drawn there and then. Opening the collection through this motional transformation may indicate that the collection is generated or constructed as it is being opened, alluding to a sense of ‘updatedness’.
In Experiment 3.7, 3.8 and 3.9, the collections are not arranged as part of a vertical list. In Experiment 3.7, collections are represented by drawn squares laid out in a grid. When selecting one of the collections, the camera zooms in on the selected square. Simultaneously, the square flips from its left side, opening to reveal media items inside the frame. This motional form, which we might call a side flip, may allude to a window, book or door being opened, with associations of entering a new space or environment available for exploration.

In Experiment 3.8, spheres arranged in a three-dimensional space represent collections. When opening a collection, the sphere transforms into a two-dimensional grid by being flattened. This motional transformation suggests a sense of elasticity and flexibility, but also alludes to processes of simplifying, arranging and laying out information. Further, its meaning potential changes gradually as it goes from the rounded and organic encapsulation to the flat, hard and static presentation. This motional form also partly manipulates the sensation of space (Eikenes & Morrison 2010), as the collection changes from a three-dimensional representation to become two-dimensional.

In Experiment 3.9, a collection is represented by a piece of paper that unfolds as the collection is opened, revealing the containing media items. The unfolding happens through several steps, including segments of unfolding and rotating that take place synchronously as well as sequentially. As a consequence, rhythm becomes an important variable for design of motional form and its potential meaning. According to van Leeuwen (2005), rhythm organizes time into measures, phrases and moves; this unfolding may be understood as a phrase. The sketch is produced using paper, and thereby limited by its material properties. If to develop the idea further through computer animation, the ‘material realism’ of the idea could be manipulated in various ways, for example through motional transformations.

These nine experiments suggest some ways in which motional form may connect to the interface action of opening a media collection. An item on a list becomes a collection through a motional stretch, and virtual kinetics is achieved through the motional forms of expansion and contraction. A collection may be revealed through a range of motional forms resulting in rather different meaning potentials: a curl, a centre flip, a spring flip, an expansion, and a side flip. Further, opening is achieved through a motional transformation resulting in spatial manipulation. Finally, a collection is opened through unfolding, which in this case points to the need for considering rhythm in designing motional form.

Reflection across experiments

A range of motion sketches has been produced through design experimentation within a practice-based project. A total of 19 motion sketches have been presented and analysed. Several aspects have emerged in this process, of which I want to focus on two.

The first aspect concerns how motional form connected to interface actions are meaningful by the use of connotations and experiential metaphors. These concepts from social semiotics have been useful for understanding the meaning potential of motion in interfaces. According to van Leeuwen, they are also essential for achieving ‘semiotic innovation’ (van Leeuwen 2005, p. 276), and should therefore be important for design. The connotations of motional forms are culturally shared, while experiential metaphors draw more directly on our individual bodily experience of movement in the physical world. However, these are not always easy to separate. For example, the ‘pan’ motion may provide a way of looking based on our embodied experiences of looking by rotating our heads, but is also made meaningful through cultural conventions of spatial organisation and movement, e.g. as developed in film. Further, experiential metaphors are commonly shared within a culture (Lakoff & Johnson 1999). Importantly, these meanings are not fixed; they emerge and are transformed through use in context, and through interplay with other media. This points to the need for further investigations and discussion beyond what is possible here.

The second aspect concerns the direct ways in which motional forms can play a communicative role in web browsing. Based on motion sketching experimentation and analysis I suggest ten preliminary principles or ‘semiotic functions’ for motional form connected to interface actions. These are not definite or normative principles, but rather potential resources that may be used as a starting point for designing new motional forms:

These principles indicate that motional form may play multiple communicative ‘functions’ in the interface, in addition to being a means for artistic expression or style. This suggests that motional form is tightly connected to function and interface actions; they work together in complex mediation. However, decomposing dynamic screen interfaces in terms of interface actions and motional form seems useful for design experimentation and analysis.

Conclusions

Movement is essential to human activity and communication; we perform actions as well as communicate by moving our bodies through space. Increasingly, movement is employed on the screens of digital artifacts. These artifacts and their web browsers mediate a range of activities of work and pleasure, and there is a need to understand what role movement may play in interface design in general, and especially in activities related to web browsing.

Manovich (2001, p. 90) argues that there is a tension between representation and control in the interface. This tension is similar to the one between form and function (D. Wood 2007). The interface allows the user to do things and thereby mediates activities, but it also communicates in itself. There is a need to connect and integrate views on semiotic mediation and instrumental mediation in interface design, to account for what Bødker & Andersen (2005) call complex mediation. While Bødker and Andersen focus on semiotic mediation in terms of activities (such as verbal communication), I have focused on the semiotic properties of the interface itself. This article has suggested the notion of motional form to bring attention to the design and shaping of movement, while the concept of interface action has been introduced to denote specific actions that a user carries out through the interface. Connections between such forms and actions may be seen as instances of complex mediation. Meaning at the interface emerges through use and complex mediation over time.

This article has investigated how movement may be employed in web browsing through practical design experimentation. Practice-based research is needed to unfold the design potential of emerging digital media, and to build theory and concepts that are linked to design practice. This is a process that oscillates between the creative and practical on one hand, and the analytical and theoretical on the other. The article demonstrates the need for research to engage with design experimentation to find potentials that are difficult to discover by testing or analysing existing interfaces.

A range of design techniques have been explored for sketching movement; motion sketching may be achieved through video, stop motion, computer animation, or code. Motion sketching demonstrates a great potential for design practice, education and research, both in terms of the activity of sketching, and the resulting artefacts that embed the potential of motional form. The different techniques provide different possibilities; a certain technique enables certain motional forms to be sketched while discouraging others. Sketching through code has not been applied much here, and provides a potential for further investigation.

Based on motion sketching and analysis, I have proposed ten provisional principles or ‘functions’ for motional form in web browsing. In addition, it has been suggested that movement conveys a rich meaning potential by the use of connotations and experiential metaphors. This complicates the notion of form and function in interface design, suggesting that motional form may play an important ‘function’, and indicate that movement is underused in web browsing and interface design today. The concepts and principles developed here could for example be used as a starting point for designing a new dynamic web browser, or applied to other kinds of interfaces. The ten principles represent a novel contribution that may challenge existing concept in social semiotics as well as design research by joining instrumental mediation and semiotic mediation; social semiotics needs to take up the concept of action and engage in design experimentation to investigate emerging and dynamic forms of representation.

The complex mediation of interfaces already occurs through their use. Further investigations are needed into the interplay between motional form and other modes of communication and human actions (e.g. gestures). As the recent launch of the Apple iPad illustrates, new challenges and potentials of dynamic and complex mediation emerge as technologies develop and new devices are introduced.

Acknowledgements

Thanks to Jan Standal and Knut-Jørgen Rishaug at Opera, Andrew Morrison, Jørn Knutsen, Jonathan Romm, Lise Hansen, and other colleagues at the Institute of Design at AHO. The RECORD project is funded by the Research Council of Norway.

References

Arnheim, R. (1957). Film as art. Berkeley: University of California Press.
Baecker, R., & Small, I. (1990). Animation at the interface. In B. Laurel (Ed.), The art of human-computer interface design (pp. 251-267). Reading: Addison-Wesley.
Benyon, D. (2001). The new HCI? Navigation of information space. Knowledge-Based Systems, 14(8), 425-430.
Berners-Lee, T., & Fischetti, M. (1999). Weaving the Web: the past, present and future of the World Wide Web by its inventor. London: Orion Business.
Bødker, S., & Andersen, P. B. (2005). Complex mediation. Human-Computer Interaction, 20(4), 353-402.
Bolter, J. D., & Grusin, R. A. (1999). Remediation: understanding new media. Cambridge, MA: The MIT Press.
Buchan, S. (2006). Animated 'worlds'. Eastleigh: John Libbey.
Buchenau, M., & Suri, J. (2000, Aug. 17-19). Experience prototyping. the conference on Designing interactive systems: processes, practices, methods, and techniques, New York.
Buxton, W. (2007). Sketching user experience: getting the design right and the right design. San Francisco, CA: Elsevier.
Chang, B., & Ungar, D. (1993). Animation: from cartoons to the user interface. Proceedings of the 6th annual ACM symposium on user interface software and technology, Atlanta.
Drucker, J. (2009). SpecLab: digital aesthetics and projects in speculative computing. Chicago: University of Chicago Press.
Eikenes, J. O. (2009). Social navimation: engaging interfaces in social media. Paper presented at Engaging Artifacts, the 3rd Nordic design research conference (NORDES).
Eikenes, J. O., & Morrison, A. (2010). Navimation: exploring time, space & motion in the design of screen based interfaces. International Journal of Design, 4(1).
Eisenstein, S. (1988). Eisenstein on Disney. London: Methuen.
Engholm, I. (2007). Design history of the WWW.: website development from the perspective of genre and style theory. Artifact, 1(4), 217-231.
Feijs, L., Kyffin, S., & Young, B. (2005, Nov. 11). Design and semantics of form and movement, DeSForM 2005.
Gedenryd, H. (1998). How designers work: making sense of authentic cognitive activities. Unpublished doctoral thesis, Lund University.
Gloor, P. A. (1997). Elements of hypermedia design: techniques for navigation & visualization in cyberspace. Boston, MA: Birkhäuser.
Hummels, C., Overbeeke, K. C., & Klooster, S. (2007). Move to get moved: a search for methods, tools and knowledge to design for expressive and rich movement-based interaction. Personal Ubiquitous Computing, 11(8), 677-690.
Jeamsinkul, C., & Poggenpohl, S. (2002). Methodology for uncovering motion affordance in interactive media  Visible Language 36(3), 254-280.
Jensen, K. B. (2005). Interface://Culture: the World Wide Web as political resource and aesthetic form. Frederiksberg: Samfundslitteratur Press.
Johnson, S. (1997). Interface culture: how new technology transforms the way we create and communicate. San Francisco, CA: Harper Edge.
Jones, W., Bruce, H., & Dumais, S. (2001). Keeping found things found on the Web. Proceedings of the 10th international conference on Information and knowledge management, Atlanta, GA.
Kaptelinin, V., & Nardi, B. A. (2006). Acting with technology: activity theory and interaction design. Cambridge, MA: The MIT Press.
Keller, R. M., Wolfe, S. R., Chen, J. R., Rabinowitz, J. L., & Mathe, N. (1997). A bookmarking service for organizing and sharing URLs. Computer Networks and ISDN Systems, 29(8-13), 1103-1114.
Kress, G. R., & van Leeuwen, T. (2006/1996). Reading images: the grammar of visual design (2nd ed.). London: Routledge.
Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago Press.
Lakoff, G., & Johnson, M. (1999). Philosophy in the flesh: the embodied mind and its challenge to western thought. New York, NY: Basic Books.
Leont'ev, A. N. (1978). Activity, consciousness, and personality. Englewood Cliffs, N.J.: Prentice-Hall.
Liestøl, G. (2003). "Gameplay": from synthesis to analysis (and vice versa). In G. Liestøl, A. Morrison & T. Rasmussen (Eds.), Digital media revisited: Theoretical and conceptual innovation in digital domains (pp. 389-413). Cambridge, MA: The MIT Press.
Loke, L., Larssen, A., Robertson, T., & Edwards, J. (2007). Understanding movement for interaction design: frameworks and approaches. Personal and Ubiquitous Computing, 11(8), 691-701.
Löwgren, J. (2007). Pliability as an experiential quality: exploring the aesthetics of interaction design. Artifact, 1(2), 85-95.
Manovich, L. (2001). The language of new media. Cambridge, MA: The MIT Press.
Manovich, L. (2007). Deep remixability. Artifact, 1(2), 76-84.
Martinec, R. (1998). Cohesion in action. Semiotica, 120(1/2), 161-180.
McCloud, S. (1994). Understanding comics: the invisible art. New York: HarperPerennial.
Metz, C. (1974). Film language: a semiotics of the cinema. New York, NY: Oxford University Press.
Morrison, A. (forthcoming 2010). Inside multimodal composition. Cresskill, NJ: Hampton Press.
Müller-Prove, M. (2002). Vision and reality of hypertext and graphical user interfaces. Unpublished masters thesis, Universität Hamburg.
Nardi, B. A. (Ed.). (1996). Context and consciousness: activity theory and human-computer interaction. Cambridge, MA: The MIT Press.
Nielsen, J. (1995). Guidelines for multimedia on the Web.   Retrieved Feb. 21, 2010, from http://www.useit.com/alertbox/9512.html
Petersen, H., & Nielsen, J. (2002). The eye of the user: the influence of movement on users' visual attention. Digital Creativity, 13(2), 109-121.
Russett, R., & Starr, C. (1988). Experimental animation: origins of a new art (Revised edition ed.). New York: Da Capo Press.
Schön, D. (1983). The reflective practitioner: how professionals think in action. New York: Basic Books.
Schraefel, M. C., & Zhu, Y. (2001). Interaction design for web-based, within-page collection making and management. Proceedings of the 12th ACM conference on Hypertext and Hypermedia, Århus, Denmark.
Sevaldson, B. (2004). Designing time: a laboratory for time based design. Paper presented at Futureground 2004.
Skjulstad, S., & Morrison, A. (2005). Movement in the interface. Computers and composition, 22(4), 413-433.
Tatarkiewicz, W. (1980). A history of six ideas: an essay in aesthetics. Warzawa: Martinus Nijhoff.
Thomas, B. H., & Calder, P. (2001). Applying cartoon animation techniques to graphical user interfaces. ACM Transactions on Computer-Human Interaction, 8(3), 198-222.
van Leeuwen, T. (1999). Speech, music, sound. London: Macmillan Press.
van Leeuwen, T. (2005). Introducing social semiotics. London: Routledge.
van Leeuwen, T. (2006). Towards a semiotics of typography. Information Design Journal, 14(2), 139-155.
Vertelney, L. (1989). Using video to prototype user interfaces. SIGCHI Bulletin, 21(2), 57-61.
Vygotsky, L. (1978). Mind in society: the development of higher psychological processes. Cambridge: Harvard University Press.
Weinberger, D. (2007). Everything is miscellaneous: the power of the new digital disorder. New York: Times Books.
Wood, A. (2007). Digital encounters. London: Routledge.
Wood, D. (2007). Form and function in web design: a humanistic perspecive. Design Principles and Practices: An International Journal, 1(3), 71-78.
Wright, A. (2009). Ready for a Web OS? Communications of the ACM, 52(12).

Notes

1. The World Wide Web is commonly referred to as the web, and is a system of interlinked hypertext documents contained on the Internet. The web is mostly accessed through a web browser, which is a dedicated software application for accessing and exploring the web.

2. For example, Manovich argues that the ‘cultural interfaces’ of new media are shaped by the three cultural traditions of print, cinema, and human-computer interface.

3. Similar approaches has been taken to interface design through Activity Theory (see for example Kaptelinin & Nardi 2006; Nardi 1996). However, these approaches focus mostly on activities of work, and need to be integrated with a social semiotic perspective on the interface.

4. Semiotic resources may travel from one medium to another; in media studies, this process has been described as remediation (Bolter & Grusin 1999) or intertextuality. For example, features from film or animation may be taken up in interface design, and elements of interface design may be employed and adapted in television advertising.

5. Martinec (1998) argues that physical movement may be understood as a semiotic system, by analysing how stretches of physical body movement can form a whole through cohesive resources.

6. In RECORD (www.recordproject.org), several small experimental design projects have been carried out in co-operation between the Oslo School of Architecture and Design (AHO, www.aho.no) and several business partners.

7. Opera Software: www.opera.com  

8. www.opera.com/company/

9. Hyperlinks and the ‘page metaphor’ of websites are radically different from the WIMP (Windows, Icons, Menus, and Pointing device) conventions of the operating system, with its desktop metaphor and features of ‘drag and drop’ and ‘direct manipulation’ (Müller-Prove 2002). However, websites are increasingly incorporating conventions such as drag and drop, which used to be dedicated to desktop applications. Similarly, desktop applications increasingly use conventions we know from the web, such as hyperlinks.

10. Web browsers typically provide tools for saving web addresses for later retrieval through ‘bookmarks’ or ‘favourites’. However, Jones et al. (2001) found that these tools are underused in favour of other methods to manage web information, such as sending e-mails or copying web addresses into a document. Keller et al. (1997) argue that a web-based bookmarking service may be a better solution, by providing access from any device connected to the Internet, and the possibility to easily share information with others. Several ‘social bookmarking’ services exist today, but these are not provided by the web browser companies, and thereby poorly integrated in the web browser interface.

11. Screen-based movement has been approached from a range of fields and disciplines that can only be briefly pointed to here. Approaches range from cognitive ones to semiotic ones, including visual perception (Arnheim 1957) and media studies (Skjulstad & Morrison 2005), to more experimental ones based on artistic practice (Russett & Starr 1988). Motional form has also been addressed from design perspectives, such as in interaction design (Jeamsinkul & Poggenpohl 2002; Löwgren 2007), industrial design (Feijs, et al. 2005) and systems-oriented design (Sevaldson 2004). For example, Jeamsinkul & Poggenpohl (2002) showed through a quantitative study that interpretation of motion is affected by culture and professional training. Motional form in the interface may partly be understood by drawing on studies of film and animation (e.g. Eisenstein 1988; Metz 1974). HCI has previously looked to animation, mostly with a focus on what cognitive benefits animation may provide for the user (Baecker & Small 1990; Chang & Ungar 1993; Nielsen 1995; Petersen & Nielsen 2002; Thomas & Calder 2001). Few have addressed motional form in the interface from a social semiotics perspective on design, the closest being Skjulstad & Morrison (2005).

12. Adobe Flash: http://www.adobe.com/products/flash. Java: http://www.java.com 

13. Cascade Style Sheeting (CSS) is a widespread scripting language that describes the presentation and appearance of electronic documents online. Consequently, CSS technology makes a clear distinction between content and visual form of a website. Drucker (2009) argues that such a separation of content and form may be problematic if graphic design and interface design is “regarded as window dressing, a skin to be grafted, at the last minute, onto an already formed information structure” (p. xv). As she argues, such an understanding misses the point that ‘design is information’; visual forms may communicate as much as the ‘content’ itself. Therefore, they should be considered as a unity, not separate from each other. However, there is no question that CSS has played a key role in developing a web of visually complex and coherent websites. Opera has been an important agent in the development of CSS, by being one of the first browsers to implement CSS standards.

14. Some applications for web browsing employ movement, such as SpaceTime (http://www.spacetime.com/home.php), which presents websites as floating in a three-dimensional space. Cooliris (http://www.cooliris.com/) is a web browser plugin that allows users to browse images and videos in full screen, aligned along a flat wall in a simulated three-dimensional space (see analysis in Eikenes & Morrison 2010).

15. In a social semiotic frame, representation and classification of information is always ‘motivated’ and ideological (Kress & van Leeuwen 2006/1996). Providing users with the means to easily make their own digital collections allows them to be in charge of the organization of the information, as advocated by Weinberger (2007).

16. The media files themselves should by default not be saved locally on users’ computers. This is similar to traditional browser bookmarks, and services such as Spotify (http://www.spotify.com), where content is streamed over the Internet. This allows the collections and the referred content to be available on different devices through the Internet. Recently, Spotify has allowed people to share and subscribe to their friends’ playlists by connecting through Facebook (www.facebook.com). I take this as a positive indication of the relevance of the proposed service, as it suggests a similar infrastructure.

17. In these professions, sketching is generally understood as a specific activity, but also closely connected to the resulting physical artefact - the sketch itself. The sketch facilitates a dialogue between the designer and the material (Schön 1983). This is a conversation between the mind of the designer and the sketch, in which reading and interpreting creates new knowledge. Importantly, the sketch also becomes a mediating artefact(Vygotsky 1978) that may be shared between various actors in activities of design and development, and thereby facilitate and provoke discourse.

18. For example, Buchenau & Suri (2000) define prototypes as “representations of a design made before final artifacts exist," and propose the term 'Experience Prototype' to describe "…any kind of representation, in any medium, that is designed to understand, explore or communicate what it might be like to engage with the product, space or system we are designing" (p. 2).

19. For example, learning from comics drawing, motion may be depicted within a single image by using techniques such as ‘motion lines’ (McCloud 1994, p. 110) and arrows indicating the speed and direction of movement. Motion may also be indicated by juxtaposing several images, such as commonly done in cartoons and storyboards.

20. Similar techniques are used in puppet film production, and have also been used in interaction design, such as in the famous ‘sketch-a-move’ (Buxton 2007).

21. Animation is commonly understood as a range of techniques that make inanimate objects or drawings appear to move in motion pictures or computer graphics.

22. Processing: http://processing.org/

23. Prezi: http://prezi.com

24. Some of the sketches analysed here have been presented and discussed in an online 'Living Lab' environment, with designers and usability experts. This has informed my understanding of motional form in this article.

25. This understanding of connotation and denotation is based on the work of Roland Barthes.