Contextual User Interfaces and beyond

The new interfaces are winning people over because they are based on usage patterns instead of choices. The key thing about new UIs is that they are contextual – presenting the user with minimal components and then changing in reaction to user gestures. Thanks to Apple, we have seen a liberating movement towards simplistic, contextual interfaces. But can these UIs become the norm?

Over on his blog, Alex Iskold has written a wonderful piece on The Rise of the Contextual User Interfaces. In it he contrasts old school traditional user interfaces from the days when Microsoft Windows dominated everyones interaction with a computer, to the new generation of contextual user interfaces that the likes of Flickr, 37 Signals, Shelfari etc. all seem to have embraced.

I think contextual UI’s are something that we all subconsciously appreciate but don’t really think about , they just seem to work, they just seem to let us do what want to – and therein lies their beauty. It’s their elegance and their simplicity that makes them a pleasure to use. They only tell us what we need to know, if we need to know it, they don’t confuse us by presenting a plethora of options that we then have to decipher before we can continue, or indeed hide important functionality behind an “advanced” setting somewhere. It’s for this reason I completely agree with Alex when he describes how one of philosophies of the old UI approach was entrenched in the idea of presenting the user with all the information all of the time, which was overwhelming. The move towards Contextual User Interfaces is really about building user interfaces that respond to the way that users interact with them – and ideally to the individual user him/herself.

As hardware and processors have become more powerful we have become better able to pre-process and analyse information for users in order to give them exactly what they need when they need it.
This transition towards being context aware isn’t something that happened overnight, it’s happened gradually over time as technologies have matured and improved to allow us to do things that weren’t necessarily possible before. One often touted example of this is the Spell Checker, I remember when you had to explicitly invoke the spell checker in Microsoft Word, whereas now its done automatically in the background as you type. So I do wonder how long it will be before processing becomes cheap enough for us to process entire databases for users in order to derive better context.

But one of the stumbling blocks is that whilst we can derive or assume some context within an individual application we still don’t have the tool’s to computationally describe and communicate context where reasoning and inference is distributed. Why is that important? well to me my context, as an individual, is in some ways predictable and in others it’s highly temporal. In an ideal world there would be a way to describe who I am, what my interests are in general, but also what also what my interests are at a given point in time. If we could formalise that description, using an standardised ontology, then we could provide that ontology as an input into any application we used. That’s where a lot of the work that my friend Alan has been doing has been focussed, and it’s also one of my areas of interest.

That’s why Alex’ post was so wonderful, it resonates with an articulates many of the things that I’ve been thinking about for a while.

PhotoSynth at TED

PhotoSynth is a Microsoft technology that I have talked about before, it’s certainly one of the most impressive visual technologies I have seen.  Here’s a video of Blaise Aguera y Arcas at TED this March presenting PhotoShop to the conference attendees and getting a standing ovation … quite deservedly in my opinion. The video is also a great introduction to the technology and what its capable of doing.

I agree with Blaise when he says Simply put, it could utterly transform the way we experience digital images.

Microsoft adds copyrighted books to its Live Search

Microsoft has added copyrighted books to its online library, stating it has permission to offer the works t searchers on the internet .  They have made deals with with authors and publishers to include their works in the Live Search, and in doing so Microsoft have managed to sidestep the controversy that Google initially triggered when they began their book digitising project to offer the worlds written works online.

User’s have to log into Live Search in order to read the content online, it appears that user’s can only read a certain number of pages and the software keeps track of how many pages visitors have read online for free. Most of the searches I have performed allow me to read 10% of the pages in any Copyrighted book that I’m browsing. I actually find this far more useful than simply being able to browse the table of contents or view selected extracts as one can in Google’s Book Search. This enables me to make a more informed decision as to whether I want to go and buy the book, and also, if im just searching for the answer to a question or need to read up on a chapter on some specific subject, I can read those bits for free online which probably will mean I wont need to go out and buy the book.

Amongst the publishers who have agreed to allow this type of access to their materials are Simon and Schuster, Mcgraw-Hill, Rodale and Cambridge University Press. Microsoft obviously provide direct links to where the books you are reading online can be purchased from. It’s an interesting move by Microsoft when you consider that Google’s wholesale scanning of copyrighted works in library collections will give it a larger database of books, but by working with publishers and obtaining their permission, Microsoft is seemingly able to offer a better experience to users.

JavaScript can provide a richer user experience without compromising Accessibility.

When Rob and I originally put Cenote together one of the constraints we had to work within was that the application must work without relying on JavaScript. This is primarily because relying heavily on JavaScript can introduce some accessibility issues, these issues include:

  • Making Navigation difficult: Site’s that rely on JavaScript can make it very difficult to navigate around a site using a keyboard or other assistive technologies, and impossible if JavaScript is disabled in the browser.
  • Hides Content: Sites that rely heavily on JavaScript can present content and functionality to the user that is not accessible to assistive technologies, such as Text Readers.
  • Removes User Control: User’s have no control over automated content changes. For user’s have no accessibility needs this isn’t a problem, but for users who rely on assisstive technology there is no way of knowing that content has changed. This can prove to be very confusing/disorienting since this can also involve altering normal browser functionality and triggering events that the user is not aware of.

This doesn’t necessarily mean that pages that contain JavaScript cannot be fully accessible, such pages can be accessible if the functionality is device independent ( in other words it does not require only a mouse or only a keyboard to work ) and the content is always available to assistive technologies.

So based on this knowledge Rob and I made the conscious decision that what we would do was develop Cenote first without using any JavaScript at all. This would ensure that the content rendered on the pages was always available to assistive technologies. We agreed that we could then use JavaScript to enhance the user experience by using JavaScript to manipulate the DOM of the page in the browser.

To see this in action visit this page in Cenote, if you scroll to the bottom of the page you’ll notice a list of libraries that you navigate through using a paging control (pictured). You’ll also notice that the book jacket in the top left hand corner of the page that has a nice reflection effect. Both these effects are made possible through JavaScript, and provide a slightly richer user experience. To see what I mean try turning off JavaScript in your browser1, and hit refresh. The page now displays the book jacket without the reflection and the complete list of libraries without the paging control.

So how is all this achieved? Well lets take a closer look at how the paging control. The snippet of html below shows a simplified2 version of markup that actually represents the items in the list3, take particulare note of the empty div after the unordered list this is the placeholder we inject the paging control into:

  1.  
  2. <ul id="Libraries">
  3.   <li><a class="item-library" href="#">Aberdeenshire Libraries</a></li>
  4.   <li><a class="item-library" href="#">Barnsley MD Library</a></li>
  5.   <li><a class="item-library" href="#">Blackburn</a></li>
  6. </ul>
  7. <div id="pagingcontrol"></div>
  8.  

Now when you view the page without JavaScript enabled the browser renders this markup exactly as it is. Accessibility or Assitive technology only use the HTML markup on the page. In the case of Text Readers this HTML is parsed and the software reads the content on the page back to the user. Now when JavaScript is enabled the markup on the page does not change, it stays exactly the same. We use JavaScript to manipulate the browsers DOM which is the model the browser generates internally to represent the page. Now in the example above the Unordered List is identified using the id “Libraries” which makes it easy to find in the DOM as this snippet of code illustrates:

  1.  
  2. var libraries = new Array();
  3. var ul = document.getElementById(‘Libraries’);
  4. var holdings = ul.getElementsByTagName(‘LI’);
  5. for( var x=0; x<holdings.length; x++ )
  6. {
  7.     libraries[x] = holdings[x].innerHTML;
  8. }
  9.  

Here we use its identifier to find the unordered list we want to augment with our paging control. Each list element in the unordered list is what we refer to as a holding, we simply extract the contents of each of those nodes and temporarily store them in an array called libraries. In the case of the page I pointed you to earlier this array would contain 50 items.

Once we have found and copied the original complete list of libraries we can now proceed with updating the DOM to display to the user only the first 10 libraries, and render a paging navigation control. The function below updates the DOM by removing the original contents of the Libraries node, and replaces it with 10 items indexed from the specified startPosition.

  1.  
  2. function renderLibraryLinks( startPosition )
  3. {
  4.     if (libraries.length > 0)
  5.     {
  6.         currentStartPosition = startPosition;
  7.     d = document.getElementById("Libraries");
  8.         d.innerHTML = ;
  9.         d.innerHTML += ‘<UL>’;
  10.         for ( var x = startPosition; x < (startPosition + 10); x ++ )
  11.         {                    
  12.             if( x < libraries.length)
  13.             {
  14.                 d.innerHTML += ‘<LI>’ + libraries[x] + ‘</LI>’;
  15.             }
  16.             else
  17.             {
  18.                 d.innerHTML +=‘<LI> </LI>’;
  19.             }
  20.         }
  21.         d.innerHTML += ‘</UL>’;
  22.         renderNavigationControls();
  23.     }
  24. }
  25.  

The final line in the function above calls a method to render the navigation controls which I’ve copied below. You’ll notice that we find the placeholder identified by “pagingcontrol”, and render the preview, next and individual page links ( renderPageNumbers() ) into it.

  1.  
  2. function renderNavigationControls()
  3. {
  4.     if( libraries.length > pageSize )
  5.     {
  6.         lpc = document.getElementById(‘pagingcontrol’);
  7.         lpc.innerHTML = ;
  8.    
  9.         if ( currentStartPosition > pageSize-1 )
  10.         {                                              
  11.             lpc.innerHTML += ‘<a href=\”javascript:printPreviousPage();\” alt=\”Previous Page\” title=\”Previous Page\”>< previous</a>’;
  12.         }
  13.         else
  14.         {
  15.             lpc.innerHTML += ‘< previous’;
  16.         }                  
  17.        
  18.         lpc.innerHTML += ‘  ‘ + renderPageNumbers() + ‘  ‘;
  19.        
  20.         if( currentStartPosition < libraries.length – pageSize )
  21.         {
  22.             lpc.innerHTML += ‘<a href=\”javascript:printNextPage();\” alt=\”Next Page\” title=\”Next Page\”>next ></a>’;
  23.         }
  24.         else
  25.         {
  26.             lpc.innerHTML += ‘next >’;
  27.         }
  28.     }
  29. }
  30.  

To reiterate, because this technique manipulates the DOM, the original markup isnt changed at all. Project Cenote was recently open sourced you can download the entire source base, which includes the javascript file, pagingcontrol.js, from which I took the extracts above, and get a feeling for how it works and experiment with it.

Now the above example might not seem all that impressive, its just a little paging control, right? Well, its a simple example that illustrates the technique which can be very powerful. Consider the following images which show a Calendar that view that was created by one of my colleagues, Andrew, which uses an unordered list to represent a series of events, and that’s what you see when JavaScript is disabled:



However with JavaScript enabled the DOM of the page is rewritten and combined with some CSS it looks like this:

Now thats more impressive, right? I hope this serves to illustrate how you can use JavaScript to provide richer user experiences without compromising accessibility. It’s an area I’ve been thinking about a lot recently and there are wider issues around how to develop better JavaScript libraries that understand how to expose dynamic content that needs to be accessible and whether this needs to be considered a best practise that more developers need to adopt. These issues beginning to have wider implications as my friend Alan put it earlier this week:

it will address issues of how traditianal HCi changes in the face of web2.0: when people are doing mash-ups (so there are more ‘artisan’ interface constructors), or end users configuring their own spaces (as in MySpace), or how you build ‘good practice’ into javascript libraries like prototoype or yahoo! YUI

.. when he contacted Justin and me earlier this week and invited us along with others from Talis to attend this years 2007 – British HCi Conference taking place at Lancaster University. Recognising how we to develop best practises for the development of truly accessible Javascript libraries is one of the issues I’ll be raising to the panel when I attend the conference.

  1. In Firefox you can achieve this by going to Tools -> Options -> Content and uncheck the box labelled JavaScript. However if you don’t already use it I recommend installing the Web Developer extensions for Firefox [back]
  2. I’ve removed the url and the title attribute and the icon image from each link to simplify the example [back]
  3. I’m using Dean’s Syntax Highlighter but specifying html means that the Snap Shots tries to execute against it, so I’ve specified XML instead [back]