Search is an evolutionary, iterative process. Like a Grizzly Bear foraging for food in the forest, people jump from one information source to the next as we seek to satisfy our curiosity.

Presented at Apache Lucene EuroCon 2010 in Prague, “The Scent of Search” seeks to apply the principles of Information Foraging Theory, and in particular, Information Scent, to the usability of search interfaces. Below are the main recommendations we set forth in the talk.

The Search Box

1. The searchbox should look like a searchbox. Drastically changing its appearance will result in fewer users discovering it.
2. Place the searhbox in the top right corner of the page. Users have come to expect it in this location, so moving it anywhere else will reduce the number of users who actually find it.
3. Provide as-you-type query suggestions. Search suggestions reduce spelling errors, save time, and make users more confident of their query.

Search Results

1. Indicate the number of results. This helps users gage the validity of their search. Numerous results can act as a vote of confidence, while few results may raise a red flag.
2. Titles should be easily comprehendible. Filenames make for lousy titles. Use natural language titles that accurately describe the contents of the result.
3. Matching words should be highlighted. Emphasising queried words when they appear in the search results makes the result list easier to scan.
4. Make sure visited links are easily discernible from unvisited links. Visited link colours prevent users from accidentally reviewing the same result twice.
5. Differentiate results when real differences exist. Clearly identifying which category a result belongs to can make results easier to scan.
6. Avoid having zero results. A search results page with no results is a roadblock to users that could make them give up searching altogether. Consider using automatic spelling suggestions.

Faceted Navigation

1. Indicate the number of results that match each filter. Filter result counts give insight into the shape of the data.
2. Use breadcrumbs to reflect the user’s query and applied filters. This helps users know where they are, how they got there, and get back if necessary (breadcrumbs should be removable).
3. Make field values clickable. Important fields in the search results should be applied as filters when clicked, making it easy for the user to filter the results.
4. Find ways to meaningfully visualise facets. Some facets may work best as a list, others as a chart, still others plotted on a map.

Word on the street is that Google is rolling this new design out to everyone over the next 48 hours. As with any change, some people are bound to complain, but I think the redesign introduces many significant improvements.

What’s Changed

• The Google logo is about 30% larger.
• There is now a permanent sidebar left of the results that allows filtering by news, blogs, images, etc., as well as time range filters and options for changing how search results are displayed.
• The searchbox is now the full width of the results column, slightly taller, and has a slight drop shadow rather than the previous inner shadow.
• No more top vertical bar. The result count now sits just below the searchbox and is much smaller than before. The filter for searching locally moved from under the searchbox to the sidebar.
• Search results are now 55 pixels higher on the page and have have a higher density overall (there’s slightly less vertical space between results, and indented results have only one third of their previous margin).
• Cached, Similar, Show more, and Related links all changed from a muted purple to a brighter light blue and now only have an underline on hover.
• Pagination is about 30% larger, and there is still a searchbox below the pagination, though the blue background has been removed.
• Related searches are now displayed much more compactly.

Before

After

The Sidebar

The new sidebar provides an inviting mechanism for narrowing your search to a specific channel. In particular, blogs and books now have much greater emphasis than ever before. Hidden under the “more” button are updates and discussions, and lower down in the sidebar are “latest” and “past two days” time filters, so there is definitely a push towards recent and real-time search. What isn’t clear to me, however, is why the sidebar filters are in a different order than the top bar options, and why some items from the top bar (shopping, for instance), didn’t make it into the sidebar at all. It definitely seems like Google should ditch the topbar options completely and place all these channel filters in one consistent location.

Related Searches

The sidebar also provides an option for showing related searches. In the past, Google has placed a handful of these search suggestions at the bottom of the page. But in the redesign, when related searches is selected, Google now presents up to twenty different related searches.

Wonder Wheel

There is also an option to show the wonder wheel, which is another mechanism for presenting similar searches. Instead of showing a simple list, however, the wonder wheel groups related searches into clusters and allows the user to navigate from one node to another. While this visualisation seems useless for simple lookup — finding the population of Spain, for example — it could be useful for exploring broad topics (such as “UK Politics”). My biggest complaint is with its name. “Wonder wheel” is completely undiscoverable and has absolutely no information scent. It’s impossible to guess what it does without actually interacting with it.

In Conclusion

The Google redesign offers many improvements — the sidebar is a step in the right direction, the searchbar feels much more intentional. Not all of additional views, from the wonder wheel to the timeline, feel all that useful, and the top bar now seems redundant. But overall the design itself feels crisper and more concise. I like it.

Further Reading

• The Google Blog: “A spring metamorphosis”
• Google’s New Interface: Colorful and More Powerful
• Search Engine Land has just about every possible screenshot of the new interface
• Video interview with a Google VP, product manager, and, web designer talking about today’s redesign
• Google Redesign Actually Mimicks Bing (or does it?)
• The Google redesign is much more than just a Bing copy

Search Patterns by Peter Morville

“There’s no shortage of problems with search today,” says Peter Morville at the end of Search Patterns, his most recent book. Throughout the book, Morville chronicles the challenges of search and effectively communicates the best practices of building usable search experiences. While the book more than adequately accomplishes it’s stated goal — to foster greater cross-disciplinary collaboration by increasing search literacy — it would have been an even better book if the author had proposed more of his own innovative solutions to search’s problems.

In the preface Morville sets out to tear down the walls between disciplines, and at this he succeeds. From user psychology to technical considerations to the specific components of the user interface, Search Patterns has something for everyone involved in implementing search.

The first two chapters lay the groundwork for the rest of the book, discussing both why people search and the individual components that make up search. The book is in full swing by chapter three where Morville discusses user behavior, elements of interaction, and — my personal favourite — the principles of design.

In talking about design principles, Morville describes search both as a conversation and a jazz-like improvisation. He urges the architect to make search an easy, simple process to initiate, followed by a progressively more sophisticated toolkit that enables users to iteratively refine their query. He argues for a no-suprises approach to the user interface in which valuable options are highly visible and elements of interaction are easily predictable. Many of the principles are applicable to a much broader context than search alone, but that only stregthens their merit.

The real heart of the book is chapter four, which looks at 10 design patterns over 50 pages. It considers the obvious patterns — autocomplete, faceted navigation, advanced search — as well as more disparate groupings like federated search and personalization. I was disappointed that the patterns only summarised existing conventions rather than challenging any bad habits. And while this chapter does consume 30% of the book by page count, it manages to stay at a very high level, even to a fault. I would have preferred more permutations and analysis of the cornerstone patterns, such as faceted navigation, while spinning some of the less well-defined patterns into chapters of their own for more consideration.

However, the lack of innovative solutions or granular analysis is quelled by the wealth of thought-provoking examples throughout the book, with an especially high concentration of desktop, mobile, and even kiosk visuals in chapter five.

The final chapter of the book is a forward-looking vision of what search could look like in the distant future, as well as what it probably won’t look like (sorry, no knowledge navigator robot assistants). While an enjoyable thought exercise, I would have preferred more concrete insights into the near-future possibilities for search (collaborative, task-oriented searching, for example).

Despite a lack of new ideas, Search Patterns is a must-read for everyone collaborating on a search application. It will give your team of designers, engineers, and business stakeholders a common vocabularly and greater awareness of the many sides of search.

Morville ends the book — as I will end this review — by urging the reader to get to work on making search better: “For every unsolved problem,” he says, “there are countless instances in which we know the solution, but nobody has bothered to implement it. Discipline and attention to detail would go a long way toward improving the world of search.”

You should get started by reading this book.

Mining the Web 2.0 to Improve Search

Ricardo Baeza-Yates, Yahoo Research

Ricardo Baeza-Yates talked about how web usage data can to be used improve relevance and accurately provide related queries.

He first talked about how user tagging can be combined with user-generated image annotations to increase the relevance of image search. What I found most interesting, however, was how Yahoo keep track of all the iterative queries people enter in order to get to a specific result (for example, if the user typed ‘furry animal,’ reformulated the query to ‘white black furry animal,’ and then chose an article on Pandas, Yahoo would associate both queries with ‘Panda’). Yahoo would then use this data to suggest related queries (i.e. if the user searched ‘furry animal’, Yahoo would suggest ‘white black furry animal’ as a related concept).

Google Squared: Web Scale, Open Domain Information Extraction and Presentation

Dan Crow, Google

Dan Crow opened by asserting that complex tasks, such as planning a trip or writing a research paper, are still very difficult on the web. Google’s user studies revealed that people make spreadsheets, email themselves things to remember, and add post-it notes to their computer monitors when undertaking these complex search activities. They also found that people seemed to love tables (and chose to adopt this table style for data presentation themselves).

Google Squared is an attempt to help people cope with comparison-heavy, list-driven search activities and it operates at three different levels. First, it seeks to discover the topic behind the user’s query (does “Ford” refer to U.S. Presidents or the car manufacturer, for example?). Once the topic has been isolated, Google Squared then tries to find attributes of that topic (price, horsepower, colour). And finally, it tries to fill in the values for each of the attributes.

Google Squared combines offline analysis (such as mining wikipedia categories and combing the web for attribute / value pairs) with run-time queries for finding specific values.

Relevance Challenges at Bing

Milad Shokouhi, Microsoft Research

I really enjoyed Milad Shokouhi’s talk about clever ways of boosting relevance in search, though I must say it stretched my vocabulary a bit. His initial slide outlining the challenges to relevancy included temporal queries, heterogenous verticals, and pre-retrieval query alteration. He did go on, however to clearly articulate each and offer concrete examples from Bing.

Like Ricardo Baeza-Yates, Milad Shokouhi touched on how query reformulation data can be used to enhance relevance of ambiguous terms. An example from Bing was the ranking of results for the query “wow.” The unadjusted top hit for this query was a cable company. But a look at the query reformulation logs indicated that a majority of people who queried for “wow” went on to search for “world of warcraft.” Bing then adjusted the ranking of the results so that world of warcraft appeared first.

He also talked about query trends, specifically the “spiking” of certain queries. Some queries quickly spike, but then quickly disappear. Other queries (like “iPad”), spike and remain high. Still others spike seasonally (such as “Halloween costume”).

These spikes of queries could be used to trigger the appearance of a news story, seasonally suggest related queries, or even forecast future events (elections, for instance).

Search User Experience, the Essentials of Great Search Design

Vegard Sandvold, Comperio

Vegard Sandvold argued for cross-disciplinary collaboration. Everyone should be involved in the design process, from stakeholders, to techies, to users. “Innovation happens happens where disciplines intersect.”

Vegard advocated a “Sprint Zero” phase of one to four weeks in which to set forth a plan of action for the project. During this time, he typically talks to stakeholders about their goals, interviews users, creates personas, and attempts to identify all of the problems that must be solved in the project. He also strives to prototype and test both the basics of the interaction design and the capacity of the underlying technology.

His talk can be summarised by his final slide: “We discover the best solutions together.”

Getting Value from the Search Master’s Toolbox

David Hawking, Funnelback

David’s talk was primarily about fine-tuning relevance in enterprise search deployments within organisations. He advocated picking a selection of random queries (that realistically reflect the distibution of queries within an organisation), trying those queries yourself, and tuning the engine until those queries produce the most relevant results.

Enterprise Search: State of the Market 2010 & Beyond

Nick Patience, The 451 Group

Nick Patience is a market analyst and provided some revealing insights into the industry. It’s a bit smaller than I would have guessed: in 2009 it totalled $1.3bn and is estimated to reach $2.8bn by 2013 (at a 22% annual growth rate).

Project Plaza – A New Approach to Information Management in the Construction Sector

Rob Blackwell, Active Web Solutions

Rob and his colleague demonstrated a search-driven web application for managing construction projects.

Rethinking the Library Catalogue

Sally Chambers, The European Library

Sally Chambers from the European Library talked about the challenges they faced in integrating a vast number of library corpora using a federated search approach. She described the issues they faced in providing relevant results due to the asynchronous nature of the system; where certain searches might time out and providing a blended set of relevant results was not feasible.

With users expecting the same user experience they get on the web, they initiated a move towards a single index of bibliographic and full text (including optically recognised) content. Some of the more persistent challenges that still remained had to do with the multi-lingual nature of the content, and the disparity in the metadata formats used by the individual institutions (such as MARC vs. Dublin Core).

Collaborative Research, Technology Transfer and Networking

John Tait and Francisco Webber

John Tait presented Francisco Webber’s ideas on the information retrieval innovation cycle. He advocated enhancing a mutually beneficial ecosystem between academia, government, and industry by aligning the incentives: academics need to publish papers, politicians need to get re-elected, and businesses need to generate revenue.

He insisted that innovation in technology hinges on open data. There was a rigorous discussion in the Q&A over the merit and drawbacks of open business models. A member of the audience asserted that companies aren’t realistically going to give information away for fear of competition exploiting it. John defended this criticism well by arguing that businesses who have embraced open business models have benefited from higher volume and the formation of communities around their products (his example was GE).

Wolfram Alpha – the New Computational Knowledge Engine

Jon McLoone, Wolfram Alpha

Jon McLoone began by saying that knowledge consists of four components: opinions, facts, methods, and understanding. While traditional search is geared towards retrieving opinions and facts, Wolfram Alpha focuses on revealing method and understanding, while leaving out opinion altogether.

The Wolfram Alpha approach asserts both that the desire for information does not indicate the ability to use it, and that computed data is more valuable than data alone (for example, turn-by-turn directions are more valuable than just a longitude and latitude).

This computational approach is evident in the examples that Jon demonstrated. Every type of data has a corresponding visualisation. Planets get plotted on a sky chart, financial markets are graphed, chemical elements are shown on the periodic table of elements, and recipe ingredients are even turned into a nutritional value chart.

One comment brought up in the Q&A session was the discoverability of all these visualisations, which is a concern that I share.

Using AI to get Answers from the Internet

Simon Overell, True Knowledge

True Knowledge is a platform for answering questions. It takes natural language questions and returns numerous well-cited answers by gathering information from both structured and unstructured sources.

Panel Discussion: Leveraging Semantics to Enable Better Search Experiences

Dan Crow (Google)
Gjergji Kasneci (Microsoft Research)
Jon McLoone (Wolfram Alpha)
Simon Overall (True Knowledge)

The primary thrust of the panel discussion was a debate over curated knowledge verses the wisdom of crowds.

Google Squared doesn’t bake in any knowledge or curate any information, it relies completely on scraping the open web in search for the answers. Wolfram Alpha, on the other hand, relies completely on authoritarian facts. True Knowledge takes a middle ground of presenting differing facts and opinions from around the web, and clearly citing the sources.

But not all as-you-type search suggests are created equal. Careful observation seems to reveal three different approaches: completion, suggestion, and instant results. These approaches range in cognitive burden on the one hand, and utility on the other. We’ll look at several examples of each and consider when they should be used.

Completion

The simplest of these three approaches is query completion. The sole role of completion is to reduce the friction involved in getting an idea out of your head and onto the computer screen. Just like a fluent translator improves the conversation between two people who speak different languages, so query completion helps a person and a computer communicate a bit more coherently.

Completion makes the most sense when dealing with a narrow strand of finite data. It’s commonly employed for concepts like geography (airports, cities, countries), finance (ticker symbols, company names), and people.

National Rail

Google Finance

Facebook

Query completion has many advantages over a plain-text search. It saves the user time by cutting out unnecessary keystrokes (i.e. I could type “Birm” and accept the suggestion of “Birmingham”). It also helps prevent spelling mistakes (I can just type “Mitsub” instead of remembering how to spell “Mitsubishi”). And, unlike some of the other forms of search suggestion, query completion does not suffer from being complicated or mentally tiresome. On the contrary, it can be such a natural behaviour that we almost don’t even realise it’s occurring.

My consensus, then, is to use completion whenever and wherever possible. It will make your users more productive without the risk of confusing them.

eBay

Suggestion

Suggestion is by far the most ubiquitous auto-complete approach. The obvious examples are search engines such as Google and Yahoo, though it’s used on virtually every category of website.

While completion helps get an idea from your head onto the screen, suggestion actually throws new ideas into the mix. It’s not too dissimilar to what happens when you tell your travel agent that you’d like to go on holiday. The agent comes back with a handful of suggestions like the Swiss Alps, Jamaican beach, or Vietnamese jungle. Similarly, if I type “guitar” into eBay’s search field, I get suggestions for “electric guitar,” “acoustic guitar,” and “guitar hero.”

Not only do suggestions save the user keystrokes, but they can actually help construct a more useful query than the user would have thought of on their own.

Amazon

But there are a number of opportunities for maximising the power of query suggestions that are often overlooked, but which Google seem to get right. The lowest hanging fruit of these are spelling suggestions. If I type “guitaf” into Google, the first suggestion that appears in the list is the correctly-spelled word, “guitar.” Surprisingly, this elementary feature is missing from giants like eBay, Amazon.com, and Yahoo. A second technique is to display the number of results for each selected term, as Google do with a right alignment. This has a risk (which we’ll get to in a bit), but it can help the user gauge the popularity of a given suggestion. And lastly, it’s often helpful to infuse suggestions with recent searches. One approach, which Safari utilises, is to simply have two groups, one for query suggestions and another that lists the user’s recent searches. A more sophisticated approach could be to use search log data (either for one user, a group of users, or all users) to augment the scoring of the query suggestions themselves.

Google

When deciding what features to include in search suggestions, it is important to consider the mental load that they will exert upon the user. Every addition of new information will add precious milliseconds of processing time for the user’s brain to digest that information. The trick is to strike the right balance between usefulness and cognitive burden within the context of your application.

Instant Results

The third approach differs greatly from the first two forms of search suggestions. In fact, instant results doesn’t offer query suggestions at all, but instead offers actual results to the user, often organised into a handful of categories.

Apple is perhaps the most well-known purveyor of this technique, both on their website and on OS/X’s desktop search, Spotlight. When I type the letters “mou” into the search box on Apple.com, I get five search results, the first being Magic Mouse. If I accept that suggestion, however, I am not taken to a search results pages with the query of “Magic Mouse,” but am actually taken directly to the Magic Mouse product page.

Apple.com

Last.fm

A key feature of instant results is that it can often diminish the need of having a dedicated search page. Instead, search results are provided in a contextual drop-down without leaving the page. The benefit is of course that the user is able to view results as they type, cutting out the additional time required to load a second page.

However, there are risks associated with instant results. First of all, the ability to provide query suggestions is lost since results are shown instantaneously. A second concern is that instant results usually only have space to show a handful of results, and often require quite a lot of space even at that. More specifically, instant results carries with it a much greater cognitive burden on the user than do either completion or query suggestion. It simply takes more time to parse through the multiple categories and apprehend all the metadata associated with each result.

Nutshell CRM

But there is a time and place for everything. Instant results work great when there is highly structured data neatly divided into a handful of categories. Take the forthcoming Nutshell CRM application, for example. The search field only covers three entities: leads, accounts, and contacts. In this context, query suggestions would be useless. It would also be very rare for a result to be buried very far down the list, so the user would almost never need a full page of search results. In this situation, instant results helps the user find what they’re looking for quickly, and with as little friction as possible.

In Summary

These three approaches differ greatly in both their usefulness and the effort required to use them. Completion is easy to understand and almost always results in a net benefit for the user. Suggestion, because it throws new ideas in the mix, is slightly more complex, but that is usually outweighed by its helpfulness. And finally, instant results work well for simple, highly structured data, but at times runs the risk of overwhelming the user.

In part two, we’ll look at some fresh ideas for search suggestions that blend elements from each of these three approaches.

Further Resources

• Tyler’s Flickr collection of search suggestion examples
• Peter Morville’s collection of search patterns
• Google’s original blog post announcing the release of search suggestions
• Jared Spool’s article on spending quality time with your search log
• Smashing Magazine’s article on designing the search box

The Deck

Tyler presented a talk on search suggestions at the Enterprise Search London Meetup on February 4, 2010. The presentation is available on SlideShare.

The problem

Consider the case when enterprise search is applied to a document management system. Each and every document might well end up in the same index, meaning that when searches are conducted it is imperative that confidentiality is maintained and users are only shown results they’ve got access to. This is typically known as ‘document-level’ or mapped security, where each piece of content is accompanied by the associated security metadata (known as an Access Control List, or ACL).

The problem is further compounded by the fact that search engines are being increasingly used to break down silos of data by making searchable, disparate sources of data that don’t share a single access control system.

Under the hood

This security metadata, or entitlements, get stored in the index, thus forming a part of the query evaluation process carried out by the search engine. What that means is that the set of documents found to correspond to the user’s query will be further whittled down by matching this metadata with the user’s credentials (authorisation). Given how intrinsic this is to the overall process of matching, this piece of the puzzle will generally fall under the remit of the underlying search engine. The other constituent part is the authentication or the process of establishing who the user is, so that his credentials (such as the groups he belongs to) can be provided along with the query. However, many providers of search technology have seen the former as their only remit in the security life cycle. To be fair, this lack of vendor support for the entire security life cycle has been somewhat understandable, given the plethora of authentication mechanisms out there. But the problem’s not going away.

How do we deal with it?

Given this, and the fact that almost every internal search project faces this dilemma, we often have to provide or facilitate a solution. We therefore chose to provide a TwigKit Security Module which offers integration with leading authentication providers such as Active Directory, LDAP servers, databases, OpenID, and proprietary systems, via a variety of methods and protocols such as form-based, basic, NTLM, Kerberos delegation, X.509 certificate exchange, and ‘container managed’ authentication. What this means is that if an organisation already has a single sign on infrastructure we’re can easily tap into that, but equally importantly, in greenfield scenarios, we can quickly implement an effective security solution. Thankfully, there was no need to reinvent the wheel since the Twigkit Security Module builds on the industry-leading Spring Security framework.

The TwigKit Security Module essentially fills three roles. First, it initiates the authentication of the user by using any one or combination of the supported methods. Second, once the user is authenticated, it will pass the identity of the user to search engines that support security using each platform’s specific methodology (e.g. the FAST Security Access Module). Finally, the TwigKit Security Module provides simple methods for restricting access to individual components or aspects of the search user interface. This allows us to in effect go a level deeper than document-level security, restricting access to individual fields (somewhat analogous to column level security in a database).

Some code

Let’s say our search user interface is built using the TwigKit Tag Library and provides access to some kind of product catalogue. In this case we might want certain parts of the product information to be only available to a group of ‘Product Managers’. This could be information stored in the search engine, such as as profit margins, or functional aspects of the interface like the ability to manipulate discount levels. In the following code sample you can see how you would go about displaying product attributes such as features and price, whilst restricting other parts to members of the group “PRODUCT_MANAGERS”:

...

<search:field fieldName="productFeatures" label="Features" />
<search:field fieldName="priceGBP" label="Price" prefix="£" />

<!-- Only show margins and discount level link to 'Product Managers' -->
<security:conditional allowedGroups="PRODUCT_MANAGERS" deniedGroups="">
	<search:field field="profitMargin" label="Profit Margin" />
	<a href="/EditDiscount/1234/">Change Discount</a>
</security:conditional>

...

Any information or components placed within the ‘conditional tags’ from the security library (highlighted) will only be visible to members of the groups mentioned, or with an alternative configuration, visible to everyone unless they belong to the groups being explicitly denied access. This example shows you how to interject at a user interface level, but TwigKit provides other hooks to apply similar logic using Query and Response Processors to correspondingly protect against potential injection attacks or do even do last minute checks and filtering. We will be covering processors in a separate post.

To wrap up, other tags from the TwigKit Tag Library provide more innocuous facilities such as providing particular attributes and features for authenticated users:

<security:userDetails authenticated="true">
	Hello <b>${user.name}</b> (<a href="/logout">sign out</a>)
</security:userDetails>

If you’ve come across other frameworks that tackle this, or  just have some thoughts on the matter, then we’d really like to hear from you – so put it in a comment or drop us a line.

Final thoughts

Unfortunately it’s hard to do such a complex topic proper justice in a short post like this. Admittedly it covers the problem at a very high level, ignoring other aspects of securing search such as ensuring the integrity of the search index when permissions change, and the overarching concerns addressed in security policies (such as physical and network security).

Despite these caveats we hope it you found it helpful from a functional perspective. Last, but certainly not least, I want to thank my good friend and search guru Christian Moen from Atilika for all his invaluable thoughts on the subject.

Job adverts for 'project managers' broken down by category on a popular recruitment site

Facets will quickly break down the dimensions in all the data we’re storing or even just the stuff that meets our search criteria. In either case we can get some form of statistical feedback e.g. on which top-level categories exist, their names and how many documents each represents. Take this search for positions as a ‘project manager’ as an example. Using faceted search, we can quickly see that some of these are are in the ‘Engineering’ field, with still more for IT professionals.

Not only does this give the user insight into what information is available, but also guides them in their search, allowing them to slice and dice the data to get precisely to the information they’re after. The question is, how do we best represent this information and make it useful (and meaningful) to us?

As you saw in Tyler’s previous posts in most cases there might be sufficient utility in just getting the broad strokes, preferably in a manner that minimises the cognitive burden of taking it in. In some cases proportions may give us the visual cues we’re after. For example it may be useful enough for us to see that there are 1) almost no orders pending shipment this week (phew), 2) a bunch in transit, with 3) the vast majority already delivered. And, thanks to faceted search all the detail on each group or dimension is a mere click away.

Facet information displayed with TwigKit as a 3D pie chart

To achieve this, the TwigKit UI libraries provide widgets that will turn facet information from the search platform into pretty pictures, charts and graphs. Traditionally, a developer would have written some code to extract the necessary information from the facet, integrated a visualisation library, and displayed the result on a web page. But we’ve done all that for you.

In the code snippet below you can see how to create visualisations using the TwigKit JSP Tag Library. All you’d need to do is specify which facet to display, the format (such as column, line or pie chart) and the result is an interactive visualisation – where clicking a particular aspect will further refine your search. Easy as pie :)

<widget:facetChart
	type="Column3D"
	facet="${response.facets.manufacturer}"
	numberOfFilters="6"
	color="ffbb33"
	backgroundColor="fbfbfb"
	query="${query}"
	width="700"
	height="250"
	title="Top Manufacturers"
	subTitle="Number of products per manufacturer"
	showAverage="true" />

Simple example of Facet information on products, broken down by manufacturer and represented as a column chart.

The important thing here is that search engines have a myriad of ways to efficiently mine vast volumes of data, providing insights that simply weren’t achievable in the traditional relational paradigm. However it is often the little things that transform that analysis into meaningful, every day tools that truly alters the way we consume information.

Just before Christmas we put the finishing touches on a prototype internal search application for British broadcaster ITV. We’ll be working with ITV in the coming months to roll out the application across their entire organisation, but we wanted to give you a sneak peak in the meantime.

Information Architecture: The Big Picture

The application searches ITV’s product catalogue, which contains a large archive of television shows and scheduling information. Our largest effort surrounded the information architecture for the application. Our goal is always to create an empowering user experience that helps people easily digest information and nimbly find what they’re looking for. To accomplish that goal, we first had to understand the data ourselves, and, just as importantly, figure out how the different user groups at ITV understand the data.

Catalogue/Schedule toggle, search field, and hierarchy of Programme, Series, Episode, and Production.

We realised that seemingly everyone at ITV was familiar with the organisation’s programme hierarchy of programmes (such as “The Office”), series (Season 2), episodes (Episode 1), and finally productions (International Version). Because it was so ubiquitous, we chose to make it the backbone of the search application. We placed the hierarchy prominently at the top of the page and branded each piece of the hierarchy with its own colour, which we also used to categorise each search result. Our goal was to help users easily traverse the hierarchy and to instinctively understand the level of each result.

Every detail page has a breadcrumb trail.

The hierarchy remains prominent after the user drills down to a single result. We included a breadcrumb trail so that the user is aware of her current position. In addition, every detail page (of a programme, for instance), lists all of its children (series, in this case). This means the user can continually navigate both upwards and downwards throughout the hierarchy.

The page of a programme always lists all of that programme's series.

Tags, capsules, category markers, and metadata boxes

Information Architecture: Presenting Metadata

In addition to thinking about the top-level hierarchy and navigation, we also focused on the presentation of metadata. We wanted the metadata to enhance the search experience and help the user hone his search. We also wanted to prioritise the information, make the important bits really jump out. We worked to shape the data in two primary ways:

• Any metadata that the user can facet on gets styled as a capsule. We chose to style keywords in tag-like containers, and other metadata, such as “partners,” in pill-shaped capsules. This communicates to the user that the items are clickable and will refine the search query.
• Important metadata should be visually prominent. (It’s obvious, we know.) We created large boxes to display the most important metadata and placed the boxes at the top of the page. Users should not need to hunt for important information, it should jump out at them.

The tag cloud indicates the popularity of the top keywords.

Data Visualisation

If information architecture was our top concern, data visualisation was a close second. The ITV search application offers two primary perspectives on the data—a catalogue-centric perspective and a schedule-oriented view. We utilised visual metaphors that fit best with each of the two perspectives: a tag cloud for for the keyword-based catalogue view, and a date histogram for the scheduling information.

The tag cloud.

The date histogram indicates how many scheduled television shows meet the search criteria on a given day.

Gallery

The catalogue view

Tag cloud

Query suggestions

Detail page of a production

Schedule with the date histogram

]]> http://blog.twigkit.com/behind-the-scenes-at-itv/feed/ 1 http://blog.twigkit.com/data-visualisations-in-search/ http://blog.twigkit.com/data-visualisations-in-search/#comments Wed, 23 Dec 2009 13:07:08 +0000 Tyler Tate http://blog.twigkit.com/?p=59

Last.fm's bar list

One of the greatest achievements of modern search engines is the ability to see the relationships between many different facets of the data. This is enabled by what’s known as “deep faceting” – knowing exactly how many results there are for each facet of the data.

The authors of DataFlow put it this way: “By giving shape to data, we… provide access and insight to the hidden patterns of meaning.”

In my earlier post, Precise to a Fault, I argued that result counts are most useful when they are used to indicate proportionality between facets. In this post, I’d like to look at some of the methods of presenting search-based information that help yield insight into how the data is related.

Tag Clouds

The first and most ubiquitous example is the tag cloud. It instantly communicates which terms are popular, and which are not. It is perhaps best suited for displaying words or short phrases where order is not important, such as tags on Flickr.

Flickr's tag cloud

Horizontal Bars

But tag clouds are ill-suited for more complex information where order is important. For linear lists, horizontal bars can be quite effective, essentially measuring each item against a standard. Last.fm pulls this off nicely to indicate how many tracks I listened to from each of my top artists over the last year.

Last.fm's bar list

Pie Charts

While tag clouds and horizontal bars are both easily achievable with traditional HTML and CSS, pie charts can also be a viable option with more creative techniques. Obviously pie charts require their components to total 100%, and so are quite useful for showing how individual slices compare to the whole.

Every year graphic designer Nicholas Felton compiles assorted statistics from his life into an amazingly illustrated report.

Histograms

Histograms can be quite useful for compactly representing large amount of chronological information. We were recently working on a project for a media company and needed to show how many times certain TV shows were scheduled to run over the next six weeks. We accomplished this with a histogram that plotted each day as a vertical bar, with the height of each bar indicating the number of shows for that day.

TwigKit combines a histogram visualisation with a range selection interface to give you insight into what you're selecting.

Mix and Match

Visualising Information can be a powerful tool for revealing the interconnectedness of the data. While each of these four visualisations has it’s own advantages and disadvantages, what’s important is that each piece of information be presented in a way that best reinforces it’s meaning.

]]>

Unfortunately, pagination doesn’t always do the best job of guiding the user through the search results in the most beneficial manner. I’ve noticed three common problems.

Problem 1: The “Last” button

Time Out's pagination includes a link to the very last page of results.

If the first page of results is the engine’s best guess at what the user is interested in, the last page of results is inherently the engine’s worst guess. Which begs the question: why do you often see a link to the last page of results?

For the user to find what they’re looking for, it’s important to get highly relevant results in front of them. Encouraging users to look at irrelevant results is simply counter-productive. Avoid having a “last” link.

Problem 2: Pagination above results

Career Builder's search page has pagination above the results.

A second harmful practice is placing the pagination above the search results. Again, the purpose of search is to present the most relevant results first. It doesn’t make sense to encourage users to skip to page 2 before they’ve encountered all of page 1.

Problem 3: Next

If the first two problems are akin to sending the user on unnecessary detours, the third problem is like not giving the user a green light when we should.

Google, not surprisingly, got it right by giving their "next" button a prominent graphic and bold text.

Imagine you enter a query, scroll down the page as you look over the results, and reach the bottom of the page without finding what you’re looking for. What would you want to do next? Go to page 7? No, you’ll most likely want to go to the next page of results. So, why doesn’t the pagination always offer us a big, whopping “Next” button?

In Summary

TwigKit's default pagination.

All three of these side effects are the result of one illness: forgetting to optimize pagination for helping users move from highly relevant results to less relevant results in a proper progression.

For TwigKit, we also wanted to make it easy for people to get back to the earlier pages of more relevant results once they have navigated to a later page.