The Dào of Search

Search is an evolutionary, iterative process. Like a Grizzly Bear foraging for food in the forest, people jump from one information source to the next as we seek to satisfy our curiosity.

Presented at Apache Lucene EuroCon 2010 in Prague, “The Scent of Search” seeks to apply the principles of Information Foraging Theory, and in particular, Information Scent, to the usability of search interfaces. Below are the main recommendations we set forth in the talk.

This morning I got out of bed, ate my cereal, took my shower. Everything was proceeding pretty predictably. But then I did a Google search — usually a pretty mundane task — but this morning, Google looked very different than it did yesterday.

Word on the street is that Google is rolling this new design out to everyone over the next 48 hours. As with any change, some people are bound to complain, but I think the redesign introduces many significant improvements.

“There’s no shortage of problems with search today,” says Peter Morville at the end of Search Patterns, his most recent book. Throughout the book, Morville chronicles the challenges of search and effectively communicates the best practices of building usable search experiences. While the book more than adequately accomplishes it’s stated goal — to foster greater cross-disciplinary collaboration by increasing search literacy — it would have been an even better book if the author had proposed more of his own innovative solutions to search’s problems.

The event consisted of 12 different speakers each presenting for exactly 20 minutes, with about 10 minutes of Q&A after each. I particularly enjoyed the presentations from the major search engines: Yahoo, Google, Bing, and Wolfram Alpha. A topic that seemed to arise in each of those talks was how query reformulation data can provide a feedback loop to make search better. But without further ado, here are my summaries of each talk.

You used to be expected to type for yourself. But today people have come to expect a reasonable amount of help at even this task. Our phones now help us form correctly-spelled words, our browsers fill in long addresses after we’ve typed only a few characters, and search engines recommend searching for “Humphrey Bogart” after we’ve typed just “boga.”

But not all as-you-type search suggests are created equal. Careful observation seems to reveal three different approaches: completion, suggestion, and instant results. These approaches range in cognitive burden on the one hand, and utility on the other. We’ll look at several examples of each and consider when they should be used.

When a search engine is brought to bear on content with restricted access, it becomes evident that security and preserving the integrity of permissions can be an important and often thorny issue. Consider the case when enterprise search is applied to a document management system. Each and every document might well end up in the same index, meaning that when searches are conducted it is imperative that confidentiality is maintained and users only see results to which they have access.

Facet information displayed with TwigKit as 3D pie chart

Most modern enterprise search platforms provide some inherent capability to illustrate the shape and nature of the data within. Take for example faceted search. Facets will quickly break down the dimensions in all the data we’re storing or even just the stuff that meets our search criteria. In either case we can get some form of statistical feedback e.g. on which top-level categories exist, their names and how many documents each represents. This will not only give the user insight into what information is available, but also guides them in their search, allowing them to slice and dice the data to get to the information they’re after. The question is, how do we best represent this information and make it useful (and meaningful) to us?

Just before Christmas we put the finishing touches on a prototype internal search application for British broadcaster ITV. We’ll be working with ITV in the coming months to roll out the application across their entire organisation, but we wanted to give you a sneak peak in the meantime.

Last.fm's bar list

One of the greatest achievements of modern search engines is the ability to see the relationships between many different facets of the data. This is enabled by what’s known as “deep faceting” – knowing exactly how many results there are for each facet of the data.

The authors of DataFlow put it this way: “By giving shape to data, we… provide access and insight to the hidden patterns of meaning.”

In my earlier post, Precise to a Fault, I argued that result counts are most useful when they are used to indicate proportionality between facets. In this post, I’d like to look at some of the methods of presenting search-based information that help yield insight into how the data is related.

The purpose of search is to help people find what they’re looking for as quickly as possible. Search engines attempt to facilitate this by taking the user’s query and responding with results, placing what it thinks are the most relevant results first.

Unfortunately, pagination doesn’t always do the best job of guiding the user through the search results in the most beneficial manner. I’ve noticed three common problems.