Search engines are among the most-used resources on the internet.
However, even today's most successful search engines struggle to
provide high quality search results. According to recent studies as
many as 50 percent of web search sessions fail to find any relevant
results for the searcher. Researchers have proposed social search
techniques, in which early searchers provide feedback that is used to
improve relevance for later searchers. In this paper we investigate
foundational questions of social search. In particular, we directly
assess the degree of agreement among users about the relevance ranking
of search results. We developed a simulated search engine interface
that systematically randomizes Google's normal relevance ordering of
the items presented to users. Our results show that (a) people are
biased toward items in the top of the search lists, even if the list is
randomized; (b) people explicit feedback is not biased and (c) people's
shared preferences do not always agree with Google's result order.
These results suggest that social search techniques might improve the
effectiveness of web search engines.