Monday, September 26, 2005
The Power of Default Values (Jakob Nielsen's Alertbox):
"Summary:
Search engine users click the results listings' top entry much more often than can be explained by relevancy ratings. Once again, people tend to stick to the defaults...
...What is interesting is the researchers� second test, wherein they secretly fed the search results through a script before displaying them to users. This script swapped the order of the top two search hits. In other words, what was originally the number two entry in the search engine's prioritization ended up on top, and the top entry was relegated to second place.
In this swapped condition, users still clicked on the top entry 34% of the time and on the second hit 12% of the time...
...The researchers also assessed whether the search engine in fact placed the best hit on top. Obviously, this can only be determined by having humans judge which websites are best, and there is no objective criterion for "best website to answer a given question." Still, the researchers derived relevancy ratings by averaging the judgments of five people, and that's probably as good as you get in terms of estimating information relevance.
On average, the top hit in the original search listings was judged as being the most relevant 36% of the time; the second hit was most relevant 24% of the time; and the two hits were equally relevant 40% of the time. In other words, the search engine tended to be right, but was wrong one-fourth of the time. (If the two hits are equally relevant, it doesn't matter which one is placed on top, so I counted these as being right.)
Given how often the search engine was wrong, users clicked the top hit far too frequently. And when the two top hits were swapped, too few users changed their behavior. In other words, we can conclude that there is a strong bias in favor of clicking the top link, though not so strong that link quality is irrelevant..."
"Summary:
Search engine users click the results listings' top entry much more often than can be explained by relevancy ratings. Once again, people tend to stick to the defaults...
...What is interesting is the researchers� second test, wherein they secretly fed the search results through a script before displaying them to users. This script swapped the order of the top two search hits. In other words, what was originally the number two entry in the search engine's prioritization ended up on top, and the top entry was relegated to second place.
In this swapped condition, users still clicked on the top entry 34% of the time and on the second hit 12% of the time...
...The researchers also assessed whether the search engine in fact placed the best hit on top. Obviously, this can only be determined by having humans judge which websites are best, and there is no objective criterion for "best website to answer a given question." Still, the researchers derived relevancy ratings by averaging the judgments of five people, and that's probably as good as you get in terms of estimating information relevance.
On average, the top hit in the original search listings was judged as being the most relevant 36% of the time; the second hit was most relevant 24% of the time; and the two hits were equally relevant 40% of the time. In other words, the search engine tended to be right, but was wrong one-fourth of the time. (If the two hits are equally relevant, it doesn't matter which one is placed on top, so I counted these as being right.)
Given how often the search engine was wrong, users clicked the top hit far too frequently. And when the two top hits were swapped, too few users changed their behavior. In other words, we can conclude that there is a strong bias in favor of clicking the top link, though not so strong that link quality is irrelevant..."
Comments:
Post a Comment
