Searching the Search Engines

Update

This column appeared in May, 1998. As often happens in the digital world, it wasn't long before almost everything had changed. The original column appears below, but please note that as of early 2000:

• Yahoo! still finds the major sites effectively, but the human-indexing system just can't keep up with the wave of new submissions. You'll miss a lot if you rely on Yahoo! for your gay travel research.
• Excite and Webcrawler now get very useful results most of the time;
• HotBot and Alta Vista have slipped.
• The new system at GoTo works well; not many sites are paying for a listing yet, but non-paying sites are listed appropriately.
• A new search engine, Google, is consistently effective, and would undoubtedly get the #2 ranking if I did the test again.
• Infoseek, now consolidated with Go, continues to get excellent results. Lycos remains in the middle, not useless, but you can do better.

Search tips

You'll get more from your searches if you pick two or three engines, and familiarize yourself with how they operate. On some, you must type a plus sign before each word: to search for gay hotels in Paris you'd type:
                  +gay +hotel +paris
You'll usually get better results by using several words. If the concept of gay or lesbian is important (you want a gay cruise, a vacation with other lesbians, etc.) then include one of those words in your search terms. If you're looking for travel or vacation information, include one of those words in your terms.

From Yahoo to Webcrawler, from "gay Thailand" to "lesbian vacations": Which search engines are most useful for gay and lesbian travelers as they surf the Web?

Above: Sasha Alyson found enormous differences between popular search engines, when they were asked to search for gay travel topics.

The author: Sasha Alyson is the founder of Alyson Publications, the country's leading publisher of gay and lesbian books. He sold that company in 1995 to start Alyson Adventures, which specializes in active and adventure vacations for gay men and lesbians.

Would you like to go hiking with other gay men and lesbians? The following information should be helpful:

gay sex photos free naked men xxx pictures nudes males gay sex jeff stryker erotica stories underwear briefs

What, you didn't find that helpful? Actually, neither did I. Yet that was the "description" of the #1 site that came up on a popular search engine, when I asked it to find websites that included the concepts "gay" and "hike".

Experienced web surfers are accustomed to this. The problem is compounded for gay and lesbian users, however. That's because much of what's on the web is porn-for-pay, and those porn sites work overtime to pump up their ratings with the various search engines. Once you type the word "gay" or "lesbian" to help define your search, you open the door to these hordes.

Fortunately, some search engines prove smarter than others at detecting which sites are most relevant to your inquiry. After too many frustrating hours of watching search results download, only to find a list of spam, I undertook to test them.

I took eight queries -- word combinations such as "gay hotel paris", "gay hike", "lesbian vacation" and "gay cruise ship" -- then called up the major search engines, and looked at the first ten results provided by each engine to each of my queries. If you think I must have been bug-eyed by the end of all this, you're right.

I scored the results by asking, of each site listed, "How relevant is this likely to be for the typical user who types in the same query?"

In response to the query "gay cruise ship", a listing for RSVP Travel received a top score. The site of a gay travel agency that sells cruises also got points, but not as many. Porn, spam, and irrelevant sites lost points. So did the home page of Mark, who is gay and writes of a memorable night on a cruise ship. Good for you, Mark, but that page doesn't belong in the top ten listings.

And the winners are:

InfoSeek (www.infoseek.com) and Search (www.search.com). came out on top -- a surprise to me, as I had barely heard of either one. They share a database, but InfoSeek consolidates all pages from the same site into a single entry, yielding a more concise list. Both did an excellent job at filtering out spam, and getting most of the best sites.

Alta Vista (www.altavista.digital.com) was strong all-round, but especially with hotel listings.

HotBot (www.hotbot.com) found more right-on-target sites than any other engine, but was weaker at filtering out the porn and spam.

Lycos (www.lycos.com) was generally strong, but weak at finding hotel information.

Yahoo (www.yahoo.com) actually hires human beings to look at each site before it gets listed. (Others use so-called "artificial intelligence" software.) Consequently, for most topics, Yahoo produced fewer junk sites than anyone else. Yet occasionally, your subject just doesn't fit into their scheme: I had no luck tracking down gay cruises, for example, on Yahoo. It also has a smaller database, and missed many relevant listings. Those humans go home at night, while computers keep on analyzing.

Excite (www.excite.com) and Webcrawler (www.webcrawler.com) got rock-bottom scores. These turkeys share the same database, but apply different search algorithms, and so yield slightly different results. But "Garbage In, Garbage Out," as they say in the computer business, and both these search engines proved virtually useless at producing relevant listings. (It was Excite, by the way, that came up with the odd response to my search for the "gay hike" query. The porn site that got #1 listing includes the phrase, "If you are offended by the gay lifestyle, homosexuality, images of naked men, gay sex, or any type of sexual talk, take a hike. Leave now." None of the higher-ranked search engines made the error of believing that this site was about hiking.)

Two major search engines were not rated:

GoTo (www.goto.com) is undergoing a major overhaul. Commercial sites can now pay to get a top listing; GoTo claims this will help increase relevancy. They could well be right; it's worth trying them out in a few months.

Northern Light (www.nlsearch.com) is heavily oriented towards news and text. It's not the place to look for gay vacation ideas; for current events, however, it's the place to start.

A table summarizing these results (along with links to each engine) appears below.

My conclusion: No one search engine will find everything you want. Get in the habit of using any two or three, from the top six. If you've been relying on Excite or Webcrawler, and could never understand why other people found the web so useful and you didn't -- well, now you understand.

May, 1998