Best Bets in Searching the Internet

Once you get connected to the Internet, you need to know how to find sites and newsgroups that will be important and of interest to you. Some people enjoy wandering, or surfing from site to site, going wherever the links take you. But sometimes you just don't have the time and you need something specific and you need it now. When it's one of those times, you simply have to use a search engine.

Below, is a compilation of useful information on how to find what you need and where you'll most likely find it.

Search engines

Which search engine should I use?

Best Bets

Ten Essential Tips

Seven Stupid Mistakes

If nothing else works

Search engines

A search engine is a service that indexes, organizes, and often rates and reviews Web sites.  It can help you find what you're looking for among the millions of documents available on the Internet. Different search engines work in different ways:

• Some rely on people to maintain a catalog of Web sites or pages.
• Some use software to identify key information on sites across the Internet.
• Some combine both types of service.
• So when you search their "holdings," you're bound to get different results.

Which search engine should I use?

No search engine keeps track of all the content on the Internet. Even the major search engines—such as Yahoo, Alta Vista, Infoseek, Lycos, and Snap—won't give you everything. (Some studies indicate that even the top search engines find less than half of what's really out there!).  So try several search engines to see which produce the most useful results for the types of information you usually look for. Whenever a search fails to produce the results you want, try another search engine.

Here's a quick introduction to some of the major search engines:

Alta Vista - At AltaVista, you can search for keywords or type your query in the form of a question. Other features include Babelfish, which translates sites in several languages; Photo & Media Finder, which searches for images; and Family Filter, which helps you limit the types of results produced by the search.

Infoseek - Infoseek's search engine boasts an extensive catalog of sites. Infoseek has a directory, too, that categorizes sites by topic using software. Some sites have been reviewed and are recommended.

Lycos  - Like Excite and Infoseek, Lycos is a search engine with a directory, organized by subject.

Google - Google Inc. is one of the fastest growing search engines on the web.  The key benefits of Google's search engine include its ease of use and clean interface, and highly relevant results.

Snap - Snap is a directory of Web sites that the Snap directory team has chosen, divided into topic areas, and summarized. Within each topic area, sites are divided into more specific subjects to help you locate the type of information you want. Snap's top picks are marked with a Best icon.

Excite  - A hybrid search engine, Excite fulfills the traditional search engine definition (it is always visiting Web sites and cataloging them using a software program) but also has a directory—a subsection of its cataloged sites that have been reviewed by people.

Yahoo! - This is actually a directory rather than a search engine, meaning humans compile and categorize the sites it searches. So you may get fewer results. But Yahoo! makes it easy to search for sites by category and to continue your search using a search engine if you want to.

MetaCrawler - Rather than keeping its own database of Web sites, MetaCrawler conducts your search simultaneously on several search engines including Excite, Infoseek, Lycos, and Yahoo! This takes a little longer than using just one search engine, but it is likely to yield a more comprehensive set of results.

Major search sites generally provide more than a search engine for finding Web sites. They also allow you to look up information such as recent news stories, newsgroup postings, reference material (such as dictionary entries and maps), and e-mail addresses, street addresses, and telephone numbers of business and individuals.

Here are two guidelines for picking a search engine:

• If you're looking for a broad, common topic, such as how to buy a personal computer, use a search engine that tends to give you fewer results of higher quality (usually the type that relies on people to catalog sites), such as Yahoo!
• If you're looking for a rare topic, such as 19th-century Hungarian authors, use a site that may be less discriminating but yields more results (usually the type compiled by a software program), such as Excite.

Ten Essential Search Tips

Here are ten essential searching tips that range from basic, common-sense searching strategies, to advice on when to use specialized search tools.

Choose The Best Search Tool

Every search is different, and you'll get the best results if you choose the best search tool for each unique situation. The most fundamental choice is whether to use a search engine vs. a directory.  Search engines are actually full-text indexes of Web pages. They're similar to telephone white pages, which contain nothing more than lots of names with phone numbers.  Directories, on the other hand, are collections of links to Web sites compiled by people, not software robots.  Directories are similar to telephone yellow pages, because they are organized by category or topic, and often contain more information than bare-bones white pages listings.

As a general rule, if you're searching for a specific fact or particular document, use a search engine like AltaVista or Google. On the other hand, if you want to browse for information to get a general overview of a topic, use a directory like Yahoo or the Open Directory Project.

Consider using specialized search tools for certain types of information. For example, if you're looking for a person or a place, use specialized people finders or maps and directions search sites.  There are specialized search tools for virtually every topic or subject and each search engine has its own tools.  It’s worthwhile to check them out before you start a search.

Use Multiple Search Terms

If a stranger came up to you and said "anchovy paste!" or "sibberidge!" how would you respond? Most people would either laugh, or ask questions to find out what the person was talking about. Search engines don't have that option—they simply make a guess and provide the best results they can with limited information.

Good queries are rich in keywords that can narrow the focus of a search. Better queries than those above would be "anchovy paste in licorice tahini pizzas" to get the mouth-watering recipe you're looking for, or "sibberidge marriage banns" for more information on a wedding tradition common in the Middle Ages.

Use Natural Language Search Strings

Most search engines handle natural language queries quite well. In fact, sentence structure can provide valuable clues to the search engine that might be lost if you only enter a few search keywords.

Rather than entering a non-grammatical string of keywords, try a plain-language question. Instead of "snow blowers Minnesota" try "Where can I find a snow blower dealer in Grand Marais, Minnesota?"

Capitalize Proper Nouns

Most search engines are case-sensitive. This means you'll get much better results if you capitalize names, places, and other proper nouns. Results for "John Bull" will likely be about Britain's mascot; results for "john bull" may also contain off-topic (though strictly speaking quite legitimate) information about restrooms in bullfighting arenas.

Don't Make Dumb Mistakes

Well, duh! Obviously you shouldn't make dumb mistakes. But do you really know the range and variety of mistakes you can make while searching? Take a look at Seven Stupid Searching Mistakes to see some of the real doozies made by even smart, well-intentioned people.

These are fairly basic tips. For more advanced tips, from when to use Boolean logic to what to do when search engines fail, read on.

Be Careful With Boolean

Most search engines allow you to use Boolean operators (and, or, not) to focus your search. Unless you have lots of experience using Boolean logic, you're probably better off not using it.

There are two reasons for this. First, Boolean searching works slightly differently in each search engine. Unless you know exactly how Boolean operators work for a specific engine, you may end up misusing them and actually hampering your results.

Second, when you use Boolean operators you override the numerous other factors that determine how the search engine calculates relevance. In essence, you're saying to the search engine "I'm smarter than you so let me tell you how to do your job." This is rarely a winning strategy, online or off.

Think Before You Click

There are two parts to a successful search equation: A well-formed query, and an accurate, reliable result. You can save a lot of time by quickly analyzing your search results list before you click through to any specific pages.

For one thing, the best result may not be #1 or #2. Get a feel for all of the results on a page and make your own judgement. An important part of this analysis is to compare the title and the URL. Do they make sense together in the context of your search?

For example, if two documents are titled "One Hundred Faces of Faith," the page from www.vatican.va/faith.html will likely be quite different from www.babepix.com/faith.htm. Of course, the one that meets your needs depends on what you were searching for.

Learn From Expert Searchers

The best way to learn to be a great searcher is to learn from people who are acknowledged masters. Two outstanding magazines that regularly publish tips and techniques from experts are

“Online” at http://www.onlineinc.com/onlinemag, and
”Searcher” at http://www.infotoday.com/searcher/default.htm.

Develop Effective Searching Habits

Like any worthwhile skill, searching is something that needs to be practiced. It's important that you practice techniques that give you effective results. Keep track of the types of queries that give you results and use those tactics again.

Skilled searchers don't just pound away on search engines until they get results. They develop habits of thinking that allow them to quickly and effectively find what they're looking for.

Don't Give Up!

Let's face it, sometimes nothing you do seems to produce good results. Often, when your searches come up empty and you feel like you've slammed into a dead-end, it's tempting to just throw up your hands and walk away.

No need to despair. Check out these strategies and coping mechanisms for those times When Search Engines Fail. There's almost always something you can do to salvage a seemingly hopeless search.

Seven Stupid Searching Mistakes

In the lighthearted spirit of the popular books for "idiots" and "dummies," here's a look at seven common blunders that are virtually guaranteed to deliver useless, nonsensical, or completely worthless search results. Some of these gaffes might surprise you. But once you recognize them, it's easy to banish these little gremlins forever from your Web search tool kit.

Sputtering on "Stop Words"

Some search engines simply ignore certain words. They are never used to find a matching document, despite what amounts to a direct command when you type them into a search form.

These are called "stop words" because the search engine doesn't when they're found in its index. Why not? Because stop words are either too common to generate meaningful results, or are parts of speech like adverbs, conjunctions, prepositions, or forms of "be" that mean nothing unless they're part of a phrase with more "important" nouns and verbs.

If you use a stop word in a query you may get wildly irrelevant results. For example, the phrase "searching the Web" contains two stop words: "the" and "Web." Though it's not a particularly common word, Web is used so frequently on the Internet that it's virtually worthless as a finding aid.

Stripping out the stop words, "searching the Web" becomes "searching," which will naturally lead to results describing everything from criminal manhunts to quests for enlightenment—and if you're lucky, maybe even something about searching the Web.

Bungling with Boolean

Boolean operators, like "and," "or," and "not," can help narrow search results—when used properly. The problem is that Boolean operators, because of their apparent simplicity, appear to be easy to use. Maybe, maybe not.

According to Ran Hock, author of The Extreme Searcher's Guide to Web Search Engines, search engines implement Boolean features in different ways. For example, while some accept a simple "not," others require "and not" for the same effect. Additionally, some engines require that Boolean operators be capitalized, while others do not (or and do not?). Fortunately, search engines usually have a note or two on tipes for using "their" engine and it's worthwhile checking them out beforehand.

Being Ever So Vulgar

Vulgar comes from the Latin vulgus, meaning common. Like some educated sophisticates, search engines have a problem with common words. It's not that they're being snotty or pretentious. It's that some words are so common that they appear in literally millions of documents, making them virtually useless as a finding aid.

Take weather, for example. There are thousands of sites providing weather information, from local forecasts to elaborate treatises on meteorology. Tighten your query by using focusing words to narrow the scope of your search. Rather than merely searching for "weather," construct a query like "Cicely Alaska annual snowfall," or something equally specific.

Looking for a Rose, By Any Other Name

Many words have multiple meanings. For example, when using the word "bond" we have to ask ourselves, are we looking for information about financial bonds, chemical bonds, or even James Bond? Think about the possible other meanings and phrase your query accordingly.

Search engines are also easily confused by heteronyms, words that are spelled identically but have different meanings when pronounced differently. For example, "lead," pronounced LEED, means to guide. Pronounced as LED, though, the word refers to the metal element.

Committing Capital Offenses

Yet another problem for the searcher is whether to use capital letters in a query. Some engines are case sensitive, while others are not. As a rule of thumb, it's a good idea to always use lower case letters when you search. This will typically return results that contain both upper and lower case letters.

If you use uppercase letters in a query to a case sensitive engine, results will only include documents that also use upper case letters. This is usually a good thing for proper nouns like names or places, which use initial upper case letters anyway. But it might cause you to miss other documents where case-sensitivity is less important.

Close, But No Cigar

Most search engines do a good job at matching simple phrases, like "Kosovo refugees," or "space shuttle missions." You run into problems, though, with a phrase like this section's title. Searching for "close but no cigar" on one major engine (which shall remain mercifully unnamed) provided this link as its number two pick: The Common Cold: Relief But No Cure. Definitely no cigar!

The distance between one word and another in a document is referred to as proximity. Some search engines will give a positive result if your query words appear anywhere on a page, whether or not they are near each other, or are used together in a phrase.

If you're searching for something where your keywords must be near each other to get good results, your best bets are AltaVista, which allows you to use the NEAR operator, and Lycos Advanced Search, which offers several ways to specify proximity.

And now for the number one most common searching mistake:

Searching for Hits in all the Wrong Places

If you're determined to find what you're looking for on the Web, be sure you're using the right tools for the job. Search engines vary widely in scope, function, and quality. You'll waste a lot of time if you don't choose the best search engine for each specific searching task.

Should you use Excite or HotBot? AltaVista or InfoSpace? Choose the Best Search Engine shows you how to select the best search tool for any topic, by analyzing your needs and comparing them with the strengths and weaknesses of each search service.

And if you get stuck, don't be hard on yourself. One of the most ridiculous misconceptions I've ever heard is that "you can find anything on the Internet." This is about as true as saying that there are diamonds in every coal mine.

Though it may sound strange reading this in a Web search Guide, sometimes your best bet for finding information is to log off and take a trip to your local library. Libraries have tons of resources that aren't available on the Web. And librarians are trained experts who are usually more than willing to help you find what you're looking for. When you're getting nowhere on the Web, take advantage of these "human search engines."

Begone, Mistakes!

As you gain experience searching the Web, avoiding these seven searching mistakes will become almost second nature. Whenever you get weird or unexpected results, take a close look at your query and try to figure out what happened. You might discover yet another mistake to avoid.

If Nothing Else Works...

We love em, we hate em. Sometimes no matter how hard you try, you can't get the results you need. Then what?

Search engine failures range from swamping you with irrelevant results to surfacing no results at all, with degrees of failure at all levels between these extremes. Here are five common situations when search engines fail, with specific tips on how to turn failure into success.

Forty-four Gazillion Documents Found

The most common failure isn't really a failure, but it sure is annoying. The search engine isn't finding the needle in the haystack—it's returning the entire haystack.

This type of failure is almost always the result of one of two things, both of which are easily fixed. First, your query was probably short, just one or two words. And second, your query words were so common that they were found in countless documents on the Web.

If you use common words in a query, they should always be part of a multi-word phrase, at least three or more words that form a sentence fragment. Whenever you can, use synonyms for common words or better yet, think of rare or uncommon words that describe your topic. The trick here is not to outsmart yourself by being too clever. You don't want to use words that aren't likely to be used by most Web authors.

If you must use a common word or phrase, use inclusion and exclusion operators (the plus and minus signs) to tighten the search focus. For example, "+George Washington -state -D.C. -university" finds documents about George Washington, greatly reducing documents related to Washington state, Washington D.C., and George Washington University.

Your Query Found No Documents

Although not common, it is possible for a search engine to come up with zilch for your query. It's possible that the info your searching for simply isn't on the Web. This is more likely if you're looking for a person, a small or obscure company, or an esoteric fact.

However, more likely, you're searching in the wrong place. Go back to the section on Which Search Engine Should I Use. But basically, for people use a specialized people finder, for companies, start with a business service like Transium or Kompass and for facts, try Ask Jeeves or one of the specialized reference services.

Remember that the major search services vary hugely both in size and scope. Yahoo (less than a million documents) and Webcrawler (2 million) are puny compared with AltaVista, HotBot, and Northern Light, all of which have more than 100 million pages indexed. If you must use a general-purpose engine, try your query with a meta search engine like Dogpile, Mamma, or Savvy Search.

The 404-Not Found Blues

One of the stupidest, yet aggravatingly common mistakes made by Web masters make is to "tidy up" their Web site in the same way they might tidy up their personal hard disk. By bringing order to the chaotic mass of stuff on a Web site by moving everything into well-organized folders with meaningful filenames they also inadvertently break every bookmark, ruin every search engine link, and void every printed URL that formerly allowed people to find the page.

Luckily, this "404," or "document not found" message is being sent to you from the same computer where the document used to be located. With luck, it's still there, and you just need to do a bit of snooping to find its new location. To do so, start with the "URL back-up" strategy. Put your cursor after the "htm" or "html" in the address window, and delete the filename, stopping at the first forward slash. Hit enter. Continue this process until you've either found a menu with a link to your document, or the root URL, which should have either a direct link or a search form you can use to help locate your document.

In most cases, if the document is still on its original server, you'll be able to find it with a little searching.

Moved; Forwarding Address Unknown

Web documents, like people, sometimes move.  Students graduate; businesses move from modest sites on free communities to larger sites under their own domain name; people change ISPs. When this happens, your best bet is to search for either the title of the document or the person who authored it. You can do a title search by simply enclosing the title in double quotes and searching for the phrase. But a better way is to do a "field search" specifically on the title.

Field searching is an advanced function offered by most search engines. It allows you to limit the search to the HTML title that appears on the title bar on your browser (it's also used as a bookmark title). You can do field searching for titles, images, and other specific parts of Web pages.  If you search for the author of the document, try HotBot's search for "the person" option.  Remember, when searching for people, you should always capitalize the name and enclose the name in quotes.

Nothing Works: Now What?

You've tried everything. A search query that worked just fine last week now gives you nonsense results, with your target document nowhere to be found. What is going on? A dirty little secret about search engines is that pages are dropped from their indexes all the time. Sometimes thousands of pages. Sometimes these abandoned pages reappear in the index, sometimes not. Official explanations range from "pruning" the search engine database, in an attempt to keep listings fresh and relevant, to "maintenance" where the index is rebuilt from scratch. All and well, unless the specific document you're looking for never reappears in the index.

Fortunately, there are two archival sources where you may be able to find your document. Google keeps a copy of all documents in its index in "cache." Even if the direct link to the document is broken, clicking the link to the cached version will show you a copy of the page from the last time Google indexed it.

Alexa, maker of the popular toolbar that shows links to related pages, maintains an archive of over a million Web pages. Using Alexa, if your browser displays a "not found" message, you can often click on the "archive" link in the Alexa window to see a copy of the page.

Google's cached documents and Alexa's archived documents may be out of date, but in most cases, something is better than nothing.

Sources:

http://www.microsoft.com/insider/internet/articles/

http://websearch.about.com/internet/websearch/library/weekly/topicsub_searching.htm

http://websearch.about.com/internet/websearch/library/weekly/aa051299.htm

http://hoohoo.ncsa.uiuc.edu/ftp/faq.html

http://websearch.about.com/internet/websearch/library/weekly/bl_981204a.htm

http://pigseye.kennesaw.edu/~msanche1/