Anything you could possibly want to know is out there somewhere -- the trouble is finding it when you need it. Fortunately, there are a pile of tools designed to help you find what you're looking for quickly and efficiently

Unfortunately, the problem of how to best index and catalog the contents of the Internet remains one of the biggest problems facing both librarians and technologists. Although all search tools use different methods to find and organize information, we've combined them all in the "search engine" category for the sake of simplicity.

Search Engines
Search engines like AltaVista are Web sites connected to databases that contain indexed information about what is available online. You feed the search engine a word or subject, and it returns a list of Web sites related to your topic. Unfortunately, search engines work like a card catalog at the library --- you can search for different subjects and see what turns up, but you may need to describe your topic a few different ways before you find exactly what you're looking for.

Spiders, crawlers, and bots, oh my!
When you ask a search engine to find sites related to a particular topic, it doesn't go out and sort through all the Web pages on the Internet --- that would take forever. Instead, the index uses automated tools called spiders or robots to update its database in the background. Spiders crawl around the Net to collect information about what's available online, and add this information to the database. Because the spiders work automatically, they can hit thousands of sites a day. (Some spiders can crawl the entire Web in as little as a week!) As a result of the spiders' constant efforts, your search results appear in mere seconds.

Unfortunately, most of this technology is still "dumb" -- spiders index sites according to computerized criteria. The search engine counts the number of times your keyword appears in a document to determine whether or not the page is relevant to your search. Because search engines cannot evaluate context, oddball sites inevitably pop up on your searches. Tech heads refer to these bad search results as "noise."

Web directories vs. information indexes
Some of the tools that we refer to as search engines, like Yahoo!, are actually directories of Web sites. Unlike crawlers like HotBot, directories like Yahoo! are categorized by a team of human editors. When you use a directory, you drill down through a hotlinked, hierarchical list of subject categories (for example, from U.S. Government-States, to States-California, to California-Cities, and so on) to hone in on your target subject.

While you may be able to find highly relevant sites using a directory, directories do not include the quantity of sites you would find in a spidered index --- the crawlers can always hit more sites than could a roomful of humans. If you want to be sure you conducted an exhaustive search, it's important to run your query through both types of search tool. Yahoo!'s directory always lists search results from Alta Vista's search engine as well, offering you the best of both search worlds all in one place.

Meta search sites
If you find yourself searching often, you might want to consider using a "meta search" tool. Meta search sites and utilities run your query through more than one search engine or directory at the same time, so you only enter the query once and receive a single list with the combined search results. The better meta search sites organize the hits according to relevance and eliminate duplicates. There are even search utilities that allow you to run searches of multiple search sites directly from your desktop. (To find out more about these, check out our search search tools.) While meta search tools might bring back more hits than you could follow in a lifetime of Web surfing, they could also save serious time when you're in a crunch.

Finding people online
Of course, there's more to the Internet than Web sites. What if you want to find a person on the Internet or a business that performs a certain service? Just as you would consult telephone white pages for people's phone numbers or consult the yellow pages for a directory of businesses arranged according to categories, there are Internet directories of e-mail addresses that include people, businesses, and other listings.

Searching Usenet
If you want to search through Usenet newsgroups, Deja News and similar utilities provide search engine-style results. In fact, it's often amazing how much easier it is to find an answer to some questions by scouring these discussion strings rather than the Web, and the newsgroups are a great way to find out what real people think about a given topic. AltaVista, HotBot, and other search engines also offer usenet searching, though none are as thorough as Deja News.

Meta tags
Even with the best tools, searching the Internet can be a daunting task. One development that promises to improve the quality of searching is the practice of embedding keywords into a Web page, using "meta tags." The tags are invisible to the user, but search engines use them to classify a page more accurately. Automated indexes would be much more efficient if every Web site included meta tags. Until everyone starts implementing the meta tag standard though, search results will continue to be a little bit hit or miss.

Now, go to the SearchUser features page to learn more about searching, or check out our reviews, search tools, or news department, for indispensable search-related information.