The World Wide Web has gained popularity over the past two years because of better software, more Internet connection companies and an explosion of ideas for using the global data network.
Someone had to figure out how to sort through it all.A few people did, and the companies they formed - with names like Lycos, Open Text, Web Crawler and Yahoo - have become some of the most important related to the Internet.
All are privately owned but some are drawing interest from big brokers and may soon become investment opportunities.
"The upside in these companies comes from the predicted growth of the Internet," said Brian Pinkerton, who formed Web Crawler, which was recently purchased by America Online Inc. "Success depends not on how well they do, but on how well the Internet grows."
Like others involved with emerging technology, the Web indexing firms have developed new jargon, words like "spiders" and "search engines," to describe their work.
People often log on to the Web site of an index company, fill out a form, click on a "button" and wait a few minutes for a response.
Many people go to an index when they first log on to the Web, giving the indexing companies a presence among users that is attractive to advertisers.
That role of pulling information together steals some thunder from traditional on-line services like America Online, CompuServe and Prodigy, who increasingly market themselves as a leaping off point to the Internet.
The Web search companies are even mimicking the traditional on-line firms by adding desirable features like news wires and weather info. Some review Web sites. Architext Software's Excite service has even added political cartoons.
Both Architext and Open Text Corp. started as developers of software for scanning the Web. They later created their own search services.
"The value is not in the technology. It's in the products and services you can build on the technology," said Joe Kraus, president of Architext.
Most index companies allow free searches. For revenue, they rely on advertising from many companies and licensing fees from the firms that provide Internet connections.
An emerging opportunity is for their software to applied to internal data systems. Some corporations are changing their computers to take advantage of the linking concept that is a key to the design of the Web.
"If we can crack the 20 million-user, 10 million-page problem (of the Internet), then working in a smaller corporate data system is attractive," said Tim Bray, co-founder and senior technologist at Open Text.
The Web index companies are quite competitive: Executives eagerly point out differences in technical capabilities.
For instance, Lycos Inc. claims to have surveyed the largest number of Web pages, over 13 million. "Our catalog is so much larger than anything in existence," said Bob Davis, CEO of Lycos.
And the Magellan search service by the McKinley Group Inc., boasts a "dividing and parsing algorithm" that make abstracts of paragraphs within Web sites.
"We use that tool to provide the user with a bigger data base to go after," said David Hayden, chief executive officer of McKinley.
Dave Taylor, a Menlo Park, Calif., consultant and author of several Internet-related books, said the Web search companies are all trying to differentiate themselves.
"The question is do you differentiate by having editorial control or by the quantity of contents, the we'll-list-everybody approach," Taylor said.
Those that exercise some control over who they will list face decisions similar to those of traditional publishers.
For instance, firms that provide ratings must decide how often to review Web sites. Someone who gets a bad rating is often eager to improve, similar to a restaurant adjusting to a critical newspaper review.
Yahoo, a search service that requires Web site producers to fill out an electronic application to be included in the index, has encountered some difficult choices trying to decide how some sites ought to be categorized, co-founder Jerry Yang said.
For instance, a Web site for Messianic Jews, a group that observes Jewish traditions but also holds strong beliefs about Jesus Christ, was listed in both Christian and Jewish categories by Yahoo. Some Jewish groups criticized Yahoo for that decision, calling for boycotts of the service.
"We try to be as inclusive as possible," Yang said. But he added Yahoo is sometimes forced "to create divisions among things that are not obvious."
Another issue for companies that rate Web sites is whether to place more value on the depth of information or, because the Web allows sophisticated visual images, the way it is displayed.
"This pops up in a lot of different areas," consultant Taylor said. "Should it look sexy or should it be fabulous content?"
McKinley directs the 40 employees who review Web sites for Magellan to base their judgments on whether the site accomplishes what it sets out to do. That is determined by looking at the home page, the cyber equivalent of a front door to information, for a description of what's available.
The Web addresses for companies mentioned in this article:
Architext's Excite is http://www.excite.com
Lycos is http://www.lycos.com
McKinley's Magellan is http://www.magellan.com
Open Text is http://www.opentext.com
Web Crawler is http://www.webcrawler.com
Yahoo is http://www.yahoo.com