SEARCHING -WEB-PAGES- SCENARIOS : RESEARCH

 

GOOGLE : AS regulators circle, focus on the company's vast index is really increasing in leaps and bounds.

In 2000, just two years after it was founded, Google reached a milestone that would lay the foundation for its dominance over the next 20 years. It became the world's largest search engine, with an index of more than 1 billion.

The rest of the Internet  never caught up, and Google's index just kept on getting bigger. Today, it's somewhere between 500 billion and 600 billion web pages, according to estimates.

''If people are on a search engine with a smaller index, they're not always going to get the results they want. And then go to Google, and stay at Google,'' said Matt Wells, who started Gigablast, a search engine with an index of around 5 billion web pages, about 20 years ago. ''A little guy like me just can't compete.''

Websites and search engines are symbiotic. Websites rely on search engines for traffic, while search engines need access to crawl the sites to provide relevant results for users.

But each crawler puts a strain on website's resources in server and bandwidth costs, and some aggressive crawlers resemble security risks that can take down a site.

Since having their pages crawled costs money, websites have an incentive to let it be done only by search engines that direct enough traffic to them. In the current world of search, that leaves Google and  -in some cases- Microsoft's Bing.

Google and Microsoft are the only search engines that spend hundreds of millions of dollars annually to maintain a real-map of the English Language Internet.  

That's in addition to the billions they've spent over the years to build out their indexes, according to a report that summer from Britain's Competition and Markets Authority.

Google holds a significant leg up on Microsoft in more than market share. The British competition authorities said Google's index included about 500 billion to 600 billion web pages, compared with 100 billion to 200 billion for Microsoft.

Other large companies deploy crawlers for other purposes. Facebook has a crawler for links that appear on its site or services. Amazon says its crawlers help improve its voice-based assistant, Alexa.

Apple has its own crawler, Appplebot, which has fueled speculation that it might be looking to build its own search engine.

But indexing has always been a challenge for companies without deep pockets.

The privacy-minded search engine DuckDuckGo decided to stop crawling the entire web more than a decade ago and now syndicates results from Microsoft.

It still crawls sites like Wikipedia to provide results for answer boxes that appear in its results, but maintaining its own index does not usually make financial sense for the company.

''It costs more money than we can afford,'' said Gabriel Weinbergm, chief executive of DuckDuckGo. In a written statement for the House antitrust subcommittee last year, the company said ''an aspiring search engine start-up today [and in the foreseeable future] cannot avoid the need'' to turn to Microsoft and  Google for its search results.

The Honor and Serving of the Latest Global Operational Research and Thinking on Search Engines All, continues. The World Students Society thanks author Daisuke Wakabayashi.