Search engines may work in mysterious ways, but what they do is just "jumping" from one place to another. Search engines use crawlers (bots or spiders) to visit websites, and when they succeed in seeking what they want, they do what they do best.
The first thing a crawler do when they arrive at a website is collecting information from the robots.txt file to see any directives it should follow. If the crawler assumes that it's allowed to crawl all the pages within the site, it proceeds to collect information from all of the pages in can access, and feeds them back to the search engine's database.
Crawlers have their own schedules. The system is not just to prevent them to crawl websites over and over again wasting resources from both sides, but also to prevent any duplicates and "confusions". The crawlers shuffle the pages by priority to index them later.
Crawlers Doing What They Do Best: Crawl
When search engines' crawlers collect information about a given page, they also collect a list of links the pages have to offer. If the links are internal links, the crawlers can follow them to other pages, using their scheduling system. If they're external links, the crawlers put the links in their database for later crawling.
While it’s there, it collects a list of all the pages each page links to. If they’re internal links, the crawler will probably follow them to other pages. If they’re external, they get put into a database for later.
Processing The Links
When the schedule comes for crawlers to crawl, the search engines pull all those links out of the database and connects them. Each link is cast as a vote.
When the search engines find the patterns, they assign relative values those links.
The value of each links is processed when the search engine index the page. If a page is linking to other pages, it can pass some link value to those pages. The value (link juice) is calculated by search engines' algorithm to rank them accordingly.
When the search engine sees a page worth more than others, they have a higher chance to be related to their specific keywords on the search engine's search results.
Working By Indexing
When the crawlers are done getting all the necessary information and links, the indexing process follows. When a search engine indexes, it identifies words and elements of a given page, and match those words and elements with what the search engine has in its database.
When the words are understood, they become keywords. What keywords mean to search engines is that they are the web page's internal structure that represents what it meant. Keywords are also used identify grammatical sense for structural composition, reasoning, and comprehension.
Then the search engine uses its algorithm to determine which pages in the index have those words assigned to them, evaluates links pointing to the page and the domain, and processes other known and unknown metrics to arrive at one single value.
Other algorithms, such as Google's Panda and Penguin for example, are also taken into account. If the website has poor performances on other algorithms, the value assigned will become negative, degrading its visibility. On the other hand, if the website has not stumbled into problems with other algorithms, the value will put the site higher above others. (See: Google Indexing Websites)
Another algorithm aims for geo-targeting. The search engine seeks information about the aspect of the domain (TLD and registrant's background location) and its hosting location.
One of the newer algorithm is a filter that differentiate websites by their ability in rendering mobile-friendly interface. With the algorithm, the search engine can add another type of value to specific pages of a website, and make them rank higher or lower, when the keywords aimed to them are accessed via mobile devices.
With all the criteria comes into a conclusion, search engines then use their ranking algorithm. Google uses a trademarked algorithm called PageRank, developed by its founders: Larry Page and Sergey Brin. This algorithm assigns each web page a relevancy score. The final value is what determines where in the results the page will appear.
Putting Them All Into One Big Experience
As the search engine matures, more features are added. Google, the most notable search engine has added many several new services to enhance search experience.
Some of the service are designed to help make web searches more effective and efficient, while others that have lesser roles, can be used to gather even more data about its users. Like for example Google can use its Google+, Apps, Android and its mobile apps, YouTube, and others, as well as the extensive use of cookies and JavaScript on websites that use its service, in order to get better insight about the user's habit and interests.
Search engine that have extensions to their normal search engine protocols, can specialize searches so users can narrow results to specific resources. For example, search engines can show images that are related to the given keywords, searching for places in a map, getting news footage and articles, seek for products for sale, browsing to blog entries, viewing contents in books, searching for videos and scholarly papers.
Keeping Up With The Trends And Market
The market is always changing. As new things are discovered and invented every time, search engines need to cope in the ever-changing demand.
Search engines are always on the move in their relentless effort in updating and enhancing their algorithms. All the work is in order to give the best experience for its users, anywhere and anytime. That in turn will create revenue.
Looking at history, this is what made Google into a giant it is today.
Further reading: Knowing How Search Engines Operate To Know Their Major Functions