Search engines are machines who gave answers to our queries. They exist to discover, understand, and organize the internet’s content in order to offer the most relevant results to the questions searchers are asking. In order to show up in search results, your content needs to first be visible to search engines, If your site can’t be found, there’s no way you’ll ever show up in the Search Engine Results Page.
How do search engines work?
Basically, search engines have three primary functions:
1. Crawl: Scour the Internet for content, looking over the code/content for each URL they find.
2. Index: Store and organize the content found during the crawling process. Once a page is in the index, it’s in the running to be displayed as a result of relevant queries.
3. Rank: Provide the pieces of content that will best answer a searcher’s query. Order the search results by the most helpful to a particular query.
What is search engine crawling?
Crawling is the discovery process in which search engines send out a team of robots (known as crawlers or spiders) to find new and updated content. Content can vary it could be a webpage, an image, a video, a PDF, etc. but regardless of the format, content is discovered by links. The bot starts out by fetching a few web pages and then follows the links on those webpages to find new URLs. By hopping along this path of links, crawlers are able to find new content and add it to their index a massive database of discovered URLs to later be retrieved when a searcher is seeking information that the content on that URL is a good match for.
What is a search engine index?
Search engines process and store information they find in an index, a huge database of all the content they’ve discovered and deemed good enough to serve up to searchers.
Search Engine ranking
When someone performs a search, search engines scour their index for highly relevant content and then orders that content in the hopes of solving the searcher’s query. This ordering of search results by relevance is known as ranking. In general, you can assume that the higher a website is ranked, the more relevant the search engine believes that the site is to the query. It’s possible to block search engine crawlers from part or all of your site or instruct search engines to avoid storing certain pages in their index. While there can be reasons for doing this, if you want your content found by searchers, you have to first make sure it’s accessible to crawlers and is indexable. Otherwise, it’s as good as invisible. By the end of this chapter, you’ll have the context you need to work with the search engine, rather than against it.