What is a Google Crawl?
For starters, let’s define what a Google crawl is and why that is so important to the business owner. To gather information hosted all over the world wide web for organized search results a search engine like Google must deploy software often referred to as a spider (or a crawler or a bot). By using these spiders to find and review web pages, search engines are able to crawl, index and serve information.
“Crawling” = Google crawls to discover your website pages
“Indexing” = Google learns what your pages are about, including at minimum your page title
“Serving” = Google responds to a Google.com search with results including your website page, ranking position determined by many other signals
Spiders consume pages and keywords into indexes where it is all compared and ranked for the most accurate SERP (i.e. search engine result page) possible. Notice that we did not mention websites so much as we have web “pages” here. Google wants to index & compare all web pages. Google applies rank to web pages, NOT just to a website or a website’s home page and that’s important for a business to understand. Also notice we mentioned “indexes” and not index. It’s because there are different indexes for different contexts. For instance, the location of a local service search will bring an index of results relevant to that geography.
It’s why we MUST prioritize pages, posts and our content creation, including an emphasis for the keywords a prospective lead would search. For the business wishing to be visible when people are actually looking for its very products or services, the effort has to begin with content. Content that a search engine can easily identify, consume and organize into its SERPs!
Rebuilding The Google Index – The Business Opportunity!
According to Google, their spiders crawl the web regularly in order to rebuild their index of pages and related keywords. NOTE: Google says it constantly rebuilds its’ index, meaning that it is always looking for the most timely information it can find! This is important because many business owners think they can set up a website and be done with it. Wrong!
Search engines demand much more of the content they recommend in search engine result pages (SERPs). And because search engines are constantly rebuilding, the business owner must share the most current information and knowledge in order to remain relevant in these SERPs.
- Is mobile- friendliness important? Yes!
- Do inbound links matter? Some, when they are considered by Google to be relevant to the subject … and also trusted?!
- Does social activity matter? Some, particularly when it drives traffic out from the network to a website page!
- Is Google looking at how often a page is visited and how long users stay? Yes, reportedly more and more because it is incrementally better evidence for how valuable the page is to visitors.
- Does Google lose interest in pages? To underestimate the enormous amount of competition is foolish, so we must consider what signals confirm that “this website page remains worthy of a high ranking position for its keywords”!
In a competitive world for traffic and leads, search engine relevance is NO static exercise. Freshness matters and being timely with relevant content is certainly a competitive signal. Blogging about “what’s new”, for instance, can motivate a search engine to crawl one website more often than another that never adds new content. The more often a website is crawled, the more pages, posts, and keywords it will see consumed into the search engine indexes!
Deep Spider Crawls and Fresh Spider Crawls
The search indexes are in constant motion (in fact, they are kinetic) and Google crawls the web at varying depths and on several different schedules. Some are believed to be deep crawls and other fresh crawls. It is believed that there are comprehensive crawls (the deep crawls), which occur on a per month basis, and then there are intermediate or random crawls (fresh crawls), which occur more often, but don’t go as deep or index as much.
Some folks refer to the constant motion of the indexes as “the Google Dance”, but the fact is new information is ALWAYS being indexed. Website owners can raise the chance of being crawled deeper and more often, by adding new content frequently. In other words, blog and update or add new useful content!
Google’s spider is capable of comparing previous per page crawls to the current visit at light speed. If it identifies new content often it will schedule more frequent visits. Think of it like a web presence is being judged on each visit. Why? Because it’s highly inefficient and expensive for Google to crawl a website and its pages over and over when nothing changes. NOTE: there is only so much crawl bandwidth and search engines only succeed when they satisfy their search traffic. In order to satisfy search, a business must provide the best user experience possible, which includes the most timely and accurate information possible. ALSO NOTE: due to the “Google Dance” and the constant re-evaluation of website pages, there is never a guarantee a page will remain indexed in the same position from one day to the next… it means, in terms of rank or position, that pages will move up and down!
Motivate Google To Help Your Business: Update Your Pages & Blog!
In all, website pages will be re-evaluated over and over. Like anything else in life worth having, rank and position are earned. So a website needs to be current, consistent and on- topic! And while certain things can help (i.e. page submissions into Google Search Console), focus the most on great new content that people will appreciate, use, link to and share! WHY?: Because Google is looking for the best new content. Help Google help you: write often and write well!