What is the ‘Google Crawl’?
Spiders, Crawling and Web Pages
For starters, it may help to define what a ‘Google crawl’ is and what Google spiders do. To find and organize information on the World Wide Web, a search engine deploys software often referred to as a spider (or a crawler or a bot). By using these spiders to crawl web pages search engines are able to identify information and keywords. They then consume it all into organized indexes where it is compared and ranked for the most accurate search engine result pages (SERPs) possible. NOTE: we did not mention websites so much as we have ‘web pages’ here. Google applies rank to web pages, NOT just to a website or a website’s home page and that’s important to understand because Google wants to index & compare all web pages. It’s why we MUST prioritize pages, posts and useful content creation, including with an emphasis for the keywords a prospective buyer would search!
Rebuilding The Google Index – The Business Opportunity!
According to Google their spiders crawl the web regularly in order to rebuild their index of web pages and keywords. NOTE: Google says it wants to constantly rebuild their index, meaning that it is always looking for the most timely information and knowledge it can find! This is important because many business owners think they can set up a website and be done with it. Wrong! Search engines demand much more of the content they recommend in search engine result pages (SERPs)! And because search engines are constantly rebuilding, then you want to share the most current information and knowledge in order to remain relevant in their SERPs (search engine result pages). Is mobile- friendliness important? Yes! Do inbound links matter? Some! Does social activity matter? To the degree it drives traffic to a website page! Is Google looking at how often a page is visited and how long those users stay on the page? Yes, and reportedly more and more because it is incrementally better evidence for how valuable the page’s content is to its’ visitor!
In a competitive world for traffic and leads, search engine relevance is NO static exercise! Freshness matters and being timely with relevant content is certainly a competitive signal. Blogging ‘what’s new‘, for instance, often motivates a search engine to crawl the website more often than, for instance, a site that never adds any new content. NOTE: the more often a website is crawled, the more pages, posts and keywords it will see consumed into the search engine indexes.
Deep Spider Crawls and Fresh Spider Crawls
The search indexes are in constant motion (in fact, they are kinetic) and Google crawls the web at varying depths and on several different schedules. Some are believed to be ‘deep crawls’ and others ‘fresh crawls’. It is believed that there are comprehensive crawls (= the deep crawls), which occur on a per month basis. And then there are intermediate or random crawls (= fresh crawls), which occur more often, but don’t go as deep or index as much. Some folks refer to the constant motion of the indexes as ‘the Google Dance’, but the fact is new information is ALWAYS being indexed. Website owners can raise the chance of being crawled deeper and more often, by adding new content frequently. In other words, blog and update or add new useful content!
Google’s spider is capable of comparing previous per page crawls to the current visit at light speed. If it identifies new content often it will schedule more frequent visits. Think of it like a web presence is being judged on each visit. Why? Because it’s highly inefficient and expensive for Google to crawl a site over and over when nothing changes. NOTE: there is only so much crawl bandwidth and search engines only succeed when they satisfy their search traffic. In order to satisfy searchers they must provide the best user experience possible, which includes the most timely and accurate information possible. NOTE: due to the ‘Google Dance’ and the constant re -evaluation of website pages, there is never a guarantee a page will remain indexed in the same position from one day to the next… and that means your pages will move up or down!
Motivate Google To Help Your Business: Update Your Pages & Blog!
In all, a website and its’ pages will be re- evaluated over and over. Like anything else in life worth having, rank and position is earned. So a website needs to be current, consistent and on- topic! And while certain things can help (i.e. page submissions into Google Search Console), focus the most on great new content that people will appreciate and share! Why? Because Google is looking for the best new content. SO help Google help you, write often and well 🙂