So if you are wondering what on earth is a Google crawl budget, fear not.
Every website wants to show up in Googles search results and to enable this, Google has something called a Google Bot. This is just a bit of code that zips round the web hoovering up web pages to display in Googles search results.
Sometimes it's called a spider or web robot.
How often and how much time the Google bot spends hoovering up your web pages is called your "crawl budget" as the Google Bot crawls all over your website.
You can see how much Google is crawling your site or even amend your crawl rate by logging into your Google Console. If you have never used your Google Console, it's time to start. It's free and gives lots of valuable information about the health of your website and what Google thinks about it. It's fairly straight forward to set up, just follow the prompts, but let me know if you get stuck.
Once logged into your console, you will see a menu on the left hand side. If you scroll down to the Crawl folder and expand it, you will be able to see any crawl errors for your website.
This gives you a chance to see any problems Google has identified with your website. The next tab down is Crawl Stats.
Here you can see how many pages the Google Bot is indexing from your website. For us, you can see where we launched our new site, which has a much better structure than the old one, with the Google Bot hoovering up around 250,000 pages a day.
Now your site is likely to be much smaller, but as you update content then you are likely to see pages getting regularly updated. For pages which remain static then it may be 3 or 4 months before the Google Bot revisits.
When it does land on your site is dictated by your Crawl Budget. There are a myriad of factors that cause this, from how often you update your website, the speed of your website, any problems with your site, popularity and a whole host more.
For a definitive study about Googles crawling habits, and how to influence it, there is a great post by Jennifer Slegg - Google Crawl Budget.
There are also some great tools out there that can help analyse your site and resolve issues such as Deep Crawl and Screaming Frog.
Back in your Google Console you will be able to see how many pages Google has hoovered up and put in it's index, ie will be displayed in the search results.
Ours again shows the results of our recent upgrade and the number of pages being indexed. But this is a valuable reference point because you will want to make sure the majority of your pages are being displayed in Googles index. If they're not, then this will point to problems with your site, or individual pages.
Tackling and understanding things like crawl budgets and indexing can seem daunting, but if you are responsible for marketing your website, then take the time to become familiar. Spotting problems in your search console can save you all sorts of headaches later on.
If you have any questions, problems or want to know more, just comment below.
