Understanding Googles crawl budget - what on earth is it?

By : Administrator
Published 20th January 2017 |
Read latest comment - 30th January 2017

So if you are wondering what on earth is a Google crawl budget, fear not.

Every website wants to show up in Googles search results and to enable this, Google has something called a Google Bot. This is just a bit of code that zips round the web hoovering up web pages to display in Googles search results.

Sometimes it's called a spider or web robot. 

How often and how much time the Google bot spends hoovering up your web pages is called your "crawl budget" as the Google Bot crawls all over your website.

You can see how much Google is crawling your site or even amend your crawl rate by logging into your Google Console. If you have never used your Google Console, it's time to start. It's free and gives lots of valuable information about the health of your website and what Google thinks about it. It's fairly straight forward to set up, just follow the prompts, but let me know if you get stuck.

Once logged into your console, you will see a menu on the left hand side. If you scroll down to the Crawl folder and expand it, you will be able to see any crawl errors for your website.

This gives you a chance to see any problems Google has identified with your website. The next tab down is Crawl Stats.

Here you can see how many pages the Google Bot is indexing from your website. For us, you can see where we launched our new site, which has a much better structure than the old one, with the Google Bot hoovering up around 250,000 pages a day.

Now your site is likely to be much smaller, but as you update content then you are likely to see pages getting regularly updated. For pages which remain static then it may be 3 or 4 months before the Google Bot revisits. 

When it does land on your site is dictated by your Crawl Budget. There are a myriad of factors that cause this, from how often you update your website, the speed of your website, any problems with your site, popularity and a whole host more.

For a definitive study about Googles crawling habits, and how to influence it, there is a great post by Jennifer Slegg - Google Crawl Budget.

There are also some great tools out there that can help analyse your site and resolve issues such as Deep Crawl and Screaming Frog

Back in your Google Console you will be able to see how many pages Google has hoovered up and put in it's index, ie will be displayed in the search results.

Ours again shows the results of our recent upgrade and the number of pages being indexed. But this is a valuable reference point because you will want to make sure the majority of your pages are being displayed in Googles index. If they're not, then this will point to problems with your site, or individual pages.

Tackling and understanding things like crawl budgets and indexing can seem daunting, but if you are responsible for marketing your website, then take the time to become familiar. Spotting problems in your search console can save you all sorts of headaches later on.

If you have any questions, problems or want to know more, just comment below.


Steve Richardson
Gaffer of My Local Services
My Local Services | Me on LinkedIn
Comments

Great article .. I use this every day and remove errors as soon as I find them .. I have noticed google seems to have fixed their site in the way of removing pages as I used remove the same pages over and over and over again, now days they seems to be different pages all the time with only the odd repeat 


Thanks
Andy-C | Pewter World

Actually I have a question

Way back when I first started using google I was advised to use both the www. and non www. ..So as I read this thread I was playing around with it again and went into the non www. site which has  errors in the sitemap as there is a redirect to www. Now if I go into the www. it says everything is fine and only have 24 missing from 1387 submitted ..Is the sitemap for the non www. not needed as everything redirects to www.


Thanks
Andy-C | Pewter World

Is the sitemap for the non www. not needed as everything redirects to www.”
 

Nope, you only need the sitemap for the URL you need, ie the www one.

It has been a historical pain as Google has always said you need to verify each version if your site, which for us was m.mylocal... mylocal... www.mylocal and now https://www.mylocal...

But Google is getting smarter, and although they still recommend you verify each version, you only need to worry about the correct version of your site, assuming you have your redirects in place. eg http:// to https:// or non www to www.


Steve Richardson
Gaffer of My Local Services
My Local Services | Me on LinkedIn

If my google account is set up  for the www's do I need 1 for https , as only my important stuff is under this like sign up, checkout and account


Thanks
Andy-C | Pewter World

Sorry mate, hectic week last week, got side tracked. 

If my google account is set up  for the www's do I need 1 for https , as only my important stuff is under this like sign up, checkout and account”
 

We had a similar issue, where we were only part https, so in our case we also set up an account for https as well as www, even though 99% of pages were only http.

But it left me thinking if we are already using https for some pages, we might as well move the whole site to https which is what we did. But we let the http and https pages run side by side for a few months, with a canonical tag (tells google which is your preferred page) pointing to https.

Then after a few months, once Google had pretty much re-indexed everything, we just redirected http to https. 

Google console still shows the www version, which is useful to track progress and any probs.


Steve Richardson
Gaffer of My Local Services
My Local Services | Me on LinkedIn