Did A Googlebot Really Crawl My Website's Dedicated IP Address?

By Jon Waraas - First Published: May 4th, 2024

Why did Google do that??

I have learned a lot since launching the Googlebot Tracker 4200 tool a couple of months ago.

Today, I want to share one of my lab results with you.

A little backstory: I developed the Googlebot Tracker 4200 to track all of the Googlebot traffic on my blog here (Waraas.Com). My goal it to research and study the cute little Googlebots, so I can better understand how Google search works.

One thing I have found..

Googlebots will crawl your website's dedicated IP address.

As you can see from the screenshot below, a Googlebot crawled my /robots.txt page via the websites dedicated IP address.

Photo showing a verified Googlebot accessing this website via the dedicated IP address.

^ Above: You can see that a verified Googlebot checked the /robots.txt URL via the dedicated IP address.

Note the load time for that bot, it's fast AF! The reason is because the IP version of the website doesn't have to run the DNS, which saves a lot of load time. Thus faster load times.

What's a Dedicated IP Address?

So most websites are hosted on something called "shared hosting", which hosts lots of different websites all on one IP address.

There are only a limited number of IP addresses on the World Wide Web, so in order to save time and money, most hosting providers use shared IP's.

However, you can upgrade your hosting to get a "dedicated IP address", like I did for my blog here (Waraas.Com).

The advantage of using a dedicated IP address is the speed. There is only 1 website on the IP address, which means less resources are being used, thus faster loading speeds.

Dedicated IP's also add on a trustworthiness factor, since they are rare, and are mostly used by popular/trustworthy websites. This factor is why some believe that dedicated IP's might help their website's rank higher on Google.

So it's interesting to see that Google is actually crawling this website via the dedicated IP address.

What Does This Mean??

Whether or not you believe that dedicated IP addresses are a ranking factor on Google.. we can all agree that Google checks to see if your website has a dedicated IP address.

Google certainly checked to see if this blog (Waraas.Com) has a dedicated IP address.

The problem is that the IP version of this blog will spew out SSL errors, since the SSL isn't set up to run on the dedicated IP address version. So I need to do some testing, and see how much Google uses the websites dedicated IP address.

If the Googlebots are getting errors (such as SSL errors), then Google might rank my site lower in the search engine results. I need to test this aspect more.

Testing & Updates

Is this a fluke? Did I make a mistake coding the Googlebot Tracker 4200?

Just to be safe, I did make a few updates to the tool.

1st: I added a function that will check to see if the crawled URL is secure. This is important to check, since my websites SSL (https) doesn't work correctly when you view the website via the dedicated IP.

I want to see if the Googlebot is getting the secured version of the website when it's browsing via the dedicated IP.

Photo showing php functions that will check if URL is secured or not.

^ Above: The function above will check if the URL is secured or not.

2nd: I added the "requestedResource" aspect into its own function. This aspect grabs the URL that is being used, no matter if its the domain or dedicated IP version.

I made these updates so I can reuse this feature in other places, and to clean up the code in general.

Photo showing a function that will grab the whole version of the URL.

^ Above: The function above will grab the whole version of the URL that is being crawled by the Googlebot. It will show if the dedicated IP or domain name version of the website was used.

You can see my current Googlebot Tracker 4200 code below:

Photo showing the current code for the Googlebot Tracker 4200.

^ Above: You can see the current code for the Googlebot Tracker 4200.

With the small updates above, I should be able to track the Googlebots better, so I can study them even more.

Please leave a comment below if you have any suggestions at all for the Googlebot tracker tool, or have any questions :)

Conversation:

No comments yet. Please contribute to the conversation and leave a comment below.

Leave A Comment:

Ever since building my first website in 2002, I've been hooked on web development. I now manage my own network of eCommerce/content websites full-time. I'm also building a cabin inside a old ghost town. This is my personal blog, where I discuss web development, SEO, eCommerce, cabin building, and other personal musings.