Experiment: Googlebot Tracker 4200
Below you will see all of the Googlebot (Google web crawler) visits to my entire website (Waraas.Com). I built this tool out of boredom, and because I want to see the crawler data.
G o o g l e b o t
A Googlebot is the web crawling bot (sometimes also called a "spider") used by Google. It's a crucial part of Google's indexing algorithm, helping to discover new and updated URL's to be added to the Google search engine.
This tool tracks ALL of the website visits, including the bad URLs (404s), sitemaps, feeds, images, etc. The data will be open and free to the public.
Overtime, I will compile the data and share the results so we can all better understand how the Googlebot's work.
Updates:
4/10/24 - Because of some fake Googlebot traffic. I have updated the Googlebot Tracker to now log the hostname, and if the bot is on a public or private IP address. Because of those two changes, I have updated the Googlebot Tracker version from 4000 to 4200. The tracker is now called The Googlebot Tracker 4200 :)
4/16/24 - The tracker now logs the Googlebots "Session ID" and also the page load time of each Googlebot session.
Googlebot Sessions On Waraas.Com:
Refresh (F5) this page to see any new data.
Visit Time: May 7th, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-70-3.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: p4792cfo61d2c4r7gs6kap5i55
IP Address: 66.249.70.3
HTTP Status: 200
Load Time: 0.007 seconds
Visit Time: May 6th, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: tb050vu8o5gqhgqahhr8rsvfjs
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: 0.147 seconds
Visit Time: May 4th, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: 2a9elr0p3ortd2akeu0r5130sq
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: May 4th, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-70-4.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: ql7e7fertcu78j53723rh9h3m4
IP Address: 66.249.70.4
HTTP Status: 200
Load Time: NULL
Visit Time: May 4th, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-70-3.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: h4ms5qmcrkjitb0csp91ktk9mf
IP Address: 66.249.70.3
HTTP Status: 200
Load Time: NULL
Visit Time: May 4th, 2024
Page URL: https://waraas.com/experiments
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: 9ucvoqb4jcduahlb5i5o46ivq0
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: May 1st, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-70-3.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: mfnqti7mdnn60mja398l7f09sj
IP Address: 66.249.70.3
HTTP Status: 200
Load Time: 0.021 seconds
Visit Time: May 1st, 2024
Page URL: https://198.54.114.122/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-66-41.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: 5sqndb19hu10qrsiradd3n0ies
IP Address: 66.249.66.41
HTTP Status: 200
Load Time: 0.003 seconds
Visit Time: April 30th, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-72-4.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: h4vc2ch78im472rqcpobgk3ph0
IP Address: 66.249.72.4
HTTP Status: 200
Load Time: 0.005 seconds
Visit Time: April 29th, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-70-3.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: as84oig6fhosrdau62cesk9i78
IP Address: 66.249.70.3
HTTP Status: 200
Load Time: 0.003 seconds
Visit Time: April 29th, 2024
Page URL: https://waraas.com/robots.txt
Referrer URL: No URL
Hostname: crawl-66-249-65-168.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: r7kkl6ro5qvngaju0irdhpadrf
IP Address: 66.249.65.168
HTTP Status: 200
Load Time: 0.003 seconds
Visit Time: April 27th, 2024
Page URL: https://waraas.com/who-is-jon-waraas
Referrer URL: No URL
Hostname: crawl-66-249-70-3.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: 7t0vslroe1ib2hlgubomf7d4qd
IP Address: 66.249.70.3
HTTP Status: 200
Load Time: 0.004 seconds
Visit Time: April 26th, 2024
Page URL: https://waraas.com/category/personal
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: 78003n08n72ajn7dhn3jt7njho
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: 0.050 seconds
Visit Time: April 26th, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: crawl-66-249-70-3.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: lcro5hjf2nkegdm3dnh928bmg4
IP Address: 66.249.70.3
HTTP Status: 200
Load Time: 0.005 seconds
Visit Time: April 23rd, 2024
Referrer URL: No URL
Hostname: crawl-66-249-79-201.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: r4tbkml64c6o4uu7t8osm9ibrl
IP Address: 66.249.79.201
HTTP Status: 200
Load Time: 0.009 seconds
Visit Time: April 22nd, 2024
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Googlebot-Image/1.0
Session ID: 5n6vs5gnv30ghk7h5k65fle5h5
IP Address: 66.249.70.2
HTTP Status: 404
Load Time: 0.006 seconds
Visit Time: April 20th, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: 5pu9bdippld18ed2iom8pu2j6h
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: 0.010 seconds
Visit Time: April 20th, 2024
Page URL: https://waraas.com/
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: laobbh85p9tgbrtapnhumaafn7
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: 0.010 seconds
Visit Time: April 16th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-70-3.googlebot.com
User Agent: Googlebot-Image/1.0
Session ID: ef5cd8at702g19gngc6khoapnt
IP Address: 66.249.70.3
HTTP Status: 404
Load Time: 0.005 seconds
Visit Time: April 15th, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: crawl-66-249-70-3.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: el0gngia1mehg65ogvhh3d6tfb
IP Address: 66.249.70.3
HTTP Status: 200
Load Time: NULL
Visit Time: April 12th, 2024
Page URL: https://waraas.com/terms
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: April 11th, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: crawl-66-249-70-2.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: April 5th, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.4
HTTP Status: 200
Load Time: NULL
Visit Time: April 5th, 2024
Page URL: https://waraas.com/
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.3
HTTP Status: 200
Load Time: NULL
Visit Time: April 2nd, 2024
Page URL: https://waraas.com/privacy-policy/
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.4
HTTP Status: 200
Load Time: NULL
Visit Time: April 2nd, 2024
Page URL: https://waraas.com/terms
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.4
HTTP Status: 200
Load Time: NULL
Visit Time: April 1st, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: March 31st, 2024
Referrer URL: No URL
Hostname: No hostname
User Agent: Googlebot-Image/1.0
Session ID: No Session ID
IP Address: 66.249.70.3
HTTP Status: 404
Load Time: NULL
Visit Time: March 31st, 2024
Page URL: https://waraas.com/
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: March 31st, 2024
Page URL: https://waraas.com/
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: March 31st, 2024
Page URL: https://waraas.com/
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.4
HTTP Status: 200
Load Time: NULL
Visit Time: March 30th, 2024
Page URL: https://waraas.com/
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: March 28th, 2024
Page URL: https://waraas.com/contact-jon
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66.249.70.2
HTTP Status: 200
Load Time: NULL
Visit Time: March 27th, 2024
Page URL: https://waraas.com/privacy-policy
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Visit Time: March 27th, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Visit Time: March 26th, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Visit Time: March 24th, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Visit Time: March 24th, 2024
Referrer URL: No URL
Hostname: No hostname
User Agent: Googlebot-Image/1.0
Session ID: No Session ID
IP Address: 66
HTTP Status: 404
Load Time: NULL
Visit Time: March 23rd, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Visit Time: March 23rd, 2024
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Visit Time: March 23rd, 2024
Page URL: https://waraas.com/terms
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Visit Time: March 23rd, 2024
Page URL: https://waraas.com/favicon.ico
Referrer URL: No URL
Hostname: No hostname
User Agent: Googlebot-Image/1.0
Session ID: No Session ID
IP Address: 66
HTTP Status: 404
Load Time: NULL
Visit Time: March 23rd, 2024
Page URL: https://waraas.com/
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6261.94 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Visit Time: March 22nd, 2024
Page URL: https://waraas.com/sitemap.xml
Referrer URL: No URL
Hostname: No hostname
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: No Session ID
IP Address: 66
HTTP Status: 200
Load Time: NULL
Googlebot Related Blog Posts:
This category encompasses all of my laboratory work related to my experiments on the Google web crawler, known as "Googlebot".
Did A Googlebot Really Crawl My Website's Dedicated IP Address?
How the Googlebot tracker 4200 revealed unexpected insights in my latest tests....
Googlebot Testing: 404 vs 410?
In this article, I explore the impact of HTTP status codes 410 and 404 on Googlebot's crawl budget....
The Art of Bot War: What Are Fake Googlebots?
Here I discuss what Googlebots are, and how to detect the little creatures. ...
The Googlebot Tracker 4000 Is Live!
I built a Googlebot tracker using sessions, PHP, and SQL....
Ever since building my first website in 2002, I've been hooked on web development. I now manage my own network of eCommerce/content websites full-time. I'm also building a cabin inside a old ghost town. This is my personal blog, where I discuss web development, SEO, eCommerce, cabin building, and other personal musings.
Brett : Very cool to get the back story and will be neat to watch the progress. Hoping eventually to so do something the similar on the west coast of Canada somewhere. Amazing that in 2006 I first found your site for it's myspace page information and how to build PHP site header/footers for resale. How times change hah. Anyways, keep up the great posts, looking forward to the updates.
Posted on: April 11, 2024