Experiment: Googlebot Tracker 4200
Below you will see all of the Googlebot (Google web crawler) visits to my entire website (Waraas.Com). I built this tool out of boredom, and because I want to see the crawler data.
G
o
o
g
l
e
b
o
t
A Googlebot is the web crawling bot (sometimes also called a "spider") used by Google. It's a crucial part of Google's indexing algorithm, helping to discover new and updated URL's to be added to the Google search engine.
This tool tracks ALL of the website visits, including the bad URLs (404s), sitemaps, feeds, images, etc. The data will be open and free to the public.
Overtime, I will compile the data and share the results so we can all better understand how the Googlebot's work.
Refresh (F5) this page to see any new data.
Updates:
6/12/24 - I added two things; a pagination, and a simple function that shows the Googlebot sessions count for each URL on this website.
5/6/24 - I added a function to track if the Googlebot was using SSL.
4/10/24 - Because of some fake Googlebot traffic, I have updated the Googlebot Tracker to now log the hostname, and if the bot is on a public or private IP address. Because of those two changes, I have updated the Googlebot Tracker version from 4000 to 4200. The tracker is now called The Googlebot Tracker 4200 :)
4/16/24 - The tracker now logs the Googlebots "Session ID" and the page load time of each Googlebot visit.
Googlebot Sessions Per URL:
This report displays the number of times Googlebot has visited each URL on Waraas.com. It provides insights into how frequently the site's content is being accessed by Google's web crawler.
View All Googlebot Sessions:
The list below displays each individual Googlebot visit to this website.
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-164.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: gj6j7ibflekeh4fvg3nohhl5hc
IP Address: 66.249.66.164
HTTP Status: 200
Load Time: 0.005 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-165.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: elvo1dusq36rk1if5r5q39cjt4
IP Address: 66.249.66.165
HTTP Status: 200
Load Time: 0.008 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-163.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: p044gvcgc8esa1t9oafopjo1u1
IP Address: 66.249.66.163
HTTP Status: 200
Load Time: 0.003 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-165.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: t4laen6410t0e66qrfun90skrg
IP Address: 66.249.66.165
HTTP Status: 200
Load Time: 0.002 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-163.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: e8rkpedp8voodpsukiqcmbu5ol
IP Address: 66.249.66.163
HTTP Status: 200
Load Time: 0.286 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-163.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: i39ae9e3snckh89q05gjil8kb9
IP Address: 66.249.66.163
HTTP Status: 200
Load Time: 0.344 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-165.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: oro16mk876t26soid2it2q5dlu
IP Address: 66.249.66.165
HTTP Status: 200
Load Time: 0.003 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-163.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: mib50iq9lc4o861pp7cbbgalal
IP Address: 66.249.66.163
HTTP Status: 200
Load Time: 0.007 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-164.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: adq7k1flpmg3p07h3kjr1j1gs6
IP Address: 66.249.66.164
HTTP Status: 200
Load Time: 0.003 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-164.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: 8aqh3pq0m4ron4epbng377o1r5
IP Address: 66.249.66.164
HTTP Status: 200
Load Time: 0.005 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-165.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: gh9ucu8sedf28su8b763oc5dc2
IP Address: 66.249.66.165
HTTP Status: 200
Load Time: 0.005 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-163.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: s0vi5q268nshtfqhum282no4ua
IP Address: 66.249.66.163
HTTP Status: 200
Load Time: 0.285 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-164.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: 1n05pin5v7he9rlcfudfdqtg7b
IP Address: 66.249.66.164
HTTP Status: 200
Load Time: 0.003 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-164.googlebot.com
User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: l6tmgndo7ucg8hbu2c1sul6i54
IP Address: 66.249.66.164
HTTP Status: 200
Load Time: 0.320 seconds
Visit Time: July 27th, 2024
Referrer URL: No URL
Hostname: crawl-66-249-66-165.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Session ID: mbgajkm61ccbms2nmpp5p7c8nt
IP Address: 66.249.66.165
HTTP Status: 200
Load Time: 0.002 seconds
Googlebot Related Blog Posts:
This category encompasses all of my laboratory work related to my experiments on the Google web crawler, known as "Googlebot".
Brett : Very cool to get the back story and will be neat to watch the progress. Hoping eventually to so do something the similar on the west coast of Canada somewhere. Amazing that in 2006 I first found your site for it's myspace page information and how to build PHP site header/footers for resale. How times change hah. Anyways, keep up the great posts, looking forward to the updates.
Posted on: April 11, 2024