Experiment: Googlebot Tracker 4200
Sub-pages: Googlebot Directly Accessing The IP Googlebot Accessing Broken URLs Googlebot Activity on Other Websites Logged Googlebot IPs & Hostnames
Below you will see all of the Googlebot (Google web crawler) visits to my entire website (Waraas.Com). I built this tool out of boredom, and because I want to see the crawler data.
G o o g l e b o t
A Googlebot is the web crawling bot (sometimes also called a "spider") used by Google. It's a crucial part of Google's indexing algorithm, helping to discover new and updated URL's to be added to the Google search engine.
This tool tracks ALL of the website visits, including the bad URLs (404s), sitemaps, feeds, images, etc. The data will be open and free to the public.
Overtime, I will compile the data and share the results so we can all better understand how the Googlebot's work.
Refresh (F5) this page to see any new data.
Updates:
9/28/25 - I updated the Googlebot Tracker 4200 code to get the "redirect URL", "post type", and "content type" data from the Googlebots now. I also updated the code so it handles NULL better.
9/25/25 - I updated the /robots.txt file to not crawl the Googlebot tracker pages (googlebot-tracker?page= AND googlebot-tracker/?page=) on this website. I also fixed a error that showed broken URL's.
6/12/24 - I added two things; a pagination, and a simple function that shows the Googlebot sessions count for each URL on this website.
5/6/24 - I added a function to track if the Googlebot was using SSL.
4/16/24 - Because of some fake Googlebot traffic, I have updated the Googlebot Tracker to now log the hostname, and if the bot is on a public or private IP address. Because of those two changes, I have updated the Googlebot Tracker version from 4000 to 4200. The tracker is now called The Googlebot Tracker 4200 :)
4/10/24 - The tracker now logs the Googlebots "Session ID" and the page load time of each Googlebot visit.
3/28/24 - The first version of the Googlebot tracker is launched.
Googlebot Tracker Stats
Googlebot Sessions Per URL:
This report displays the number of times Googlebot has visited each URL on Waraas.com. It provides insights into how frequently the site's content is being accessed by Google's web crawler.
| Title | Slug | Visit Count |
|---|---|---|
| Homepage | / | 737 |
| Robots.txt | /robots.txt | 8714 |
| Sitemap.xml | /sitemap.xml | 98 |
| Experiment: Googlebot Tracker 4200 | /googlebot-tracker/ | 46857 |
| Who Is Jon Waraas? | /who-is-jon-waraas/ | 134 |
| Googlebot Accessing Broken URLs on Waraas.com | /googlebot-broken-url-access/ | 133 |
| Privacy Policy | /privacy-policy/ | 123 |
| Jon's Projects | /projects/ | 116 |
| Jon's Experiments | /experiments/ | 107 |
| Contact Jon Waraas | /contact-jon/ | 93 |
| Terms Of Service | /terms/ | 93 |
| Is Google Search Console Data Accurate? | /blog/is-google-search-console-data-accurate/ | 92 |
| An Update on Our Cabin Outhouse Project | /blog/an-update-on-our-cabin-outhouse-project/ | 87 |
| I'm Launching My First Free WordPress Plugin! | /blog/launching-first-free-wordpress-plugin/ | 84 |
| Our Plans For Camp Bluebird This Month | /blog/our-plans-for-camp-bluebird-this-month/ | 80 |
| Launching AntlerSheds.Com! | /blog/launching-antler-sheds-com/ | 77 |
| From Curiosity to Ownership: Our Adventure In Gilmore, Idaho | /blog/from-curiosity-to-ownership-my-adventure-in-gilmore-idaho/ | 76 |
| Did A Googlebot Really Crawl My Website's Dedicated IP Address? | /blog/did-a-googlebot-really-crawl-my-websites-dedicated-ip-address/ | 68 |
| How Much Does It Cost to Build an Outhouse in Idaho? | /blog/how-much-does-it-cost-to-build-an-outhouse-in-idaho/ | 66 |
| How Much Did a 12x20 Amish-Built Shed Cost? | /blog/how-much-did-a-12x20-amish-built-shed-cost/ | 64 |
| The Art of Bot War: What Are Fake Googlebots? | /blog/the-art-of-bot-war-what-are-fake-googlebots/ | 62 |
| The DOJ Wants Google To Sell Chrome - What A SEO'er Thinks | /blog/doj-wants-google-to-sell-chrome-what-seo-thinks/ | 45 |
| Googlebot Accessing Waraas.com via IP Address | /direct-ip-access-by-googlebot/ | 41 |
| How Developers Can Manually Add Yotpo To WordPress Theme Files | /blog/how-to-add-yotpo-to-wordpress-theme-files/ | 39 |
| Googlebot Testing: 404 vs 410? | /blog/googlebot-testing-404-vs-410/ | 38 |
| The Googlebot Tracker 4000 Is Live! | /blog/googlebot-tracker-4000-is-live/ | 33 |
| Why Googlebot May Request /robots.txt via IP | /blog/why-googlebot-may-request-robots-txt-via-ip/ | 21 |
| Relaunching UnblockWebsites.com! | /blog/relaunching-unblockwebsites-com/ | 19 |
| Googlebot Activity on Other Websites | /googlebot-activity-other-websites/ | 15 |
| Hypothesis: How Googlebot Tracker 4200 Got External Crawl Data | /blog/googlebot-tracker-external-crawl-data/ | 13 |
| Googlebot IPs & Hostnames Logged by Googlebot Tracker 4200 | /googlebot-ips-hostnames-logged-by-googlebot-tracker-4200/ | 6 |
View All Googlebot Sessions:
The list below displays each individual Googlebot visit to this website.
Googlebot Related Blog Posts:
This category encompasses all of my laboratory work related to my experiments on the Google web crawler, known as "Googlebot".
Ever since building my first website in 2002, I've been hooked on web development. I now manage my own network of eCommerce/content websites full-time. I'm also building a cabin inside a old ghost town. This is my personal blog, where I discuss web development, SEO, cabin building, and other personal musings.
