Social Media, SEO
& Your Business

by Mitch Mitchell

Using Your Website
As A Marketing Tool

by Mitch Mitchell


Follow Me On Twitter;
Click The Bird!

Add me on Google Plus!

Embrace The Lead
by T. T. Mitchell


Free Download; right-click on book

Leadership Is/Isn't Easy
by T. T. Mitchell


Is Google Reading Your Robots.txt File?

Posted by on Mar 16, 2009

Last week, I happened upon a post by our buddy Caleb of The Market Secrets Blog titled How Search Engine Robots Read Your Site. Once I got past the image of the stupid spider (had to be a spider, didn’t it Caleb?), in the article, he has this box attached to something called Search Engine Spider Simulator, where you put in your URL (that’s the link to your blog or website, for the uninitiated), click “submit”, then it goes through and shows you how the search engines go through your site.

I put in the URL for this blog, and I got nothing. I knew that couldn’t be right, because I have the WordPress Google XML Sitemap plugin on the blog. But there is was, not a single thing. I then put in the URL of my two business sites, and all sorts of pages came up.

At that point, I decided to come back to my blog and go into the settings for the plugin. Truthfully, I’d never been in there before, just accepting however it’s default settings gave me. Near the top, under the Basic Options area, there’s a box that was unchecked for me that states “Add sitemap URL to the virtual robots.txt file.” I checked that, saved it, and waited a day. I then went back to Caleb’s article (forgetting about that stupid spider) and put in my blog’s URL once again. This time, all worked perfectly; wow!

It makes me wonder now if, later on, I might actually start attaining some kind of page rank on some of my internal pages, now that I know for sure that the search engines will be going through them more often than just when I first wrote them. I don’t know for sure, but I guess we’ll find out.

If you want to learn more about robots.txt files, you can check out the Web Robots Pages site, but I’ll tell you this truth. I don’t have a robots.txt file for most of my sites. Instead, I use a page called XML, where you can create many different types of files that you can create, then upload to your site, to help the search engines “spider” your site. It works just fine for my regular sites, but it doesn’t seem to work well for my blogs, probably because I’m uploading that file to the wrong place. No matter, now that I know better how to use the WordPress Google XML Sitemap plugin.

Over Tank Magazine Holder ? White by Spectrum®

Over Tank Magazine Holder

Digiprove sealCopyright secured by Digiprove © 2010 Mitch Mitchell
Share on Google+0Share on LinkedIn0Tweet about this on Twitter0Share on Facebook0

Tags: , ,



Thanx for the BL…you know I’m going to reciprocate soon.

And I’m more than sure your PR is about to rise!

BTW: Learn the ways of Peter Parker, Mitch… Spiders are our friends 🙂
(First time I commented it didn’t seem to go through,but if it pops up just delete it)

Caleb´s last blog post..Start Blogging Now!

March 16th, 2009 | 7:04 PM

Nope, Caleb, I hate bugs of all kinds, and that includes spiders. And, of course, you had to have a tarantula; double ick! lol

March 16th, 2009 | 9:33 PM
Mieszkania Gryfino:

That link was terrible. I mean I haven’t actualy read the article since I had to close the tab once I realise that THERE’S A HUGE FRIGGIN SPIDER IN THE MIDDLE OF THE SITE!

Mieszkania Gryfino´s last blog post..Działka budowlana w Gryfinie

March 17th, 2009 | 7:36 AM

That’s funny, Mieszkania. I actually put my hand over the spider so I could read the article. Sometimes, you’ve gotta do what you’ve gotta do.

March 17th, 2009 | 9:20 AM

You guys need to try using NLP techniques to overcome this phobia

Caleb´s last blog post..Start Blogging Now!

March 17th, 2009 | 1:45 PM

Easy for you to say, Caleb; I have more than 40 years invested in this phobia. 🙂

March 17th, 2009 | 1:50 PM

By the way, Caleb, some of your posts are duplicating, and one is showing up in the spam filter. If you’re writing more than once, you don’t have to, because I check it out.

March 17th, 2009 | 3:47 PM

Didn’t know it went through 1st time…

I see you’re not the only one who didn’t know about the “tick box”.

I suspected others might have been experiencing this and not even realize it.

Caleb´s last blog Gets Simulated

March 17th, 2009 | 6:39 PM

Caleb, we all probably just thought it would be a default and not have to go in and check it off. At least I know that’s my thought about it; wrong!

March 17th, 2009 | 7:52 PM
Caleb (Market Secrets Blogger):

That needs to be stated somewhere within the SiteMap plugin!

So who’s down for telling them ‘wassup’…Sire perhaps??

March 20th, 2009 | 2:19 PM

Yeah, I’m for passing it on to Sire; he’s got all the time with his history in the computer world.

March 20th, 2009 | 4:51 PM

I never knew about the box either and sure enough the same thing applies to my blogs, have ticked the box and will see what happens. I just assumed because I had a sitemap it would do the trick. Fingers crossed now that I have checked the box it will all get indexed.

March 17th, 2009 | 6:23 PM

Khaled, overall I’m not sure what will happen, but I noticed that my Alexa ranking has improved, for what it’s worth.

By the way, what country do you live in? I’m unfamiliar with Rhondda Cynon Taff.

March 17th, 2009 | 6:40 PM

Work is in Rhondda but I’m from Swansea, Wales, UK. Wales is a Small part (Country) of the UK which most people overseas have never heard of.

March 18th, 2009 | 7:52 AM

I’ve heard of Wales, but to be truthful, I’ve always just associated it with being more of a “state” of Britain, like our United States here. Very wrong on my part, but, oddly enough, geography has always been my worst subject, for someone who’s traveled some. lol

March 18th, 2009 | 9:47 AM

Bugs I can normally put up with but spiders, especially the big ones creep me out. I tried that link on Caleb’s site and all seemed right with the world so i reckon I configure everything OK. A welcomed change that.

Sire´s last blog post..Googles Interest-Based Advertising Sucks

March 20th, 2009 | 2:14 AM

Yeah, who’d ever expect that YOU, of all people Sire, would have done something correctly? lol And big hairy spiders,… ick!

March 20th, 2009 | 9:18 AM