The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does RSS Feed help to rank better in Google?
Hello, I heard RSS Feed helps in ranking. However, I am not sure if I should enable RSS Feed or not. Whenever I publish an article on my site , I see that many other websites have leeched my Feed and get's the same article I written published with a nofollow backlink to my website article. The worst part is that my article doesn't appear in Google search, but the website which copied my article gets ranked in Google. Although the article gets index on google (checked by using site:website.com). Although some articles show up after 24 hours by ranking higher from the sites which copied my article. Any idea what should I do? Thank you
On-Page Optimization | | hakhan2010 -
Do a bunch of footer internal links help or hurt?
We are an ecommerce site... In days gone by, having a bunch of footer links with your top products / categories was a good idea - as it created a ton of internal links to these products. Now, I am hearing that those links "dilute" the value of our other links on a page - and essentially, there is more harm than good from these. Does anyone know what I am talking about (the olds days) and should we still be doing this? Thanks
On-Page Optimization | | Ted_Cullen0 -
WordPress image urls - need a WP maven
We were having a conversation re urls that are indexed for images that are stored in various media plugins in WP. My question for anyone who is an uberWP person is: What is your opinion re best media storage plugins and how these URLs affect pages on a site for ranking, etc. I realize this is broad, but it is driven out of my concern that I cannot touch everything. When I see a url like this: http://www.drumbeatmarketing.net/wp-content/themes/drumbeat2/img/DB-LOGO-White.png I know there is no way with all the sites and clients we handle that I can get it perfect but this just bugs me for some reason. Should I just chill since it (seemingly) affects so little....?
On-Page Optimization | | RobertFisher1 -
Google crawler showing cache of another page
For the page http://www.thinkdigit.com/top-products/Laptops-and-PCs/top-10-laptops-124.php google is showing another page in cache (http://www.thinkdigit.com/top-products/Ultrabooks/top-10-ultrabooks-153.php). Please let me know how this happened and how to correct it.
On-Page Optimization | | 9dot90 -
I need some help...
I am completely perplexed here guys. I have accomplished all of the the things that the On- Page Analysis tool says that we need to perform as far as( Keyword laden page titles and webpages) yet the report comes back and gives the webpage a C and says that we still need to correct these issues. Can anyone explain this? The keywords are: " real estate augusta ga" " property management augusta ga" the address is: www.aubenrealty.com Thanks in advance, C
On-Page Optimization | | AubbiefromAubenRealty0 -
Optimization help
Looking for suggestions - one of my targeted keywords is "IT Support NY". I can't for the life of me figure out a way to use it in a sentence. Any ideas?
On-Page Optimization | | CsmBill0 -
I changed my site from HTML to PHP and I need to get some help.
Ok...so the other day I went from HTML to PHP in every part of my website. I want to know the best option for me for redirecting my pages from HTML to php. I had my site scanned with SEOMoz and I was given many 404 errors which is not at all good. I do not have any pages of my site linking to any of these html pages. All of the site links have been updated. I have checked 3 times. I have never created a robots.txt file so I would love to get a little help with this part. I was thinking it would be best to tell Google not to worry about these pages in the file. I kept the pages up and I plan to remove all code with them so that no content shows up if someone visits but the issue with that is my site is already indexed as HTML. I want to have the HTML pages redirect to the PHP without worrying that my visitors will land on my site via Google onto an HTML page. I hope I am making sense. What is the best advice you can give me. I need all pages to redirect to PHP. I used an htaccess redirect from all HTML to PHP but when I get so many of them added I get an error on my site saying too many redirects. Seriously need help.
On-Page Optimization | | TrendyHost0 -
Robots.txt file
Does it serve any purpose if we omit robots.txt file ? I wonder if spider has to read all the pages, why do we insert robots.txt file ?
On-Page Optimization | | seoug_20050