The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robot.txt file issue on wordpress site.
I m facing the issue with robot.txt file on my blog. Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue. The search result shows that "A description for this result is not available because of this site's robots.txt – learn more." Any suggestion to over come with this issue
On-Page Optimization | | Mustansar0 -
Broken links detected by SEOMOZ
hi there, the SEOMOZ crawl has detected two "broken links" : http://www.tarifakitesurfcamp.com/feed/
On-Page Optimization | | juanmiguelcr
http://www.tarifakitesurfcamp.com/comments/feed/ how can I fix the problem? I know that these two links are related to the user's subscription (hypothetical) to my website updates. Thanks in advanced.0 -
Too Many On-Page Links Reported By SEOmoz
Hi, I recently did run a crawl report for my blog dapazze.com, and found that SEOmoz is reporting many pages on my blog having more than 100 internal links. I opened OSE, and made a search for one of my pages which was reported to contain more than 100 links. And I found it to contain 464 internal links. Here is the link: http://www.opensiteexplorer.org/links?page=1&site=dapazze.com%2F2012%2F10%2Fwin-a-commentluv-premium-single-site-and-multi-site-license-worth-about-154-giveaway-of-october%2F&sort=page_authority&filter=&source=internal&target=page&group=0 Please have a look at it. I have chosen - Show "All" links from "only internal" pages to "this page" option in OSE, which reports me this. I see almost every page in my blog linking to every page. This is not the problem for me. I have also tried to make a search for some popular bloggers, like ProBlogger.net, ShoutMeLoud.com, HellBoundBloggers.com, etc, and all of them have the same problem. Should I be worrying about this problem? What is the problem actually?
On-Page Optimization | | rahulchowdhury0 -
Why is the seomoz showing it crawled 3 pages when i only have 2 pages?
I had seomoz crawl my site. I only have 2 pages. The site url is www.autoinsurancefremontca.com.
On-Page Optimization | | Greenpeak0 -
Seomoz.org Category and Tags practice
Hello, I have been checking seomoz sourcecode and architecture these days in order to learn and to apply it in my site but I havent managed to find "tags" at all. Just the "Posts by Categories" on the right sidebar. Is this the only way you are categorising content? In this case, the only way spiders have to find your content is via these category archive pages and the general sitemap? Thanks!
On-Page Optimization | | antorome0 -
Help on Avoid Keyword Self-Cannibalization
i ran the on page test and i got the avoid keyword self-canabilzation alert one example is for http://www.deporvillage.com/shimano maybe is because a'm not a native speaking english person but i do not understand what it exactly means the targeted keyword for this page is shimano and the alert i got for the links like the one below that i have in my page http://www.deporvillage.com/ciclismo/zapatillas-shimano which has a anchor text like zapatillas ciclismo shimano
On-Page Optimization | | deporvillage0 -
Why do I need to write meta descriptions?
I work at a marketing firm, and was trying to convince my boss that we need to start writing meta descriptions for all pages on the sites we develop for clients. He asked why this would be necessary since Google automatically pulls a snippet of content, containing the keyword(s) that had been searched, from any page listed in its SERPs. I didn't have an answer for this, and it got me wondering: When does Google actually display the meta descriptions that people write instead of scraped content? And is it really that necessary to write meta descriptions?
On-Page Optimization | | matt-145670 -
My report indicated that I have 340 crawl warnings. Not sure how to fix them. Please provide links on where I need to go to fix them.
My report indicated that I have 340 crawl warnings. Not sure how to fix them. Please provide links on where I need to go to fix them. http://pro.seomoz.org/campaigns/95663/issues#notice-issues
On-Page Optimization | | cyaindc0