The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do subfolders for grouping common services pages help SEO?
Hello, fellow MOZers. Here is the question: Is there any benefits/disadvantages of having pages, grouped by common type of service, to be put into subfolder with keyphrase-friendly phrase? Example: We have website https://www.hyperlinksmedia.com/ and several services under "SEO" niche, such as: SEO (page name seo-houston.php) Social Media Marketing (page name social-media-marketing-houston.php) Email Marketing (page name email-marketing-houston.php) So, is it better to have these pages in root directory like this: https://www.hyperlinksmedia.com**/seo-houston.php** https://www.hyperlinksmedia.com**/houston-social-media-marketing.php** https://www.hyperlinksmedia.com**/email-marketing-houston.php** OR is it better to group them in subfolder "houston-seo-services" for "extra" keyword and convenience reasons like this: https://www.hyperlinksmedia.com/houston-seo-services/seo-houston.php https://www.hyperlinksmedia.com/houston-seo-services/social-media-marketing-houston.php https://www.hyperlinksmedia.com/houston-seo-services/email-marketing-houston.php Any thoughts will be appreciated. Thanks!
On-Page Optimization | | seomozinator0 -
Magento Duplicate Content Question - HELP!
In Magento, when entering product information, does the short description have to be different than the meta description? If they are both the same is this considered duplicate content? Thanks for the help!!!
On-Page Optimization | | LeapOfBelief0 -
SEOmoz and Wordpress SEO plugin
I made a lot of changes with Wordpress SEO by Yoast before I signed up with SEOmoz. But SEOmoz doesn't seem to see those changes. I am still being told I have multiple meta descriptions. I don't know why but SEOmoz is seeing the default meta description and not the one I wrote using the Yoast plugin. How can I get SEOmoz to see those changes? Edit: Google has not updated all the changes I made with Wordpress SEO.
On-Page Optimization | | dealblogger0 -
Is this helping?
A few months ago, in hopes of a) helping new customers navigate the sometimes-arcane language used in the precious metals industry and b) earning some props from Google for making the site more user friendly, I talked the Powers-That-Be into allowing me to create a glossary with links from some unfamiliar terms in each product description and added appropriate <title>tages for each term.</p> <p>My question is: <strong>In the opinion of SEOMoz was this endeavor a brilliant, transformative SEO brainstorm—or just a waste of my time?</strong></p> <p>We haven't been running analytics, so we are flying blind.</p> <p>Here's the 411:<br /> <br />Website: www.goldmart.com<br />Glossary: www.goldmart.com/glossary</p> <p>A sample description: http://www.goldmart.com/1-oz-pamp-suisse-lunar-snake-gold-bar-9999-fine-in-assay.html</p></title>
On-Page Optimization | | RScime250 -
Unable to see internal link numbers on Opensiteexplorer - Need help
I'm Anuj, a regular user of SEOMOZ. I need some SEO guidance from SEO experts. I'm trying to optimize a webstore for few keywords. I am facing some issues on SEO I was using https all over the webstore and was advised by the community members to not have https through out the site (Due to various reasons). The internal links were not showing up in opensiteexplorer & Google Webmaster Tools too when the site was with https (They were just showing 1 or 2). After changing the pages from https to http, I'm now able to see all the internal links of my website on GWT. Unfortunately, the internal link count on opensiteexplorer shows a very small fraction when compared to the # of internal links shown on GWT. The link update from Opensiteexplorer was on 27th FEB 2013. I had done the https to http (for all pages) somewhere between 17-24th of JANUARY 2013. I wanted to know if I have missed something as I am unable to see those numbers on Opensiteexplorer or will it take time for opensiteexplorer to show the internal link numbers ?
On-Page Optimization | | Pepperjet0 -
Will unstyled bold tags help ranking
I intend using Bold tags to add emphasis but I don't want to
On-Page Optimization | | YGF
actually show bold words
so have amended by css. are the B tags enough to help with SEO or will my amended
CSS damage the effect? Many thanks,Sinead0 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0 -
How SEOMOZ manage to have same pagerank on all pages?
I was just wondering how SEOMOZ manage to have same pagerank on all pages? Even on profile pages.
On-Page Optimization | | DiamondJewelryEmpire0