The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta Robots index & noindex Both Implemented on Website
I don't want few of the pages of website to get indexed by Google, thus I have implemented meta robots noindex code on those specific pages. Due to some complications I am not able to remove meta robots index from header of every page Now, on specific pages I have both codes 'index & noindex' implemented. Question is: Will Google crawl/index pages which have noindex code along with index code? Thanks!
On-Page Optimization | | Exa0 -
Google ranking is HORRIBLE. Following SEOMoz suggestions and just can't climb.
First of all, the URL is stores.dhsequipment.com. In January, this online store switched from Homestead to Big Commerce. Since the store updated, we decided now is the time to update our product descriptions, URL's, title tags and meta descriptions. (For the first time, we had the ability to customize our URL's.) Product Description: I went through 2,500 products and updated the product description. I added an H1 & H2 to each description, and included pertinent information such as part numbers. Each product also received a new page title, meta description (which is usually the first line of the product description, don't know if this is bad or not) and a new URL, (which did redirect). Once I would complete a section, I would submit a new sitemap to Webmaster Tools. After a month and nothing happening, I started using SEOMoz which helped me rebuild some of my more important pages, such as the home page and main category pages such as: http://stores.dhsequipmentparts.com/stihl-ts420-parts/
On-Page Optimization | | pearldesign
http://stores.dhsequipmentparts.com/stihl-ts700-parts-stihl-ts800-parts/ I fetched these pages in Webmaster Tools after completion. However, it's been several weeks since and I'm still on page 4 or 5 in the SERPs. Just a little history on the store; this store has been in operation for more than 6 years. Previously, we ranked on page one for 75%+ of our products. My belief is because our URL's had history, probably more so than our competitors. I'm not sure what I should do. Business is super slow and we can't afford to wait much longer.0 -
How do i block an entire category/directory with robots.txt?
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now. The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc. I'm not really sure how i'd type it into the robots.txt file, or where to place the file. any help would be appreciated thanks
On-Page Optimization | | bricerhodes0 -
Latent semantic Indexing - Does this help rankings/relevance?
Hi, Does semantically related words to the target term on a page help with rankings/relevance? If your after the term 'PC Screen' and you use the term 'PC Monitor' will go make the connection and also reward you because of the relevance? Anyone do this and have you seen any positives? I've just started to try this out lately and have been combining it with Wordle.net to give me an indication of where the content piece is heading and how aggressive the content leans towards certain words (makes things a little more interesting then calculating densities).
On-Page Optimization | | Bondara0 -
Seomoz.org Category and Tags practice
Hello, I have been checking seomoz sourcecode and architecture these days in order to learn and to apply it in my site but I havent managed to find "tags" at all. Just the "Posts by Categories" on the right sidebar. Is this the only way you are categorising content? In this case, the only way spiders have to find your content is via these category archive pages and the general sitemap? Thanks!
On-Page Optimization | | antorome0 -
Followed seomoz advice for on page tweaks but dropped
Hi all We followed seomoz advice regarding on-page tweaks for this page: http://www.compactlaw.co.uk/agency-agreement.html Dropped 10 places by Google UK. Obviously could be a coincidence, but any on-page critique very gladly received. (But also seem to have been dropped by Google across the board in the past week or so. Usual trouble of trying to find causes. But specific page advice would be a start and much appreciated.) Regards P
On-Page Optimization | | dexm100 -
How unique do product descriptions need to be?
I'm rewriting approximately 10,000 product descriptions, but for feasability it's useful to sometimes reuse some of the phrases that are there (and on other websites as they come straight from the manufacturer -eg. key features), how unique does the content have to be - are we talking 100%, 75%, 50% for it to be effective in google? Same question goes for the product title! Many thanks.
On-Page Optimization | | ewanr0 -
Photogallery and Robots.txt
Hey everyone SEOMOZ is telling us that there are to many onpage links on the following page: http://www.surfcampinportugal.com/photos-of-the-camp/ Should we stop it from being indexed via Robots.txt? best regards and thanks in advance... Simon
On-Page Optimization | | Rapturecamps0