The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does RSS Feed help to rank better in Google?
Hello, I heard RSS Feed helps in ranking. However, I am not sure if I should enable RSS Feed or not. Whenever I publish an article on my site , I see that many other websites have leeched my Feed and get's the same article I written published with a nofollow backlink to my website article. The worst part is that my article doesn't appear in Google search, but the website which copied my article gets ranked in Google. Although the article gets index on google (checked by using site:website.com). Although some articles show up after 24 hours by ranking higher from the sites which copied my article. Any idea what should I do? Thank you
On-Page Optimization | | hakhan2010 -
New to MOZ and so far love tools but need some quick tips on title, keywords and descriptions
First of all:
On-Page Optimization | | nickcargill
I am excited about using all the MOZ tools! I just got back my webcrawl and have found
a lot of issues that I am working on. I am a Vacation Rental Property Manager with 150 properties, all different. 90% of my pages have the same keywords and descriptions and a lot of same page titles
too. I can change all of these by adding a few fields in my database and then
populating the meta information dynamically. For Example: <title>Big Bear Cabins | Big Bear Cabin Rentals - Chateau Alta Vista Cabin</title>
Chateau Alta Vista, Big Bear Cabins, Big Bear Cabin Rentals
Chateau ALta vista is a Big Bear Cabin with four bedrooms, three
bathrooms, it is Big Bear Luxury Property. If you notice the Title, keyword are dynamically populated from the database using the
property name of Chateau Alta Vista. The description is an extra field in the database that I just implemented that I can customize per property. I have a few questions, but here is more information: I also have 5 or 6 related pages
to each property. Pages Like general information, photos, calendars, book
Question 1): Is the best way of doing my titles, keywords and descriptions. Any concerns or recommendations. I have read, do not even use keywords anymore. Question 2) I have other pages that show maybe 8 or 10 photos. So is it ok to do the following; Title <photos of="" big="" bear="" cabin="" chateu="" alta="" vista<="" title="">Meta Keywords <photos of="" big="" bear="" cabins="" chateu="" alta="" vista="">Meta Description <check out="" photos="" of="" big="" bear="" cabin="" rentals,="" chateua="" <br="">Alta Vista, etc Currently, I have some archaic page naming with database driven url parameters such as</check></photos></photos> http://destinationbigbear.com/property_Detail_v.aspx?propid=669 and at worse http://destinationbigbear.com/Property_detail_v.aspx?propid=669&rate= $1,127.10&checkoutdate=07/13/2014&beds=4&firstnight=07/11/2014&nights=2 I do not have he ability to to full url encoding such as http://destinationbigbear.com/cabins/chateau-alta-vista but I can do http:/www.destinationbigbear.com/big-bear-cabin-info.aspx?cabin=chateau%alta%vista Question is what do I do... If I do change the page names I will lose the history of property_detail_V.aspx which only has a Page Authority of 21, but if I change the page name and dynamic navigation of my website will I not lose all my authorities, if so, is it worth doing it? My highest keyword ranking is 30 which is terrible. Nick0 -
Blocking Subdomain from Google Crawl and Index
Hey everybody, how is it going? I have a simple question, that i need answered. I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more. What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc? Hope to hear from you, Best Regards,
On-Page Optimization | | JesusD3 -
Google cache tool help
This link is for the Ebay Google cache - http://webcache.googleusercontent.com/search?q=cache:www.ebay.com&strip=1 I wanted to do the same for my homepage so I switched out the urls and it worked. When I try to get a different link in there such as mysite.com/category it wont work. I know my pages are indexed. Any ideas why it wont work for other pages?
On-Page Optimization | | EcommerceSite0 -
I have a question about having to much content on a single page. Please help :)
I am working on a music related site. We are building a feature in our system to allow people to write information about songs on their playlist. So when a song is currently being played a user can read some cool facts or information about the song. http://imgur.com/5jFumPW ( screenshot). Some playlists have over 100 songs and could be completely random in genre and artist. I am wondering if some of these playlists have over 5,000 words of content if that is going to hurt us? We will be very strict about making sure its non spammy and good content. Also for the titles of the content is it bad to have over 100 h3 tags on one page? Just want to make sure we are on the right track. Any advice is greatly appreciated.
On-Page Optimization | | mikecrib10 -
Why is the seomoz showing it crawled 3 pages when i only have 2 pages?
I had seomoz crawl my site. I only have 2 pages. The site url is www.autoinsurancefremontca.com.
On-Page Optimization | | Greenpeak0 -
How far back do you need to optimize your blog posts?
We are going through a clients blog history as they are entering a redesign phase for the blogs. We are trying to determine how far back we need to optimize past blog posts so that they can be found easier on search engines. Is it better to optimize the past years? 2 years? 6 months? only the top posts? Does anyone have any suggestions?
On-Page Optimization | | Scratch_MM0 -
Is it possible to have the crawler exclude urls with specific arguments?
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
On-Page Optimization | | djangojunkie0