When rogerbot tried to crawl my site it gets a 404\. Why?
-
When rogerbot tries to craw my site it tries http://website.com. My website then tries to redirect to http://www.website.com and is throwing a 404 and ends up not getting crawled. It also throws a 404 when trying to read my robots.txt file for some reason. We allow rogerbot user agent so unsure whats happening here. Is there something weird going on when trying to access my site without the 'www' that is causing the 404? Any insight is helpful here.
Thanks,
-
Hey Dan,
So that's the problem. Our site is up and i can manually navigate to anything including the robots.txt file. I've done this multiple times throughout the day and different days as well and manually triggered different Moz crawls at different times so i've ruled out an outage.
-
The robots.txt 404 could be a temporary outage, but it's a bit hard to tell without being able to see the actual site and robots.txt. Try checking the site is up, and you can access the robots.txt then requesting a new Moz crawl...
I do have one client who insists on blocking everything and then allowing specific crawlers, and allowing rogerbot seems to have worked fine to date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Linking shallow sites to flagship sites
We have hundreds of domains that we are either doing nothing with, or they are very shallow. We do not have the time to build enough quality content on them since they are ancillary to our flagship sites that are already in need of attention and good content. My question is...should we redirect them to the flagship site? If yes, is it ok to do this from root domain to root domain or should we link the root domain to a matching/similar page (gymfranchises.com to http://www.franchisesolutions.com/health_services_franchise_opportunities.cfm)? Or should we do something different altogether? Since we have many to redirect (if this is the route we go), should we redirect gradually?
Technical SEO | | franchisesolutions0 -
Why is my crawl taking so long?
Hi There, My crawl for albertcuyp.nl is taking very long, it started on the 10th of april. I don't know whats going on but i think 2 weeks for a crawl is extremely long. Can you help me?
Technical SEO | | KnowHowww0 -
Poor Site Performance
Hello, A couple of months ago, this site was dropped from google due to a noindex, nofollow tag thewealthymind(dot)com It's back up, but performing poorly. Take for example the term "The 4 step belief change" in the home page title tag. This site is the #1 authority on that and yet it ranks 3rd below weaker pages. There's 180 404 errors in GWT, many from past versions of pages of the site but also including thewealthymind(dot)com/index.html and thewealthymind(dot)com/index.htm even though there is a rel=cononical tag on the home page. What's the process of getting this site back to health?
Technical SEO | | BobGW0 -
If you are organizing the site structure for an ecommerce site, how would you do it?
Should you use not use slashes and use all dashes or use just a few slashes and the rest with dashes? For example, domain.com/category/brand/product-color-etc OR domain.com/anythinghere-color-dimensions-etc Which structure would you rather go for and why?
Technical SEO | | Zookeeper0 -
Site Wide Links
I have a link on pr 3 home page website placed in the side bar. It is on a WordPress website that spans a couple hundred pages and the side bar is on every page. The majority of the pages are not ranked or have any pr. Can this affect me negatively?
Technical SEO | | raph39880 -
404 appearing in Sitelinks
I am little embarresed to say that the custom 404 page is appearing in a client's Sitelinks for their brand name search term. I have demoted this page from Sitelinks for the brand term but it still appears 😞 What is the best practise for a custom 404 page appearing in a sitemap and/or being blocked by robots.txt? Thanks
Technical SEO | | mccormackmorrison0 -
Site Disappeared off of Search
A friend of mine has a site (http://bit.ly/q4iWkM ) that was ranking number one for their key word (Drimnagh() and has now completely disappeared off of the ranking. I did some checking and can't see a problem. She does have duplicate meta and titles throughout but this shouldn't be a punishable offence that I know of and is something that I am going to correct with a quick plugin install. I couldn't see any redirects or code stopping search either. When you do site:URL it shows up OK as well. She is client of mine (for website not for SEO) and she is really upset about it so any help from the forum would be appreciated. This isn't even a site I did but you couldn't get a better person to work with so I am eager to help where and if possible. Guinness all round if someone solves it next time you are in Ireland
Technical SEO | | kdaly1000 -
Should I try to optimize for SEO a site that lives only for 5 days
Hi, we are actually building a new microsite that will live on promo.domain.com. The site will have a promo for about 5 days and will be changed for a new promo after 5 days. Considering that the site www.domain.com has a high autority (67) and is well indexed in search engine, should I try to optimise this site for keywords such as promo "keyword" rebate "keyword" cheap "keyword" even if the site will be optimized for those keywords only for 5 days. We are already doing PPC campaign on these keywords but I am wondering if Google will have the time to rank us in the top 10 results in those 5 days or if I am loosing my time. My other option is to leave the TITLE of this site always the same like Groupon is doing and focus on very generic keywords. Which option do you think is the best?
Technical SEO | | Adviso0