Is seomoz rogerbot only crawling the subdomains by links or as well by id?
-
I´m new at seomoz and just set up a first campaign. After the first crawling i got quite a few 404 errors due to deleted (spammy) forum threads. I was sure there are no links to these deleted threads so my question is weather the seomoz rogerbot is only crawling my subdomains by links or as well by ids (the forum thread ids are serially numbered from 1 to x).
If the rogerbot crawls as well serially numbered ids do i have to be concerned by the 404 error on behalf of the googlebot as well?
-
Roger only crawls by links.
When you are using forum software or a CMS it is very common for numerous links to be created by the software without your knowledge. If you examine your crawl reports, they will show you the URL of the page which provides the link which is generating the 404 error.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Losing links
my blog/website is at ocpatentlawyer.com. My reports are showing that I'm losing links. Is there a way to find out which links I've been losing to see if it is something that I should be concerned about?
Moz Pro | | jamesjd710 -
Seomoz counting adwords traffic as well as organic?
Seomoz is tracking my adwords traffic through analytics and reporting it. how can I get it to track just organic?
Moz Pro | | SamCUK0 -
Total link graph question
how is it that our website and our two competitors basically have the exact same link graph, except for at different levels?
Moz Pro | | imageworks-2612900 -
SEOMOZ can we trust you?
I was thinking today what are the chances that SEOMOZ can look into my campaigns and see what keywords I am targeting, strategies etc. and then sell that information to others? There has to be some type of clause when you sign up correct? But who knows because I'm not sure. Having Rand and Matt Curts talking via twitter to each other kind of concerns me. Not that I'm worried that Google will know my strategies because we don't do any black-hat linking and anything shady but how much power does SEOMOZ really have and can they leak any information to other companies or Google? Scary stuff that's why I don't put all my eggs into one basket, I spread them out. Thoughts? Thanks in advance - Brett Shaffer
Moz Pro | | EmpireSEO1 -
On page links tool here at Seomoz
Hi Seomoz - first of all, thanks for the best SEO tools I have ever worked with (this is my first question in this forum, and also I just subscribed as a paying customer after the 30 days trial you guys offer). My question: After having worked for several weeks on getting the numbers of links in our forum on www.texaspoker.dk down, we are somewhat surprised to see that we didn't succeed in getting lower numbers. For instance, this page: http://www.texaspoker.dk/forum/aktuelle-konkurrencer/coaching-projekt-bliver-du-den-udvalgte has (that's what Seomoz seo tool tells us): 239 on page links. Can this really be true? We can't find these links, and we actuually did a lot to lower the numbers of links, for instance the forum members picture was a link before, and also there was a "go to top" link in each post in the forum. Thanks a lot.
Moz Pro | | MPO0 -
How long would a SEOMoz crawl usually take for a site with around 4000 pages?
We are working through optimising a site for one of our clients and the SEOMoz crawl progress says it has been running since the 8th Feburary. It's now almost a week later and it still hasn't finished. The first run took a few days, is there any way of restarting the process?
Moz Pro | | TJSSEO0 -
About NOFOLLOW tag for SEOmoz analysis
Hi all, Another issue while trying to resolve all the duplicate content SEOmoz reports to me. May be some of you guys can help: I have a dynamic error page on our website, generated in case of error, that can happen on many urls. Of course that one should not be indexed. I added the following tag on the HEADER: name="robots" content="NOODP,NOINDEX,NOFOLLOW" /> To me this should prevent from having this page indexed, but also from having this page reported by SEOmoz analyzer as duplicate content. Any hints?
Moz Pro | | nuxeo0 -
How to remove Duplicate content due to url parameters from SEOMoz Crawl Diagnostics
Hello all I'm currently getting back over 8000 crawl errors for duplicate content pages . Its a joomla site with virtuemart and 95% of the errors are for parameters in the url that the customer can use to filter products. Google is handling them fine under webmaster tools parameters but its pretty hard to find the other duplicate content issues in SEOMoz with all of these in the way. All of the problem parameters start with ?product_type_ Should i try and use the robot.txt to stop them from being crawled and if so what would be the best way to include them in the robot.txt Any help greatly appreciated.
Moz Pro | | dfeg0