Suggested Screaming Frog configuration to mirror default Googlebot crawl?
-
Hi All,
Does anyone have a suggested Screaming Frog (SF) configuration to mirror default Googlebot crawl? I want to test my site and see if it will return 429 "Too Many Requests" to Google.
I have set the User Agent as Googlebot (Smartphone). Is the default SF Menu > Configuration > Speed > Max Threads 5 and Max URLs 2.0 comparable to Googlebot?
Context:
I had tried NetPeak SEO Spider which did a nice job and had a cool feature that would pause a crawl if it got to many 429. Long Story short, B2B site threw 429 Errors when there should have been no load on a holiday weekend at 1:00 AM.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Webmaster tools -Fixing over 20,000+ crawl errors
Hi, I'm trying to gather all the 404 crawl errors on my website after a recent hacking that I've been trying to rectify and clean up. Webmaster tools states that I have over 20 000+ crawl errors. I can only download a sample of 1000 errors. Is there any way to get the full list instead of correcting 1000 errors, marking them as fixed and waiting for the next batch of 1000 errors to be listed in Webmaster tools? The current method is quite timely and I want to take care of all errors in one shot instead of over a course of a month.
Intermediate & Advanced SEO | | FPK0 -
Website Indexing Issues - Search Bots will only crawl Homepage of Website, Help!
Hello Moz World, I am stuck on a problem, and wanted to get some insight. When I attempt to use Screaming Spider or SEO Powersuite, the software is only crawling the homepage of my website. I have 17 pages associated with the main domain i.e. example.com/home, example.com/sevices, etc. I've done a bit of investigating, and I have found that my client's website does not have Robot.txt file or a site map. However, under Google Search Console, all of my client's website pages have been indexed. My questions, Why is my software not crawling all of the pages associated with the website? If I integrate a Robot.txt file & sitemap will that resolve the issue? Thanks ahead of time for all of the great responses. B/R Will H.
Intermediate & Advanced SEO | | MarketingChimp100 -
Duplicate content issues from mirror subdomain : facebook.domianname.com
Hey Guys,
Intermediate & Advanced SEO | | b2bmarketer
Need your suggestions.
I have got a website that has duplicate content issue.
a sub-domain called facebook.asherstrategies .com comes from no where and is getting indexed.
Website Link : asherstrategies .com
subdomain link: facebook.asherstrategies .com This sub domain is actually a mirror of the website and i have no idea how is is created.
trying to resolve the issue but could not find the clue.0 -
Google crawling different content--ever ok?
Here are a couple of scenarios I'm encountering where Google will crawl different content than my users on initial visit to the site--and which I think should be ok. Of course, it is normally NOT ok, I'm here to find out if Google is flexible enough to allow these situations: 1. My mobile friendly site has users select a city, and then it displays the location options div which includes an explanation for why they may want to have the program use their gps location. The user must choose the gps, the entire city, or he can enter a zip code, or choose a suburb of the city, which then goes to the link chosen. OTOH it is programmed so that if it is a Google bot it doesn't get just a meaningless 'choose further' page, but rather the crawler sees the page of results for the entire city (as you would expect from the url), So basically the program defaults for the entire city results for google bot, but for for the user it first gives him the initial ability to choose gps. 2. A user comes to mysite.com/gps-loc/city/results The site, seeing the literal words 'gps-loc' in the url goes out and fetches the gps for his location and returns results dependent on his location. If Googlebot comes to that url then there is no way the program will return the same results because the program wouldn't be able to get the same long latitude as that user. So, what do you think? Are these scenarios a concern for getting penalized by Google? Thanks, Ted
Intermediate & Advanced SEO | | friendoffood0 -
Duplicate Page Content Errors on Moz Crawl Report
Hi All, I seem to be losing a 'firefighting' battle with regards to various errors being reported on the Moz crawl report relating to; Duplicate Page Content Missing Page Title Missing Meta Duplicate Page Title While I acknowledge that some of the errors are valid (and we are working through them), I find some of them difficult to understand... Here is an example of a 'duplicate page content' error being reported; http://www.bolsovercruiseclub.com (which is obviously our homepage) Is reported to have 'duplicate page content' compared with the following pages; http://www.bolsovercruiseclub.com/guides/gratuities http://www.bolsovercruiseclub.com/cruise-deals/cruise-line-deals/holland-america-2014-offers/?order_by=brochure_lead_difference http://www.bolsovercruiseclub.com/about-us/meet-the-team/craig All 3 of those pages are completely different hence my confusion... This is just a solitary example, there are many more! I would be most interested to hear what people's opinions are... Many thanks Andy
Intermediate & Advanced SEO | | TomKing0 -
Wikipedia page need suggestions
http://en.wikipedia.org/wiki/Muslim_Academy Recently created this page but giving two errors at the moment. Need your advice with how to fix these two point mentioned by wikipedia. | This article has no links to other Wikipedia articles. (July 2013) This article is an orphan, as no other articles link to it. (July 2013) |
Intermediate & Advanced SEO | | csfarnsworth0 -
301 redirections done too late - What do you suggest?
Hi, When pushing our new site live, most of the 301 redirections got done too late for several reasons. Understandably, our site rankings in google have taken a hit now. So far we have just tried to perfectly optimize the pages that used to rank well (They weren't even optimized before and were still ranking) , to get our positions back. But does anyone have an idea about what else we could do? Is there a recommended "action plan" when someone is late with their 301 redirections?
Intermediate & Advanced SEO | | JohanMattisson0 -
Suggestions for a cost effective, SEO safe domain that was previously penalised
Hi, I will try to explain my situation as clearly as I can, and any positive advise would be greatly appreciated. Obviously please let me know if you have any questions but I'm sorry the domains are private. I started by business 4 years ago and launched a website (site A) and worked hard to promote it in the best way I knew how. It brought in a good income for around 3 years but then was hit with some sort of google penalty/filter! Knowing then what I know now would probably of avoided the problem altogether but that's another story... So I brought another domain (site B) and started again with a new design and completely re-branded the company, and again have been working hard to promote it. It has been preforming well and there has been steady progress 😉 However since then I have been steadily promoting the penalised site (site A) and keeping an eye on it to see if the penalty/filter may be lifted. It had initially lost around 75% of traffic, but recently has been doing much better in SERPS and is again on the increase. My problem is that with my company now being completely re branded I want to keep consistency but (site A) looks old and dated compared the new one (site B) and I don’t want to be confusing users etc... So I need a cost efficient and “safe” solution to this in terms of SEO and budget. 301 Redirect to site B I thought of a 301 redirect “BUT” I'm concerned about the penalty/filter being passed onto the new site (site B) and have read this dose happen ? Complete Redesign/brand for site A This would probably be the best option except I'm limited on funds. I would need “another” full commerce site as its just way to much money at the moment. Remove site A completely Funds are tight and I'm still feeling the affects of the penalty so really can't afford to loose any traffic at all! Use site A as a micro-site I thought a micro site with just the main product landing pages being used. I would use the same design as site B, then re-write the text and then link everything to the new site. “BUT” I'm concerned about getting another penalty (duplicate) as all the anchor text links going to site B site would be identical! EG. To use the same design as site B I would need to use the same layout etc including navbars, anchor text links in the footer etc.. and I'm worried this may trigger a duplicate content penalty ? I hope there are some suggestion for my situation and thanks in advance for your help. Thanks Chris.
Intermediate & Advanced SEO | | doorguy880