Welcome to the Q&A Forum

Hi All,

Does anyone have a suggested Screaming Frog  (SF) configuration to mirror default Googlebot crawl? I want to test my site and see if it will return 429 "Too Many Requests" to Google.

I have set the User Agent as Googlebot (Smartphone). Is the default SF Menu > Configuration > Speed > Max Threads 5 and Max URLs 2.0 comparable to Googlebot?

Context:
I had tried NetPeak SEO Spider which did a nice job and had a cool feature that would pause a crawl if it got to many 429. Long Story short, B2B site  threw 429 Errors when there should have been no load on a holiday weekend at 1:00 AM.

UPDATED 4/29/2019 4:33 PM

• I had made to many copy and pastes. Product pages are corrected

Upon researching the hreflang x-default tag, I am getting some muddy results for implementation on an international company site  older results say just homepage or the country selector but….

My Question/Direction going forward for the International Site I am working on:  I believe I can to put x-default all the pages of every country and point it to the default language page for areas that are not covered with our current sites. Is this correct?

From my internet reading, the x-default on every page is not truly necessary for Google but it will be valid implemented.

My current site setup example:
https://www.bluewidgets.com Redirects to https://www.bluewidgets.com/us/en (functions as US/Global)

Example Countries w/ code Site:- 4 countries/directories US/Global, France, Spain

Would the code sample below be correct?

1. https://www.bluewidgets.com/us/en/ (functions as US/Global)

2. US/Global Country Homepage - https://www.bluewidgets.com/us/en/

3. US/Global Country Product Page(s) This would be for all products - https://www.bluewidgets.com/us/en/whizzer-5001/

4. http://www.bluewidgets.com/us/en (functions for France)

5. France Country Homepage - https://www.bluewidgets.com/fr/fr/

6. France Country Product Page(s) This would be for all products- https://www.bluewidgets.com/es/es/whizzer-5001

7. http://www.bluewidgets.com/us/en (functions as Spain)

8. Spain Country Homepage - https://www.bluewidgets.com/es/es/

9. Spain Country Product Page(s) This would be for all products - https://www.bluewidgets.com/es/es/whizzer-5001

Thanks for the spot check

• Gravy

YES!!!!

Always use lowercase for filenames because if you use Upper and Lower (sometimes called camel case) for your internal and menu linking Google will crawl it and index the U&l. Then the fun begins when you have to match your sitemap to that!