The "webmaster" disallowed all ROBOTS to fight spam! Help!!
-
One of the companies I do work for has a magento site. I am simply the SEO guy and they work the website through some developers who hold access to their systems VERY tightly. Using Google Webmaster Tools I saw that the robots.txt file was blocking ALL robots.
I immediately e-mailed out and received a long reply about foreign robots and scrappers slowing down the website. They told me I would have to provide a list of only the good robots to allow in robots.txt.
Please correct me if I'm wrong.. but isn't Robots.txt optional?? Won't a bad scrapper or bot still bog down the site? Shouldn't that be handled in httaccess or something different?
I'm not new to SEO but I'm sure some of you who have been around longer have run into something like this and could provide some suggestions or resources I could use to plead my case!
If I'm wrong.. please help me understand how we can meet both needs of allowing bots to visit the site but prevent the 'bad' ones. Their claim is the site is bombarded by tons and tons of bots that have slowed down performance.
Thanks in advance for your help!
-
Thanks for the suggestions!! I'll keep you updated.
-
You can get the list of good robots from the list at Robotstxt.org: http://www.robotstxt.org/db.html.
I'd recommend creating an edited version of the robots.txt file yourself, specifically Allowing googlebot and others. Then send that with a link to the robotstxt.org site.
You may need to get the business owners involved. IT exists to enable the business, not strap it down so it can't move.
-
What you could do is just add Allow statements for the different Googlebots and the bots of other search engines. This will probably make the developers happy so they can keep other bots out of the door (although I doubt this would work and definitely don't think that this should be the option to keep spammers away, but that says more about the quality of development ;-)).
-
Yes, there are a ton of bad bots one may want to block. Can you show us the robots.txt file? If they aren't blocking legit search engine bots, you're probably okayish. If they are actually blocking all bots, you have cause for concern.
Can you give us a screenshot from GWT?
I use a program called Screaming Frog daily. It's not malicious, off the shelf. I just want to crawl and gather meta data. I can tell it to disregard robots.txt. It will crawl a site until it hit's something password protected. There's not much any robots.txt can do about it, as it can also spoof user agents.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Divi Help!
I've added our phone number and email address in the header settings in Divi. For whatever reason, when I'm editing the header elements I can see it, but when I view the website it's not showing... I cannot figure out what the issue is. I've never run into it before. Also, the menu looks different, it does not match what it shows in the header elements edit area vs live site. XLRpuxghzHUN LxPX4iND6B 2ekykrCH7Pn
Intermediate & Advanced SEO | | LindsayE0 -
In Search Console, why is the XML sitemap "issue" count 5x higher than the URL submission count?
Google Search Console is telling us that there are 5,193 sitemap "issues" - URLs that are present on the XML sitemap that are blocked by robots.txt However, there are only 1,222 total URLs submitted on the XML sitemap. I only found 83 instances of URLs that fit their example description. Why is the number of "issues" so high? Does it compound over time as Google re-crawls the sitemap?
Intermediate & Advanced SEO | | FPD_NYC0 -
Syntax: 'canonical' vs "canonical" (Apostrophes or Quotes) does it matter?
I have been working on a site and through all the tools (Screaming Frog & Moz Bar) I've used it recognizes the canonical, but does Google? This is the only site I've worked on that has apostrophes. rel='canonical' href='https://www.example.com'/> It's apostrophes vs quotes. Could this error in syntax be causing the canonical not to be recognized? rel="canonical"href="https://www.example.com"/>
Intermediate & Advanced SEO | | ccox10 -
Webmaster tools 404
Hey, I'm getting a soft 404 error on a webpage that has content and is deferentially not a 404. We've redirect a load of urls to the web page. The url has parameters which was used before the redirect but are no longer used on by the new url, these parameters have been carried over in the redirect. Is this whats causing the soft 404 error or is there another problem that may need addressing? Also a canonical has been set on the webpage. Thanks, Luke.
Intermediate & Advanced SEO | | NoisyLittleMonkey1 -
Trailing slash and rel="canonical"
Our website is in a directory format: http://www.website.com/website.asp Our homepage display URL is http://www.website.com which currently matches our to eliminate the possibility of duplicate content. However, I noticed that in the SERPs, google displays the homepage with a trailing slash http://www.website.com/ My question: should I change the rel="canonical" to have a trailing slash? I noticed one of our competitors uses the trailing slash in their rel="canonical" Do potential benefits outweigh the risks? I can PM further information if necessary. Thanks for the assistance in advance...
Intermediate & Advanced SEO | | BethA0 -
I dont know what happened?!? Help!
Up until yesturday I was mainly getting entrances to my site through www.moondoggieinc.com/dog-harness.php like 60-70 a day After yesterday, it's dropped to one or two! I haven't made any major changes to the site? I don't know what happened! I am starting to feel really discouraged and don't know what to do! I have no clue what my next move should be in trying to get this site working well in organic search. I feel lost now, PLEASE help! Any suggestions on what to do with http://www.moondoggieinc.com would be a great help! I feel so lost! THANKS! Kristy O
Intermediate & Advanced SEO | | KristyO0 -
Canonical URL's - Do they need to be on the "pointed at" page?
My understanding is that they are only required on the "pointing pages" however I've recently heard otherwise.
Intermediate & Advanced SEO | | DPSSeomonkey0 -
How to handle "2" homepages?
Came across an interesting problem. A site has the traditional homepage of site.com and ranks okay. Later I found that another "homepage", site.com/home.html that ranks well for several terms but actually has old branding and semi-up-to-date content. Site.com/home.html has a solid linking profile but not as strong as the current homepage (site.com). The question I have is should I try to salvage the page or 301 redirect to site.com? Thank for the help!
Intermediate & Advanced SEO | | 2comarketing0