Will robots.txt override a Firewall for Rogerbot?
-
Hey everybody.
Our server guy, who is sorta difficult, has put these ridiculous security measures in place which lock people out of our website all the time. Basically if I ping the website too many times I get locked out, and that's just on my own, doing general research.
Regardless, all of our audits are coming back with 5xx errors and I asked if we could add rogerbot to the robots.txt. He seems to be resistant to the idea and just wants to adjust the settings to his firewall...
Does anybody know if putting that in the Robots.txt will override his firewall/ping defense he has put in place? I personally think what he has done is WAY too overkill, but that is besides the point.
Thanks everybody.
-
So I spoke with our host. Basically he has been adjusting the port flood settings because of a DDoS attack we had roughly 9 months ago.
We have roughly 1000 domains on the same server, all with wordpress. I went through and changed the nameserver on around 800 of them to bring us down. In the long run, I want to bring us to 1 website. There is no reason for us to have 200, or 5 for that matter. They are redundant websites that were build simply to bolster our main website by blackhat tactics.
Our host stated that the only way to keep things kosher would be to switch all 1,000 domains to a new server every 2 years because once the "hackers" find out that there is a cluster of 1,000 domains in the same place, they will blast it.
Anyway, I'm working on cutting the domains in the safest way possible, and switching servers as soon as possible!
-
Yes it does thank you!
When I asked out dev (who also hosts our domains) to adjust the settings for the rogerbot he said
"3 pages per second is basically me undoing the portflood setting completely, thus rending the site very insecure to brute force attempts, which would inevitably drive the server load very high in anywhere from 3-24 hours."
I am glad that he is concerned about the security of our website. At the same time, I find it hard to believe we need anything near this intense. We do not have any online store, we do not collect credit card data or anything like that.
It seems overkill...
-
Unfortunately, no. The security he has in place will block the crawler access before it ever gets a chance to see the robots file.
If you have a dev who's making business-limiting decisions, you have a major problem and need to address that first.
Hope that helps?
Paul
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does "Disallow: /xmlrpc.php" in robots.txt affect moz tools ability to fetch DA?
Just checked a website for Domain Authority using Moz' tool, however it returned 1 for DA, which should be unlikely. I have been trying to find the problem and found "Disallow: /xmlrpc.php" in robots.txt. Could this affect Moz' tools ability to get the required data?
Moz Bar | | Foli0 -
Is the update site crawl feature following robot.txt rules?
I noticed that most of the errors would not be occurring if Moz's tool followed the rules implemented in sites robots.txt. Has anyone else seen this problem and do you know if Moz will fix this?
Moz Bar | | jamestown0 -
Rogerbot will not crawl my site! Site URL is https but keep getting and error that homepage (http) can not be accessed. I set up a second campaign to alter the target url to the newer https version but still getting the same error! What can I do?
Site URL is https but keep getting and error that homepage (http://www.flogas.co.uk/) can not be accessed. I set up a second campaign to alter the target url to the newer https://www.flogas.co.uk/ version but still getting the same error! What can I do? I want to use Moz for everything rather than continuing to use a separate auditing tool!
Moz Bar | | digitalascend0 -
Crawl Diagnostics: How many pages (deep) will it crawl for dup content
Does anyone know how deep the crawl diagnostics will crawl when searching for dup content? Will it crawl the entire site, or will it only crawl "x" amount of pages? Thanks!
Moz Bar | | tdawson090 -
MozBar > General Attributes > Meta Robots > noindex
I'm having a hard time figuring out where the noindex value for Meta Robots is coming from so I can fix it. Can anybody spot the issue or point me to some docs that show were the MozBar finds this information http://www.produnkhoops.com
Moz Bar | | tatermarketing0 -
I am doing the On-page SEO for a website that's never had any SEO done before. I will start with the Pages. Is it necessary to do SEO/Keywords for older Posts?
Using the On-Page Grader to perform SEO on pages of WordPress website. This website never had any SEO done before. Should I go back and perform the same work on the Post Pages?
Moz Bar | | Joseph.Lusso0 -
Where's Rogerbot?
Could someone please tell me where Rogerbot lives now?! Unless I'm having distorted memories, I used to be able to crawl websites with Rogerbot (that are not set-up as Campaigns). Could someone please let me know where to find this now? p.s. I didn't really want to Q&A this, but after a while clicking around moz I'm now even questioning myself!
Moz Bar | | GregDixson0