Will robots.txt override a Firewall for Rogerbot?
-
Hey everybody.
Our server guy, who is sorta difficult, has put these ridiculous security measures in place which lock people out of our website all the time. Basically if I ping the website too many times I get locked out, and that's just on my own, doing general research.
Regardless, all of our audits are coming back with 5xx errors and I asked if we could add rogerbot to the robots.txt. He seems to be resistant to the idea and just wants to adjust the settings to his firewall...
Does anybody know if putting that in the Robots.txt will override his firewall/ping defense he has put in place? I personally think what he has done is WAY too overkill, but that is besides the point.
Thanks everybody.
-
So I spoke with our host. Basically he has been adjusting the port flood settings because of a DDoS attack we had roughly 9 months ago.
We have roughly 1000 domains on the same server, all with wordpress. I went through and changed the nameserver on around 800 of them to bring us down. In the long run, I want to bring us to 1 website. There is no reason for us to have 200, or 5 for that matter. They are redundant websites that were build simply to bolster our main website by blackhat tactics.
Our host stated that the only way to keep things kosher would be to switch all 1,000 domains to a new server every 2 years because once the "hackers" find out that there is a cluster of 1,000 domains in the same place, they will blast it.
Anyway, I'm working on cutting the domains in the safest way possible, and switching servers as soon as possible!
-
Yes it does thank you!
When I asked out dev (who also hosts our domains) to adjust the settings for the rogerbot he said
"3 pages per second is basically me undoing the portflood setting completely, thus rending the site very insecure to brute force attempts, which would inevitably drive the server load very high in anywhere from 3-24 hours."
I am glad that he is concerned about the security of our website. At the same time, I find it hard to believe we need anything near this intense. We do not have any online store, we do not collect credit card data or anything like that.
It seems overkill...
-
Unfortunately, no. The security he has in place will block the crawler access before it ever gets a chance to see the robots file.
If you have a dev who's making business-limiting decisions, you have a major problem and need to address that first.
Hope that helps?
Paul
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Much Time It Will Take To Lower the Spam Score?
I'm facing an issue with my website. Due to little to no knowledge about link building and backlink, i created backlinks without checking the quality and spam score of the sites. Now there are many sites linking to my website but the overall spam score is very high of my website and my domain is reviewheart.com that i'm talking about. I have created a list and disavowed all he poor linking domains but still no improvement shown. Can anyone have the idea how much time moz will take to show the updated spam score as i have disavowed all the low quality spammed linking site?
Moz Bar | | rajas20191 -
Will Removing or Disavowing Toxic Links Improve MOZ Domain Authority?
The vast majority of the 140 domains that link to our website are very low quality directories or and other toxic links. Only about 20-30 domains are not toxic (according to Link Research Tools confirmed by out manual inspection of these links). Would removing some of these links improve of MOZ Domain Rank? What if we cannot remove them, can NOZ detect a disavow file? In general would improving the ratio between good quality and poor quality links improve domain authority? Thanks,
Moz Bar | | Kingalan1
Alan2 -
Is the update site crawl feature following robot.txt rules?
I noticed that most of the errors would not be occurring if Moz's tool followed the rules implemented in sites robots.txt. Has anyone else seen this problem and do you know if Moz will fix this?
Moz Bar | | jamestown0 -
Why RogerBot can't crawl site https://unplag.com
Hello Please help me to solve the problem. The on-page grader and Crawl Test are not working for Unplag.com website. Both said that they can't access the url. Yes, I've tried different variants like unplag.com, http://unplag.com One more thing - RogerBot was disallowed in robots.txt file. I deleted it from the file a week ago so maybe moz index haven't been renewed.
Moz Bar | | Targeras0 -
Meta Robots "Index, Follow"
In my MozBar under "General Attributes" it says "index, follow" next to Meta Roberts for one of our client's websites. I've never seen "index, follow" before. I've seen it say "not found." What does index, follow mean and is that a bad thing? I know the reason should be obvious but this site has had a lot of problems and I'm wondering if this is related.
Moz Bar | | SEOhughesm1 -
We Launched a new site and Rogerbot is still reporting on links/errors from the old site, is there a way to clear those out?
We are mostly a Branding agency, and have not put a lot of effort into SEO for ourselves... SEO tends to take a backseat to design most of the time, making it a little difficult for me at times when it comes to SEO. We recently launched a new site, http://Roninadv.com/ and the developer and I have done quite a bit of work to make it work well for Google. I was really looking forward to a new crawl report from Roger, but alas, It's like Roger crawled the old site? The new site has been up since last Monday. Is there a way to clear out the old errors? Do I just need to give roger more time?
Moz Bar | | PaulRonin0 -
Will the videos for the 2013 Moz Con be available for purchase in a bundle like they were in 2012? If so when? Thanks!
Will the videos for the 2013 Moz Con be available for purchase in a bundle like they were in 2012? If so when and how much? I would like to purchase. Thanks!
Moz Bar | | larahill0 -
New moz analytics...will it have reporting?
Loving the new Moz Analytics...very shiny. Just wondering if it will have reporting tied to it at some point? Thanks..Jeannie x.
Moz Bar | | Jeannie.0