Block search bots on staging server
-
I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
-
Robots.txt should be sufficient, but if you want to completely block them, here is a useful resource:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
404 vs 410 Across Search Engines
We are removing a large number of URLs permanently. We care about rankings for search engines other than Google such as Yahoo-Bing, who don't even list https status 410 code option: https://docs.microsoft.com/en-us/bingmaps/spatial-data-services/status-codes-and-error-handling Does anyone know how search engines other than Google handle 410 vs 404 status? For pages permanently being removed John Mueller at Google has stated "From our point of view, in the mid term/long term, a 404 is the same as a 410 for us. So in both of these cases, we drop those URLs from our index. We generally reduce crawling a little bit of those URLs so that we don’t spend too much time crawling things that we know don’t exist. The subtle difference here is that a 410 will sometimes fall out a little bit faster than a 404. But usually, we’re talking on the order of a couple days or so. So if you’re just removing content naturally, then that’s perfectly fine to use either one." Any information or thoughts? Thanks
Intermediate & Advanced SEO | | sb10300 -
Site under attack from Android SEO bots - expert help needed
For last 25 days, we are facing a weird attack on our site. We are getting 10x the normal mobile traffic - all from Android, searching for our name specifically. We are sure that this is not authentic traffic as the traffic is coming from Organic searches and bouncing off. Initially, we thought this was a DDoS attack, but that does not seem to be the case. It looks like someone is trying to damage our Google reputation by performing too many searches and bouncing off. Has any one else faced a similar issue before? What can be done to mitigate the impact on site. (FYI - we get ~2M visits month on month, 80% from Google organic searches). Any help would be highly appreciated.
Intermediate & Advanced SEO | | KJ_AV0 -
How to rank an ecommerce site for search terms starting with how where why
Hi guys, I just got a new SEO job for an e commerce store, the client is asking to rank the site for keywords like where to buy used phone, where to sell my used phone for for best rates and so, the question is how can i achieve that, can anyone help me with some concrete suggestion? Thanks in Advance,
Intermediate & Advanced SEO | | mkhurramali0 -
How to get product info into Google Search Result box
Hi, in the last couple of weeks I get more and more search results with a product and prices of retailers below (see sample attached). Are there Schema parameters one could use to have a bigger chance to appear there? Thanks in advance Dieter Lang 0EYJtRJ
Intermediate & Advanced SEO | | Storesco1 -
Magento Store Using Z-Blocks - Impact on SEO?
Hi Guys, I have a question relating to Z-Blocks in Magento. Our Magento store uses a lot of Z-Blocks, these are bits of content that are switched off and on depending on a customer’s user group. This allows us to target different offers and content to new customers (not logged in) and existing customers (logged in). Does anyone have any experience in how this impacts SEO? Thanks in advance!
Intermediate & Advanced SEO | | CarlWint0 -
Is a Dedicated Server a Worthwhile Investment?
Greeting MOZ Community: Our site is hosted on a virtual private server. Apparently there are dozens of other web sites hosted on the same server. The performance is usually pretty fast, with the site downloading in 1-3 seconds. However, a few times per month, the performance slows down, to say 5-6 seconds. Please see the attached image. I suspect this may have something to do with the other web sites on the server. Currently we pay about $60/month. A dedicated server would cost about $120/month. Would site performance be more consistent on a dedicated server? Could we enjoy potential SEO benefits by having our own server, i.e. could we rank slightly higher if the speed was more consistent and the performance slightly faster? Thanks, Alan 6D7kK61
Intermediate & Advanced SEO | | Kingalan10 -
Redirect Search Results to Category Pages
I am planning redirect the search results to it's matching category page to avoid having two indexed pages of essentially the same content. Example http://www.example.com/search/?kw=sunglasses
Intermediate & Advanced SEO | | WizardOfMoz
wil be redirected to
http://www.example.com/category/sunglasses/ Is this a good idea? What are the possible negative effect if I go this route? Thanks.0 -
Having Content be the First thing the bots see
If you have all of your homepage content in a tab set at the bottom of the page, but really would want that to be the first thing Google reads when it crawls your site, is there something you can implement where Google reads your content first before it reads the rest of your site? Does this cause any violations or are there any red flags that get raised from doing this? The goal here would just be to get Google to read the content first, not hide any content
Intermediate & Advanced SEO | | imageworks-2612900