Block search bots on staging server
-
I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
-
Robots.txt should be sufficient, but if you want to completely block them, here is a useful resource:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site dropped in search for 95% after 18-19 feb
Hi people. I have a client which holds a escort agency website operating in Amsterdam the Netherlands. Please take note that Escort Agency is legal and holland is a liberal country with permit and all, so just trying to point out that it's a perfectly legal crafted business. To the point: we are seeing a 95% drop in traffic (basicly back to barely 10 clicks a day) after 18 to 19 feb. We've inspected the incoming links, and we did see some things going on that where not quite healthy. At first a banner was bought on a dutch advertising website. On their behalf, this sitewide banner got published with a follow on roughly 132k of pages. That was strike 1 (i think). After tossing this in disavow for temporarily basis and informed the advertisal website to put any sitewide link to nofollow in the first place, nothing changed. We found a 2nd site doing the same mistake. Frankly the banner got exposed on roughly 100k of pages on which some of 'm where barely 2 days old. Strike 2. Solved this by putting it in disavow for the time being and asking politely to put the banner again, nofollow. We cleaned out any incoming potential spammy links by using disavow. The data we obtained was a mix of google webmaster itself and moz profiling. However we're one month further now, and the graph is still a big phat flatline. What is going on? I've noted that, one other sites, which share the same brand, but completely different websites / subnets / content and all, has the same threatment going on showing a huge drop from 18th of feb 2019 and is unable to recover. We cannot see anything 'bad' actually going on @ webmasters and there is no manual action taken. So we're kind of stuck now on a site that was my project but now completely fell into oblivion and hurting someone's business. The url is https://www.qualityescort.nl/ - anyone has a reasonable advice to this issue? full.jpg full.jpg
Intermediate & Advanced SEO | | Jvanderlinde0 -
Google Search Console > Security Issues
Hi all, *Admin please feel free to remove or add this to any existing post. I have searched the community for any similar questions. While checking in the Google Search Console, under the "Security Issues" (lone section) I have found Google pointing out specific pages of our website where the message we are seeing is "Content injection - These pages appear to be modified by a hacker with the intent of spamming search results." The Learn More link takes us to https://developers.google.com/webmasters/hacked/docs/hacked_with_spam?ctx=SI&ctx=BHspam&rd=1 We've never injected spam code or have not been injected with any spammy code so what baffles me is why would Google pick this up when we have mentioned to them very clear that our code is secure and not hacked. Has anyone received a similar message and had any luck removing the message correctly? Thanks in advance!
Intermediate & Advanced SEO | | SP10 -
Does Domain Authority remian if site owner and server changes
Hi i am looking at buying an existing domain. It is currently vacant and the site is just on a godaddy holdpage. if i buy this domain, will its DA and PA TF CF still remain if we don't lose the links? thanks
Intermediate & Advanced SEO | | Direct_Ram0 -
Internal Search Results Appear in Google SERPS
My friend is running an ecommerce store selling apparels. How can we make internal search results to appear in Google SERPS and rank them? For example: the query is "peplum dress". You type the query into the internal search box and it returns a set of results. In this case, it's product listing. How can we optimize and rank it so it appears in Google SERP? Do we do it the traditional way in terms of links? Say URL is: http://www.asos.com/search/peplum-top?q=peplum+top&r=2 And we build links to it? Some of you may ask why not create a dedicated page for this, the reason being we'd have too many categories if we were to create one for each. Thoughts?
Intermediate & Advanced SEO | | WayneRooney0 -
How do I Improve Google Local search position
Hi, I think its called local search position, what I'm referring to is when you do a search on a keyword and google lists not only the best matches but also usually the second match is a group of 3 businesses with telephone numbers, google reviews and at the bottom of the group it will say something like: "See results for <your keyword="">on a map. This is what I'm referring to. in anycase my question is if I click on the link to see more results on a map I'm listed as number 3, however on the search page before where the link is displayed which I just clicked on I'm not being listed and instead one business name is being listed three times, each of the listings uses the same address but a different telephone number, In addtion the business that is being listed three times is also listed in the results being returned above in this case position #1 for the keyword I have searched. I assume this has something to do with them also being listed in the group of local businesses below three time.. The business I'm interested in getting listed in this group of results is currently being listed page 2 position 5 for the keyword..</your> Any suggestions would be greatly appreciated.. Thanks in advance..
Intermediate & Advanced SEO | | robdob11 -
Empty search results labeled as Soft 404s?
I have a site with faceted search but sometimes when someone drills down too far it ends up with no results. The page and outlined and faceted navigation are still there. The site uses dynamic URLs for the faceted navigation but Google is reporting these no results pages as Soft 404s. How should we handle these? Should we redirect these? Can we return 404 in the status code but still show the no results page they are looking for? Thanks for your responses
Intermediate & Advanced SEO | | MarloSchneider0 -
4 sites on same server with similar theme
Hi, We're building 4 e-commerce stores at the moment which are all very similar. They all sell sofas but to different markets. Product content will be different on each site so the content won't be duplicated but the overall theme of the sites will be the same, all on the same server, being launched at the same time. Should I be concerned that Google may not index all of them? Thanks, Andy
Intermediate & Advanced SEO | | AndyMediaLounge0 -
Search listing click through to wrong domain?
Hi there, I've got an interesting case that I've not seen before and can't find anything about online. Some of our SERP listings are not getting through to the correct website. Instead they go to a domain called "watchingsquare.com" (doesn't resolve), which is not affiliated with our business or the website the listings are actually for. This only seems to happen 50% of the time. For example, if I click on the listings once I'm taken to watchingsquare.com, go back and attempt again and I am sent to the correct website. Has anyone encountered similar issues and know how to resolve?
Intermediate & Advanced SEO | | pethealthinc0