Blocking Test Pages Enmasse on Sub-domain
-
Hello,
We have thousands of test pages on a sub-domain of our site. Unfortunately at some point, these pages were visible to search engines and got indexed. Subsequently, we made a change to the robots.txt file for the test sub-domain. Gradually, over a period of a few weeks, the impressions and clicks as reported by Google Webmaster Tools fell off for the test. sub-domain.
We are not able to implement the no index tag in the head section of the pages given the limitations of our CMS.
Would blocking off Google bot via the firewall enmasse for all the test pages have any negative consequences for the main domain that houses the real live content for our sites (which we would like to of course remain in the Google index).
Many thanks
-
If you want nothing on that test subdomain indexed, verify that subdomain as its own site in Google Webmaster Tools, exclude that subdomain from being indexed in robots.txt, then request removal of that site (subdomain) in GWT.
And consider setting up a page monitor like https://polepositionweb.com/roi/codemonitor/index.php on the robots.txt of your test site (and live site). It'll check the contents of those pages once a day, and email you if there's a change. Handy if there are multiple people working on the site.
-
I'm a bit confused. Didn't blocking the test subdomain with the robots.txt already accomplish what you are trying to do? Or are the test pages still somehow indexed? Or is your main site affected by the robots.txt? Anyway, I would suggest using the .htaccess file to block search engines from accessing the subdomain rather than a firewall - http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-reposit
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain Authority vs. Page Authority: What’s More Important?
I have a couple of questions about this. First of all is one more important than the other? Also, I currently have a website setup for https://gogoanime.city. How is it possible to grow page authority yet I am having a hard time growing domain authority. Thank you !
Technical SEO | | gogoanimetp0 -
An informational product page AND a shop page (for same brand)
Hi all, This is my first foray into e-commerce SEO. I'm working with a new client who sells upscale eBikes online. Since his products are expensive, he wants to have informational pages about the brands he sells eg. www.example.com/brand. However these brands are also category pages for his online shop eg. www.example.com/shop/brand I'm worried about keyword cannibalization and adding an extra step/click to get to the shop (right now the navigational menu takes you to the information page and from there you have to click to get to the shop) I'm pretty sure it would make more sense to have ONE killer shopping page that includes all the brand information but I want to be 100% sure before I advise him to take this big step. Thoughts?
Technical SEO | | MouthyPR1 -
Will doing a 301 redirect for one domain to another give the latter domain the formers links?
I have some websites that I built a few years ago that are still in existence, but I no longer have access to the sites as they weren't hosted by myself. These sites all carry a "Designed by Me" text on the footer with a link to my (now old) website. I have since done 301 redirects on the domain names that are used in the footers of these sites so they link directly to my new site. However, will these websites now show up on Google Webmasters for example as external links to my site?
Technical SEO | | mickburkesnr0 -
Why is Google Webmaster Tools showing 404 Page Not Found Errors for web pages that don't have anything to do with my site?
I am currently working on a small site with approx 50 web pages. In the crawl error section in WMT Google has highlighted over 10,000 page not found errors for pages that have nothing to do with my site. Anyone come across this before?
Technical SEO | | Pete40 -
Advice on whether we 301 redirect a page or update existing page?
Hi guys, any advice would be really appreciated. We have an existing page that ranks well for 'red widgets'. The page isn't monetised right now, but we're bringing in a new product onto our site that we optimised for 'blue widgets'. Unfortunately, not enough research was done for this page and we've now realised that consumers actually search for 'red widgets' when looking for the product we're creating as 'blue widgets'. The problem with this is that the 'red widgets' page is in a completely different category of our site than what it needs to be (it needs to be with 'blue widgets'). So, my question is; Should we do a 301 redirect from our 'red-widgets' page to our 'blue-widgets' page which we want to update and optimise the content on there for 'red-widgets'. Or, should we update the existing red-widgets page to have the right products and content on there, even thought it is in the wrong place of our site and users could get confused as to why they are there. If we do a 301 redirect to our new page, will we lose our rankings and have to start again, or is there a better way around this? Thanks! Dave
Technical SEO | | davo230 -
Domain Forwarding / Multiple Domain Names / or Rebuild Blogs on them
I am considering forwarding 3 very aged and valuable domain names to my main site. There were once over 100 blog posts on each blog and each one has a page authority of 45 and domain authority of 37. My question is should i put up three blogs on the domains and link them to my site or should i just forward the domains to my main site? Which will provide me with more value. I have the capability to have some one blog on them every day. However, i do not have access to any of the old blog posts. I guess i could scrape it of archive.org. Any advice would be appreciated. Scott
Technical SEO | | WindshieldGuy-2762210 -
On-Page Question
Im trying to increase value to specific pages by putting history, and additional images. Will copying snippets from other sites negatively affect me? Should the content be re-written completely?
Technical SEO | | Anest0 -
301 lots of old pages to home page
Will it hurt me if i redirect a few hundred old pages to my home page? I currently have a mess on my hands with many 404's showing up after moving my site to a new ecommerce server. We have been at the new server for 2 years but still have 337 404s showing up in google webmaster tools. I don't think it would affect users as very few people woudl find those old links but I don't want to mess with google. Also, how much are those 404s hurting my rank?
Technical SEO | | bhsiao1