Blocking Test Pages Enmasse on Sub-domain
-
Hello,
We have thousands of test pages on a sub-domain of our site. Unfortunately at some point, these pages were visible to search engines and got indexed. Subsequently, we made a change to the robots.txt file for the test sub-domain. Gradually, over a period of a few weeks, the impressions and clicks as reported by Google Webmaster Tools fell off for the test. sub-domain.
We are not able to implement the no index tag in the head section of the pages given the limitations of our CMS.
Would blocking off Google bot via the firewall enmasse for all the test pages have any negative consequences for the main domain that houses the real live content for our sites (which we would like to of course remain in the Google index).
Many thanks
-
If you want nothing on that test subdomain indexed, verify that subdomain as its own site in Google Webmaster Tools, exclude that subdomain from being indexed in robots.txt, then request removal of that site (subdomain) in GWT.
And consider setting up a page monitor like https://polepositionweb.com/roi/codemonitor/index.php on the robots.txt of your test site (and live site). It'll check the contents of those pages once a day, and email you if there's a change. Handy if there are multiple people working on the site.
-
I'm a bit confused. Didn't blocking the test subdomain with the robots.txt already accomplish what you are trying to do? Or are the test pages still somehow indexed? Or is your main site affected by the robots.txt? Anyway, I would suggest using the .htaccess file to block search engines from accessing the subdomain rather than a firewall - http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-reposit
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Are Some Pages On A New Domain Not Being Indexed?
Background: A company I am working with recently consolidated content from several existing domains into one new domain. Each of the old domains focused on a vertical and each had a number of product pages and a number of blog pages; these are now in directories on the new domain. For example, what was www.verticaldomainone.com/products/productname is now www.newdomain.com/verticalone/products/product name and the blog posts have moved from www.verticaldomaintwo.com/blog/blogpost to www.newdomain.com/verticaltwo/blog/blogpost. Many of those pages used to rank in the SERPs but they now do not. Investigation so far: Looking at Search Console's crawl stats most of the product pages and blog posts do not appear to be being indexed. This is confirmed by using the site: search modifier, which only returns a couple of products and a couple of blog posts in each vertical. Those pages are not the same as the pages with backlinks pointing directly at them. I've investigated the obvious points without success so far: There are a couple of issues with 301s that I am working with them to rectify but I have checked all pages on the old site and most redirects are in place and working There is currently no HTML or XML sitemap for the new site (this will be put in place soon) but I don't think this is an issue since a few products are being indexed and appearing in SERPs Search Console is returning no crawl errors, manual penalties, or anything else adverse Every product page is linked to from the /course page for the relevant vertical through a followed link. None of the pages have a noindex tag on them and the robots.txt allows all crawlers to access all pages One thing to note is that the site is build using react.js, so all content is within app.js. However this does not appear to affect pages higher up the navigation trees like the /vertical/products pages or the home page. So the question is: "Why might product and blog pages not be indexed on the new domain when they were previously and what can I do about it?"
Technical SEO | | BenjaminMorel0 -
Many Pages Being Combined Into One Long Page
Hi All, In talking with my internal developers, UX, and design team there has been a big push to move from a "tabbed" page structure (where as each tab is it's own page) to combining everything into one long page. It looks great from a user experience standpoint, but I'm concerned that we'll decrease in rankings for the tabbed pages that will be going away, even with a 301 in place. I initially recommending#! or pushstate for each "page section" on the long form content. However there are technical limitations with this in our CMS. The next idea I had was to still leave those pages out there and to link to them in the source code, but this approach may get shot down as well. Has anyone else had to solve for this issue? If so, how did you do it?
Technical SEO | | AllyBank1 -
Pageing page and seo meta tag questions
Hi if i am using paging in my website there is lots of product in my website now in paging total paging is 1000 pages now what title tag i need to add for every paging page or is there any good way we can tell search engine all page or same ?
Technical SEO | | constructionhelpline0 -
Homepage dropping back to page 30 and being replaced by a random page?
Hi All Please accept my apologies if i have posted this in the wrong place, i am new to this. I have asked for help over and over again on Google Webmaster Forum but everytime i am faced with sarcastic, unhelpful answers and then moaned at for asking the same question again when i get no answers. Well, my website is http://www.hillfieldscampingandleisure.co.uk. The site is nearly 2 years old and is an ecommerce online camping equipment store. It is hosted on the EKMPOWERSHOP Platform. After a about a year of adding products and designing my site i decided to hire an SEO Company based in the UK, they were a good company with some big clients. Anyways to cut a really long story short....they completely ripped me off by £700 a month for 7 months for my site to keep going backwards, they wouldnt target the keywords i wanted and all they did was provide really spammy, non relevant, no page rank links...my site ended up on number 31 of Google. I managed to drop the company and try to do things myself. I optimized my sites content so it wasn't keyword stuffed I re-wrote all my alt tags to look more natural I optimized my meta and h1 tags I carried on with trying to build relevant, high page rank links Anyways i managed to get my homepage to page 3/4 of Google. It stayed there for a few weeks but over the past few weeks my Homepage is dropping back to page 28-30 and being replaced with a random page of my site on page 4-6. It corrects itself after a while and my homepage returns but then it happens all over again....today i have a random page on page 4 and my homepage is on page 29. Any ideas on what is causing this and how can i get my site up there? I have had some ideas come back that it is the EKM platform i am using but since the seo company took the p out of me, its the only one i can afford at the moment until i start selling. I am a small business with stock waiting to be sold but no matter how much i read and rules to follow my site just doesn't seem to move. Any help would be really really apreciated and be nice!
Technical SEO | | hillfields0 -
Top pages give " page not found"
A lot of my top pages point to images in a gallery on my site. When I click on the url under the name of the jpg file I get an error page not found. For instance this link: http://www.fastingfotografie.nl/architectuur-landschap/single-gallery/10162327 Is this a problem? Thanks. Thomas. JkLej.png
Technical SEO | | thomasfasting0 -
Linking from and to pages
My website, www.kamperen-bij-de-boer.com, tells people what campingssites can be found in The Netherlands for recreational purposes. In order for a campingsite to be mentioned on our website we ask them to place a link to our website (either using a text link or image link) and then we make a page for that campsite on our website with in the end a link to ther website, e.g. http://www.kamperen-bij-de-boer.com/Minicamping-In-t-Oldambt.html -> they in return link back to us. Since this comes natural will this or won't this be penalized by Google and so on for linkfarming. At this moment we have about 600 camping sites on our website alone linking to us (not all of them) and we are linking to them. Since this can be explained as link trading which is not as good for your ranking as one-way-linking what should be wise? Should i include a nofollow? I already have many links from other sites linking to mine without having to link back, is there anything else i can do with linking to ensure better ranking?
Technical SEO | | JarnoNijzing0 -
Keyword domains
Hi everyone. Two questions regarding keyword domains (e.g. "widgets.com") If we have to choose a domain with an extra word, does it make a difference to have the added word before or after? E.g. "my-widgets.com" vs "widgets-now.com" Does it make a difference if the extra word is a generic vs a 'real' word? E.g. "my-widgets.com" vs "japanese-widgets.com" Thanks a lot for your feedback!
Technical SEO | | hectorpn0 -
Sub-Domain Choice Dilemma
We have successfully rolled out 5 sub-domains using very industry specific KWs as the sub, e.g. familylawyers. We're rolling our an employment focused sub.and ideally would use employmentlawyers.XXX.com. However I'm tempted to use a long established (5+ years) sub-domain with a topic related KW that now hosts a non-active blog with PR3 - employment-law.XXXX.com. It has 49 indexed pages, some with PR2. So there's potential for getting a kickstart on traffic and trust with some redirects. Should I go for instant gratification or build for the long haul with the slightly more beneficial URL KWing? I should add that this sub-domain will have thousands of pages that are geo and sub-category focused - a typical URL would be. sub-domain.XXX.com/TX/Austin/wrongful-termination-lawyers.html THX for any opinions.
Technical SEO | | legalseo0