How did my dev site end up in the search results?
-
We use a subdomain for our dev site. I never thought anything of it because the only way you can reach the dev site is through a vpn. Google has somehow indexed it. Any ideas on how that happened? I am adding the noindex tag, should I used canonical? Or is there anything else you can think of?
-
Personally, I'd still recommend using robots.txt to disallow all crawlers, even if more steps are taken.
-
Don't use tool removal, it can go bad indeed. Now, are you sure that there are no external links coming from anywhere?
For now I'd recommend putting noindex, nofollow on that dev subdomain and do manual recrawl through GWT.
-
It just uses internal links. Do you think I should try the webmaster tools removal? That seems like it could go wrong.
-
I never used screaming frog, does it check both external and internal links?
-
I have ran screaming frog to see if there are any links to any pages and but couldn't see any. Even if Google did try to follow it the firewall would stop them. It is so strange.
-
Then my first assumption is that it's linked from somewhere - read my comment a little above.
-
Then there is a leak somewhere - Google bots can "see" your subdomain.
Or it's been simply linked from somewhere. Then Google will try to follow the link and that would make it indexed.
-
They are telling me that there are no holes, and I have tried getting to the pages but can not do it unless I am on my vpn.
-
We never updated the robots.txt because the site was behind a firewall. If you click on any of the results it will not load the page unless on my VPN.
-
Robots.txt won't help anyhow. Bots still can see that there is such directory, they just won't see what's inside of those directories/subdomains.
-
Hi there.
If what you say is true, then there are only two answers: you got a leak somewhere or your settings/configuration is messed up.I'd say go talk to your system admin and make sure that everything what's supposed to be closed is closed, IPs, which are supposed to be open for use are open and those IPs only.
-
Have you updated the dev sites robots.txt to disallow everything? It is up to the bot to listen, but that combined with removing all of the dev URLs from Google Webmaster tools should do the trick.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What to try when Google excludes your URL only from high-traffic search terms and results?
We have a high authority blog post (high PA) that used to rank for several high-traffic terms. Right now the post continues to rank high for variations of the high-traffic terms (e.g keyword + " free", keyword + " discussion") but the URL has been completed excluded from the money terms with alternative URLs of the domain ranking on positions 50+. There is no manual penalty in place or a DCMA exclusion. What are some of the things ppl would try here? Some of the things I can think of: - Remove keyword terms in article - Change the URL and do a 301 redirect - Duplicate the POST under new URL, 302 redirect from old blog post, and repoint links as much as you have control - Refresh content including timestamps - Remove potentially bad neighborhood links etc Has anyone seen the behavior above for their articles? Are there any recommendations? /PP
Intermediate & Advanced SEO | | ppseo800 -
Implications from portfolio site
I'm looking for a bit of advice regarding links coming into main site from another site in the client portfolio. The main site we are working on has been going great, organic traffic has grown considerably. The past few weeks there has been a subtle decline including ranking for a few keywords down a little. What I have noticed is that there is another site in the portfolio (that I am not working on) has had a steady tailspin in organic traffic since Jan and i've been informed it is a dying site in terms of the products offered. This has some links in the main menu going directly to the main site. My gut feeling is to isolate the secondary site from the main (no-follow or remove links), but the impact on slightly dropped rankings on the main site is not directly related to those linked pages. Would you go for it and isolate anyway?
Intermediate & Advanced SEO | | MickEdwards0 -
Empty search results labeled as Soft 404s?
I have a site with faceted search but sometimes when someone drills down too far it ends up with no results. The page and outlined and faceted navigation are still there. The site uses dynamic URLs for the faceted navigation but Google is reporting these no results pages as Soft 404s. How should we handle these? Should we redirect these? Can we return 404 in the status code but still show the no results page they are looking for? Thanks for your responses
Intermediate & Advanced SEO | | MarloSchneider0 -
Need your thoughts on my site
Hi, This is my site: http://hemorrhoidssuccess.com/ I have got some decent natural links with a mix of different anchor texts. My main keyword is "Hemorrhoids Treatment", And i got very less exact match anchor texts. Now i was able to rank for "Top Hemorrhoid Treatment" on #2 Page, but i was not in the index for the my main keyword "Hemorrhoids Treatment". Can you review my site and let me know, what i am missing? Do i need to get more links? If so with what anchor texts? Will be waiting for your replies.. Thanks in Advance
Intermediate & Advanced SEO | | Vegitss
Dhee0 -
How to Hide Directories in Search?
I noticed bad 404 error links in Google Webmaster Tools and they were pointing to directories that do not have an actual page, but hold information. Ex: there are links pointing to our PDF folder which holds all of our pdf documents. If i type in , example.com/pdf/ it brings up a unformated webpage that displays all of our PDF links. How do I prevent this from happening. Right now I am blocking these in my robots.txt file, but if i type them in, they still appear. Or should I not worry about this?
Intermediate & Advanced SEO | | hfranz0 -
Franchise sites on subdomains
I've been asked by a client to optimise a a webpage for a location i.e. London. Turns out that the location is actually a franchise of the main company. When the company launch a new franchise, so far they have simply added a new page to the main site, for example: mysite.co.uk/sub-folder/london They have so far done this for 10 or so franchises and task someone with optimising that page for their main keyword + location. I think I know the answer to this, but would like to get a back up / additional info on it in terms of ranking / seo benefits. I am going to suggest the idea of using a subdomain for each location, example: london.mysite.co.uk Would this be the correct approach. If you think yes, why? Many thanks,
Intermediate & Advanced SEO | | Webrevolve0 -
Why do I have better results with absolute search
Hi, I completed the optimization of my website, and while my rankings have improved, i seem to have consistently better resultats with absolute search terms: ex: i'm better ranked on "word1 word2" than just word1 word2 without the guillemets. Do you have any idea why? As most people perform search without guillemets, i would like to improve my rankings with these searches too. thanks, cedric
Intermediate & Advanced SEO | | smartgrains0 -
Building a mobile site.
We are building a mobile site that will be launching in another month. I’m concerned that the mobile site will start catabolizing our traditional rankings. Is there a way to keep this from happening? Should we utilize the cross domain canonical tag and point back to the traditional site URLs?
Intermediate & Advanced SEO | | SEO-Team0