Why are new pages not being indexed, and old pages (now in robots.txt) remain in the index?
-
I currently have a site that was recently restructured, causing much of its content to be reposted, creating new URL's for each page.
To avoid duplicates, all of the existing pages were added to the robots file. That said, it has now been over a week - I know Google has recrawled the site - and when I search for term X, it is stil the old page that is ranking, with the new one nowhere to be seen. I'm assuming it's a cached version, but why are so many of the old pages still appearing in the index?
Furthermore, all "tags" pages (it's a Q&A site, like this one) were also added to the robots a few months ago, yet I think they are all still appearing in the index. Anyone got any ideas about why this is happening, and how I can get my new pages indexed?
-
Is there a way for the search engines to find the new content? If you're blocking the old content from being crawled, even if there are redirects, it makes it that much harder for the new URLs to be found. Also, do you have a sitemap with the new URLs?
-
SE's dont take notive first crawl, they need to see resulst a few times before they take notive, this i think is make sure thet you mean what the see, and not just a mistake.
I would not exclude any pages with robots, this is a very crude way of doing things. if the page does not exists then it will drop out of index, if it does exist, then you would lose link juice thought all the links that point to those pages. If you have to no-index a page, use the meta tag "no-index,follow" so that links are still flollowed back out of the page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Getting rid of duplicate content remaining from old misconfiguration
Hi Friends,We have recently (about a month ago) launched a new website, and during the review of that site spotted a serious misconfiguration of our old terrible WP siteThis misconfiguration, which may have come from either sitemaps or internal links or both lead to displaying our french german and english sites on each others’ domains. This should be solved now, but they still show in SERPS: The big question is: What’s the best way to safely remove those from SERPS?We haven’t performed as well as we wanted for a while and we believe this could be one of the issues:Try to search for instance“site:pissup.de stag do -junggesellenabschied” to find english pages on our german domain, each link showing either 301 or 404.This was cleaned to show 301 or 404 when we launched our new site 4 weeks ago, but I can still see the results in SERPS, so I assume they still count negatively?Cheers!
Intermediate & Advanced SEO | | pissuptours0 -
Robots.txt question
I notice something weird in Google robots. txt tester I have this line Disallow: display= in my robots.text but whatever URL I give to test it says blocked and shows this line in robots.text for example this line is to block pages like http://www.abc.com/lamps/floorlamps?display=table but if I test http://www.abc.com/lamps/floorlamps or any page it shows as blocked due to Disallow: display= am I doing something wrong or Google is just acting strange? I don't think pages with no display= are blocked in real.
Intermediate & Advanced SEO | | rbai0 -
New Website Launch - what to do with the URLs of the pages with ranks
Hey there, So, we are "redesigning" our website, it will have a new user journey and overall layout, use, and feel. Situation: Previously, most of our keywords ranked over time organically though all of them pull up our domain.com as the landing page. Now that we are redesigning the site, most of the keywords pointing to the home page will now have their own page. Keywords properly grouped and content will now be on topic and focused per page. Q: What are the things that we need to do so we won't lose those keywords? Appreciate your help. Also, if you can cite specific SEO checklist when redesigning a site, that'll be a great help! Thanks! Jac
Intermediate & Advanced SEO | | jac.reyes0 -
Does it make sense to create new pages with friendlier URLs then redirect old pages to new?
Hi Moz! My client has messy URLs. does it make sense to write new clean URLs, then 301 redirect all old URLs to the new ones? Thanks for reading!
Intermediate & Advanced SEO | | DA20130 -
Baidu Spider appearing on robots.txt
Hi, I'm not too sure what to do about this or what to think of it. This magically appeared in my companies robots.txt file (literally magically appeared/text is below) User-agent: Baiduspider
Intermediate & Advanced SEO | | IceIcebaby
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: / I know that Baidu is the Google of China, but I'm not sure why this would appear in our robots.txt all of a sudden. Should I be worried about a hack? Also, would I want to disallow Baidu from crawling my companies website? Thanks for your help,
-Reed0 -
Index or not index Categories
We are using Yoast Seo plugin. On the main menu we have only categories which has consist of posts and one page. We have category with villas, category with villa hotels etc. Initially we set to index and include in the sitemap posts and excluded categories, but I guess it was not correct. Would be a better way to index and include categories in the sitemap and exclude the posts in order to avoid the duplicate? It somehow does not make sense for me, If the posts are excluded and the categories included, will not then be the categories empty for google? I guess I will get crazy of this. Somebody has perhaps more experiences with this?
Intermediate & Advanced SEO | | Rebeca10 -
Is it worth redirecting an old domain name which was hacked to my new website?
I had a website which got hacked and malware added to it. I have since closed that website down but I still have the domain name. That domain name prior to the malware was incredibly well ranking for its niche and had a good range of high quality links to it and a domain age of 6 years. I'm now creating a new website which is similar to the old one (the same but with a different platform and layout). Is it a good or bad idea to redirect the old domain name to the new website?
Intermediate & Advanced SEO | | james.rose0 -
New FB page feature LOOKS AWESOME how to get it for me too??
So I was commenting on an article that my girlfriend shared and the coolest thing happened. Check out the screens for details, but basically, as soon as i finished my comment and pressed enter, a little like box prompt window popped up. This was on FB not on their site, and I thought that was awesome. Anyone heard of, or no what is required to get this for other pages??? Thanks!! Tyler Abernethy 0KyS0wQ.png?1
Intermediate & Advanced SEO | | TylerAbernethy0