CMS dynamicly created pages indexed?
-
Hey Moz'erz,
Looking at the indexed pages of my clients eCommerce website I noticed that dynamically created pages are being indexed.
For example this page does not "exist" but is created by a drop down filter menu that sorts by product tag:
/collections/tools/TAG
I can only conclude that this page got indexed either through a backlink or once upon a time there was an internal link pointing to this URL and got indexed (currently there is not). Are either of these cases possibilities?
In either case before considering removal or any action I would of-course reference analytics to check for conversions, traffic and any backlinks for those "pages".
I believe at the end of the day is recommend a drop down filer that doesn't create new pages as the best solution.
Thoughts, comments and experience is greatly welcomed
-
Hey Dylan
Either of those are possibilities for Google finding and indexing a page like that. There could be many ways that happened - I've seen them spider "links" in a drop down depending on how it's implemented.
One thing you can do to check how, is looked at the text-only cache of the page (type cache:www.domain.com/page-name in your browser and click text only) - and look to see if the drop down items actually appear and clickable links. You can also try crawling the site with Screaming Frog and set the user-agent to GoogleBot and see if they got picked up.
If the filter is just for example re-sorting the list of items in a category, there is probably not a need to have this crawled or indexed, because it's just the same content in a different order.
If you do want to remove them from the index, you will want to add a meta noindex tag to the HTML, wait for them to drop out of the index, and then block crawling with robots.txt or nofollow the links that might be generated.
Hope that helps!
EDIT - I'd also check to be sure they are not showing up in your XML sitemap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
A question about title tag when the page has 2 services.
Hi all, Assuming a company has two services: SEO and PPC. Here is the situation: I would like to focus on SEO for now but also don't want to leave my PPC service out of the page. SEO accounts for 60% of the content, while PPC accounts for 40%. Assuming the content (SEO + PPC) of the page will not change, which title tag would you prefer, and why? SEO | brand name (Is it appropriate that the title focus on SEO but the content of the page contains PPC) SEO | PPC | brand name (Will the keywords dilute each other?) SEO | SEM Agency | brand name (The idea behind it is that SEM includes SEO and PPC so I think Google would be OK with the page ranking for SEO and also including PPC in the content. I really appreciate your help and explanation. Thank you!
Web Design | | Raymondlee0 -
Pushstate and Infinite Scrolling Article Pages: Is it detrimental to not change URLs as the page is being scrolled?
I've noticed a recent trend of news sites using infinite scrolling on article pages to garner more pageviews and I can assume serve up more ads. Here is an overview. Here is an article from NBC news that uses this technique: http://www.nbcnews.com/pop-culture/music/grammys-2016-here-s-why-adele-s-performance-was-out-n519186 Studies have shown that this technique has decreased bounce rates by +15% for some sites. My question is: If a site is using the technique without changing URLs as the user scrolls down what overall negative effects does this have? Obviously you wouldn't be getting credit for the extra pageviews but I was wondering if there were any indexation implications with this. Here is an example of article infinite scrolling without changing the URL: http://www.wftv.com/news/national-content/deputies-wife-attacks-husband-because-he-didnt-get-her-a-valentines-day-gift/87691927
Web Design | | Cox-Media-Group1 -
Bing Indexation and handling of X-ROBOTS tag or AngularJS
Hi MozCommunity, I have been tearing my hair out trying to figure out why BING wont index a test site we're running. We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
Web Design | | AU-SEO
This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing. We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site. In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header. With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag. However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive. I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing. I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages. I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content. Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages. Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl. Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests? Thanks in advance for your assistance....0 -
How to make sure category pages rank higher than product pages?
Hi, This question is E-Commerce related. We have product categories dividing products by color. Let's say we have the category 'blue toy cars' and a product called 'blue toy car racer', both of these could rank for the keyword 'blue toy car'. How do we make sure the category 'blue toy cars' ranks above the product 'blue toy car racer'? Or is the category page automatically ranked higher because of the higher page authority of that page? Alex
Web Design | | WebmasterAlex0 -
Location specific services pages
Hi guys, I'm working with a client who would has offices in two cities in Australia, and they provide a certain service to both cities. They would like to rank for keyword phrase for the service for both cities. E.g. Window Cleaning Brisbane, Window Cleaning Darwin. I strongly believe in focusing on relevance and visitor experience first, rather than ham-fistedly trying to rank for those phrases. Having said that, I'm thinking of creating two pages for those phrases as a sub-page to the service itself, with the title of those pages containing the geographically specific phrases. E.g.: Window Cleaning -->Window Cleaning Brisbane -->Window Cleaning Darwin The pages would highlight the 'reach' of the service in the two cities, as well as some specific information such as the history of both offices, any distinctions between the services provided, the teams at both locations, and so forth. I feel that although this seems like a valid reason for doing this, I may be overlooking something. What do you guys think?
Web Design | | carlod0 -
What is the best tool to view your page as Googlebot?
Our site was done with asp.net and a lot of scripting. I want to see what Google can see and what it can't. What is the best tool that duplicates Googlebot? I have found several but they seem old or inaccurate.
Web Design | | EcommerceSite0 -
Landing Page - is this one a good landing page?
Hello Everyone, I want to ask about this landing page: http://www.rpgdicas.com.br/builds/diablo-2/build-necromancer-summoner-diablo-2.html It's in portuguese-brazil. What should I improve? Any tipps will be apreciated keywords: Build Necromancer Summoner, build necromancer summoner diablo 2, so on... thanks, everyone.
Web Design | | augustos0 -
Is it necessary to redirect every Error page (404 or 500) found?
If I have Hundreds of pages with 404 and 500 erros should set up 301 redirects for all of them? Some of the pages have external links, some don't.
Web Design | | jmansd0