Prevent indexing of dynamic content

[SCW]

Hi folks!

I discovered bit of an issue with a client's site. Primarily, the site consists of static html pages, however, within one page (a car photo gallery), a line of php coding:

dynamically generates a 100 or so pages comprising the photo gallery - all with the same page title and meta description. The photo gallery script resides in the /gallery  folder, which I attempted to block via robots.txt - to no avail. My next step will be to include a:

within the head section of the html page, but I am wondering if this will stop the bots dead in their tracks or will they still be able to pick-up on the pages generated by the call to the php script residing a bit further down on the page?

Dino

[SCW]

Hello Steven,

Thank you for providing another perspective. However, all factors considered, I agree with Shane's approach on this one. The pages add very little merit to the site and exist primarily to provide the site users with eye-candy (e.g. photos of classic cars).

[Jinx14678]

Just personally, I would still deindex or canonical them - they are just pages with a few images - so not of much value and unless all titles and descriptions are targeting varying keywords and content is added, they will canabalize eachother, and possibly even drag down the site due to 100's of pages of thin content....

So actually from an SEO perspective it probably IS better to deindex or canonical 3 - 5 or so years ago, maybe the advice would have been keep them and keyword target - but not in the age of content

(unless the images were optimized for image searches for sale able products (but I do not think it is)

[StevenHowe]

Hi Dino,

I know this won't solve the immediate problem you asked for, but wouldn't it be better for your client's site (and for SEO) to alter the PHP so that the title and meta data description are replaced with variables that can also be dynamic, depending on whichever of the 100 or so pages gets created?

That way, rather than worrying about a robot seeing 100 pages as duplicate content, it could see 100 pages as 100 pages.

[Jinx14678]

It depends on how the pages are being created (I would assume it is off of a template page)

So within the template of this dynamically created page you would place

But if this is the global template - you cannot do this as it will noindex every page which of course is bad.

If you want to PM me the URL of the page I can take a look at your code, and see what is going on and how to recitify, as right now i think we are talking about the same principles, but different words are being used.

It really is pretty straightforward. (what I am saying) The pages that you want to be not indexed DO NOT need a nofollow they need a meta noindex

But there are many variables, as if you have already robot.txt disallowed the directory, then no bot will go there to get the updated noindex directive....

If there is no way to add a meta noindex then you need to nofollow and put in for a manual removal

[SCW]

I completely understand and agree with all points you have conveyed. However, I am not certain as to the best approach to "noindex" the urls which are being created dynamically from within the static html page? Maybe I am making this more complex than it needs to be...

[Jinx14678]

So it is the pages themselves that are dynamically created you want out of index, not the page the contains the links?

If this is so ---

noindex the pages that are created dynamically

Therein lies the problem. I did have the nofollow directive in place specifying the /gallery/ folder, but apparently, the bots still crawled it.

Nofollow does not remove from index, it only tells the bot not to pass authority, as it is still feasible that the bot will crawl the link, so without the noindex, nofollow is not the correct directive due to the page (even though nofollowed) is still being reached and indexed.

PS. also if you have the nofollow on the links, you may want to remove it, so the bots will go straight through to the page and grab the noindex directive, but if you wanted to try to not let any authority "evaporate" you can continue to nofollow, but you may need to request the dynamically generated pages (URLS) be removed using webmaster tools.

[SCW]

The goal is to have the page remain in the index, but not follow any dynamically generated links on the page. The nofollow directive (in place for months) has not done the job.

[Jinx14678]

?

If a link is coming into the page, and you have Noindex, Nofollow - this would remove from index and prevent the following of any links -

This is NOT instant, and can take months to occur depending on depth of page, crawl schedule ect... (you can try to speed it up by using webmaster tools to remove the URL)

What is the goal You are attempting to achieve?

To get the page out of index, but still followed?

Or remain in index, but just not follow links on page?

?

[SCW]

Therein lies the problem. I did have the nofollow directive in place specifying the /gallery/ folder, but apparently, the bots still crawled it. I agree that the noindex removes the page, but I wasn't certain if it prevented crawling of the page, as I have read mixed opinions on this.

I just thought of something else... perhaps an external url is linking to this page - allowing it to be crawled. I am off to examine this perspective.

Thanks for your response!

[Jinx14678]

noindex will only remove from Index and dissallow the act of indexing the specific page (or pages created off template) you place the tag in upon the next page crawl.

Bots will still follow the page, and follow any links that are readable as long as there is not a nofollow directive.

I am not sure I fully understand the situation, so I would not say this is my "reccomendation" but an answer to the specific question.....

but I am wondering if this will stop the bots dead in their tracks or will they still be able to pick-up on the pages generated

Hope this helps!