Indexation of content from internal pages (registration) by Google
-
Hello,
we are having quite a big amount of content on internal pages which can only be accessed as a registered member.
What are the different options the get this content indexed by Google?
In certain cases we might be able to show a preview to visitors. In other cases this is not possible for legal reasons.
Somebody told me that there is an option to send the content of pages directly to google for indexation. Unfortunately he couldn't give me more details. I only know that this possible for URLs (sitemap). Is there really a possibility to do this for the entire content of a page without giving google access to crawl this page?
Thanks
Ben
-
The issue is that Google won't and shouldn't index pages that are restricted.
This is best for user experience. Most people won't sign in to view the content.
You basically have to create two sites. One that is visible to all, and Google where you show or preview a bit. then the other that is protected.
-
Thanks, I will check wether this meets the legal requirements (see my reply to Brents answer).
-
As I mentioned we have 2 cases.
In the first case, we can show a preview.
In the second case we can only show the content to a certain audience (which is a legal question). So the free registration is a legal requirement. Still people will be looking for it via Google. Since the content found on those pages is useful for a fairly large audience so why wouldn't we want Google to index the pages. Of course without Google knowing that there is relevant content on those pages, they will neither index nor propperly rank those pages.
-
I found this information for you but you should definitely check that it doesn't break any of Google's guidelines before incorporating it to your website.
This is a simple code to allow bots to bypass the password on password protected pages
$allow_inside = ($is_logged_in) || substr_count($_SERVER['HTTP_USER_AGENT'],'Googlebot');
http://davidwalsh.name/google-password-protected-areas
The reference post is older, so this code could have been updated
-
Fetch as Google Bot is submit to index. This is why I believe it should work with it.
-
I guess my questions is why would you want Google to index something that is only available to registered users?
In order for it to be indexed, it has to be open to everyone.
You will have to figure out what can be shown as a preview and what can't. If you want something to be indexed, then you will have to create a separate section for your preview content (since Google won't index your protected content.)
-
Hi Istvan,
"The Fetch as Googlebot tool lets you see a page as Googlebot sees it."
Since Googlebot has no access to the entire site (login required) it will probably not display anything (just tried it logged in and it would not display any of the content). How could this tool theoretically help us indexing the content of the internal page?
Ben
-
Hi Ben,
Maybe fetch as Google Bot can be a solution to your issue. But not 100% sure of this.
Gr.,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Dynamically Changing pages same content
Hey there Mozzers, I have a commerce site that is dynamically adding more products in the same page when you scroll down. I have added SEO Content on the footer of the page. The url is changing when you scroll to ?page-2, ?page-3 and so on. The content stays the same even though the page is dynamically changing. Is there a way to solve that issue? Should I always use canonical pointing to the initial page thus solving the duplication but indicate rel=next and rel=prev to the other pages etc? Thanks in advance
Intermediate & Advanced SEO | | AngelosS0 -
Pages with Duplicate Page Content (with and without www)
How can we resolve pages with duplicate page content? With and without www?
Intermediate & Advanced SEO | | directiq
Thanks in advance.0 -
Google de-indexed a page on my site
I have a site which is around 9 months old. For most search terms we rank fine (including top 3 rankings for competitive terms). Recently one of our pages has been fluctuating wildly in the rankings and has now disappeared altogether from the rankings for over 1 week. As a test I added a similar page to one of my other sites and it ranks fine. I've checked webmaster tools and there is nothing of note there. I'm not really sure what to do at this stage. Any advice would me much appreciated!
Intermediate & Advanced SEO | | deelo5550 -
Which is more valuable in a landing page, content or functionality?
I have two possible landing pages to focus off page links and paid ad links to, one page has space for content but basically only serves as a springboard to a map view style listing page. The idea is to use this page full of good content to build search engine value. The map view page is the most functional and is what visitors would ultimately be seeking, but has no real room for content. Are these content landing pages useful? Would it be better to focus on user functionality even though there is no space for content, and would search engines naturally apply for value to these pages? Are these landing pages necessary? The url's in question are http://www.rentcollegepads.com/marquette/search and http://www.rentcollegepads.com/marquette Thanks guys!
Intermediate & Advanced SEO | | Dom4410 -
No-index pages with duplicate content?
Hello, I have an e-commerce website selling about 20 000 different products. For the most used of those products, I created unique high quality content. The content has been written by a professional player that describes how and why those are useful which is of huge interest to buyers. It would cost too much to write that high quality content for 20 000 different products, but we still have to sell them. Therefore, our idea was to no-index the products that only have the same copy-paste descriptions all other websites have. Do you think it's better to do that or to just let everything indexed normally since we might get search traffic from those pages? Thanks a lot for your help!
Intermediate & Advanced SEO | | EndeR-0 -
Certain Product Pages Not Indexing
Hey All, We discovered an issue where new product pages on our site were not getting indexed because a "noindex" tag was inadvertently being added to section when those pages were created. We removed the noindex tag in late April and some of the pages that had not been previously indexed are now showing up, but others are still not getting indexed and I'd appreciate some help on why this could be. Here is an example of a page that was not in the index but is now showing after removal of noindex: http://www.cloud9living.com/san-diego/gaslamp-quarter-food-tour And here is an example of a page that is still not showing in the index: http://www.cloud9living.com/atlanta/race-a-ferrari UPDATE: The above page is now showing after I manually submitted it in WMT. I had previously submitted another page like a month ago and it was still not indexing so I thought the manual submission was a dead end. However, it just so happens that the above URL just had its Page Title and H1 updated to something more specific and less duplicative so I am currently running a test to see if that's the problem with these pages not indexing. Will update this soon. Any suggestions? Thanks!
Intermediate & Advanced SEO | | GManSEO0 -
Same content pages in different versions of Google - is it duplicate>
Here's my issue I have the same page twice for content but on different url for the country, for example: www.example.com/gb/page/ and www.example.com/us/page So one for USA and one for Great Britain. Or it could be a subdomain gb. or us. etc. Now is it duplicate content is US version indexes the page and UK indexes other page (same content different url), the UK search engine will only see the UK page and the US the us page, different urls but same content. Is this bad for the panda update? or does this get away with it? People suggest it is ok and good for localised search for an international website - im not so sure. Really appreciate advice.
Intermediate & Advanced SEO | | pauledwards0 -
How long until Sitemap pages index
I recently submitted an XML sitemap on Webmaster tools: http://www.uncommongoods.com/sitemap.xml Once Webmaster tools downloads it, how long do you typically have to wait until the pages index ?
Intermediate & Advanced SEO | | znotes0