Getting pages that load dynamically into the SE's
-
SEO'ers,
Am dealing with an issue I cannot figure out the best way to handle. Working on a website that shows the definitions of words which are loaded dynamically from an open source. Source such as: wiktionary.org
When you visit a particular page to see the definition of the word, say; www.example.com/dictionary/example/ the definition is there. However, how can we get all the definition pages to get indexed in search engines? The WordPress sitemap plugin is not picking up these pages to be added automatically - guess because it's dynamic - but when using a sitemap crawler pages are detected.
Can anybody give advice on how to go about getting the 200k+ pages indexed in the SE's? If it helps, here's a reference site that seems to load it's definitions dynamically and has succeeded in getting its pages indexed: http://www.encyclo.nl/begrip/sample
-
I see what you mean there - thanks for sharing your expertise and views on this issue. Much appreciated
-
The only way I'd let those pages be indexed is if they had unique content on them AND/OR provided value in other ways besides just providing the Wiki definition. There are many possibilities for doing this, none of them scalable in an automated fashion, IMHO.
You could take the top 20% of those pages (based on traffic, conversions, revenue...) and really customize them by adding your own definitions and elaborating on the origin of the word, etc... Beyond that you'd probably see a decline in ROI.
-
Everett, yes that's correct. I will go ahead and follow up on what you said. I do still wonder what the best way would be to go about getting it indexed - if I wanted to do that in the future. If you could shed some light on how to go about that, I'd really appreciate it. Thanks so much in advance!
-
It appears that your definitions are coming from wiktionary.org and are therefore duplicate content. If you were providing your own definitions I would say keep the pages indexable, but in this case I would recommend adding a noindex, follow robots meta tag to the html header of those pages.
-
Hi Everett, I've been looking at the index for word definitions and there's so many pages that are very similar to each other. It's worth giving it a shot I think. If you can provide feedback please do. Here's the domain: http://freewordfinder.com. The dictionary is an addition to users who'd like to see what a word means after they've found a word from random letters. You can do a search at the top to see the results, then click through to the definition of the word. Thanks in advance
-
Ron,
We could probably tell you how to get those pages indexed, but then we'd have to tell you how to get them removed from the index when Google sees them all as duplicate content with no added value. My advice is to keep them unindexed, but if you really want them to be indexed tell us the domain and I'll have a look at how it's working and provide some feedback.
-
Hi Keri, did you think that the site might get penalized because it would in essence be duplicate content from another site? Even though the source is linked from the page? Please let me know your thoughts when you can
-
No they currently do not have additional information on them. They are simply better organized on my pages compared to the 3rd party. The unique information is what drives visitors to the site and from those pages it links to the definitions just in case they're interested understanding the meaning of a word. Does that help?
-
Do the individual pages with the definitions have additional information on them, or are they just from a third party, with other parts of the site having the unique information?
-
Hi Keri, thanks for your response. Well, I see what you're saying. The pages that show the definition pulled from the 3rd party are actually supplementary to the solution the site provides (core value). Shouldn't that make a difference?
-
I've got a question back for you that's more of a meta question. Why would the search engines want to index your pages? If all the page is doing is grabbing information from another source, your site isn't offering any additional value to the users, and the search engine algos aren't going to see the point in sending you visitors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Doesn't Google Use My Title Tag and Meta Description?
Hi fellow Moz SEOs, Need your URGENT help! We set an optimised title & meta description for our client websites. These titles are approved by our clients. Before somedays, they checked on Google, noticed the title & meta description were not the same. Next moment, they notified me about this issue. The title & meta description looks fine when I checked the source code. So, why Google use title & meta description differently? For example: Title approved by client: Top Specialist Divorce & Family Lawyer - Yeo & Associates LLC
International SEO | | Verz
Google set our title: Yeo & Associates LLC: Top Specialist Divorce & Family Lawyer Title approved by client: Filing For Divorce Online in Singapore | DivorceBureau®
Google set our title: DivorceBureau®: Filing For Divorce Online in Singapore Title approved by client: Halal Buffet & Bento/Packet Meals Event Caterer Singapore | Foodtalks
Google set our title: Foodtalks - Halal Buffet & Bento/Packet Meals Event Caterer Singapore Title approved by client: Child Care Centre in Singapore| Top Preschool | Carpe Diem
Google set our title: Carpe Diem: Child care Centre in Singapore| Top Preschool Every day, they are requesting me to update Google's title with their approved title. Also, asking me these questions.
Why did this happen?
Why didn't set their recommended title? Is there any way to set our approved titles? Please, help me to find the solution. ASAP Thanks in advance!0 -
Important pages are being 302 redirected, then 301 redirected to support language versions. Is this affecting negatively the linking juice distribution of our domain?
Hi mozzers, Prior to my arrival, in order to support and better serve the international locations and offering multiple language versions of the same content the company decided to restructure its URLs focused on locale urls. We went from
International SEO | | Ty1986
https://example.com/subfolder to https://example.com/us/en-us/new-subfolder (US)
https://example.com/ca/en-us/new-subfolder (CAN)
https://example.com/ca/fr-ca/new-subfolder (CAN)
https://example.com/de/en-us/new-subfolder (Ger)
https://example.com/de/de-de/new-subfolder (Ger) This had implications on redirecting old URLs to new ones. All important URLs such as https://example.com/subfolder were
302 redirected to https://example.com/us/en-us/subfolder and then 301 redirected to the final URL. According to the devs: If you change the translation to the page or locale, then a 302 needs to happen so you see the same version of the page in German or French, then a 301 redirect happens from the legacy URL to the new version. If the 302 redirect was skipped, then you would only be able to one version/language of that page.
For instance:
http://example.com/subfolder/state/city --> 301 redirect to {LEGACY URL]
https://example.com/subfolder/state/city --> 302 redirect to
https://example.com/en-us/subfolder/state/city --> 301 redirect to
https://example.com/us/en-us/new-subfolder/city-state [NEW URL] I am wondering if these 302s are hurting our link juice distribution or that is completely fine since they all end up as a 301 redirect? Thanks.1 -
Which pages to put hreflang on?
Hi, we are running a site which is a directory consisting of numbers of phone spammers. It contains descriptions, comments and so on. We are currently present in 9 countries. The websites all have the same structure, but, of course, the spam numbers in each country are different ones. If I want to tell Google that our website is available is several locations/languages, do I only put my hreflang tag on the start page then? Thanks
International SEO | | Roverandom
Thomas0 -
Recent Google Link Scheme Updated ? What's Your Reaction against Link Building, Link Exchanging ?
Many Bloggers and Webmasters are upset over this !
International SEO | | Esaky
Recent Google Link Scheme Updated ? What's Your Reaction against Link Building, Link Exchanging ? https://support.google.com/webmasters/answer/66356?hl=en What will you Do, if we are good at traffic to our blog and advertiser link will be no-follow - will they accept it ! and guest post also. They need a do-follow link back to their blog or website they hired for !0 -
How do I get a UK website to rank in Dubai?
We are trying to get a UK-based children's furniture website to rank in Dubai. We have had a couple of orders from wealthy expats in Dubai and it seems to be the correct target market. Does anyone have any specific knowledge of this area? We are promoting the same website as for the UK market. Also does anyone know any user behaviour stats on expatriates using search engines? Do they carry on using the version of Google they are used to, or do most change to the local version of Google? Thanks in advance
International SEO | | Wagada0 -
Geotarget subfolders with the same language or get rid of duplicates but lose option to geotarget?
Hi, we have a domain that is aimed to cover LatAm region. Currently, the homepage contains country selector for ~20 countries. 95% of them hold content in Spanish. We have only homepages for each regions as separate subfolders, i.e.
International SEO | | eset
www.maindomain.com/co
www.maindomain.com/cl
www.maindomain.com/br
etc. but once the user clicks on menu item he is taken back to main domain subpages, i.e. www.maindomain.com/comprar My struggle is to decide whether it is better to: A) copy all content for each subfolder, which will create huge amount of duplicates (there are no resources to create unique content and it is even impossible taking into account nature of the product - mostly tech.specs, etc.) and implement hreflang sitemaps and configure GWT to target each country with its own Spanish content (the same for each country) OR B) remove all local subfolders and keep only main domain in Spanish that will serve all countries within the region. With this option, we will get rid of duplicates but also lose option to geotarget. So, my questions is which option will do less harm, or if there is any other approach that comes to your minds. I consulted with two agencies but still haven't got clear answer. Thanks a lot for your help!0 -
Are my translated pages damaging my ranking?
Hi there, I have a site in English but with duplicates in different languages. The first problem is that these translated versions of my site receive no ranking on google stars (while the english does) - why is this? The second problem is that SEOmoz counts the errors on my site and then duplicates this error count for all the translated versions of my site - meaning I have a huge amount of errors (too many on-page links). Add this to the fact that I use affilite ID´s to track different types of traffic to my site - so all page urls in english and other languages, with an affiliate id on the end of the url, count as an error. This means I have a huge amount of on page errors indicated by SEOmoz, plus no ranking for my translated pages - I think this is really harming my overall ranking and site trust. What are your opinions on this?
International SEO | | sparkit0 -
Alternate tag. Anybody had success getting English websites only with localized currency served with alternate tag?
I have an English website with USD prices and US phone.
International SEO | | lcourse
Via currency dropdown visitors in Ireland can choose EUR as currency, visitors from Denmark Danish crown etc and via GEO IP I also serve local contact phone numbers. So I though it made sense to define this with the alternate tags, but now after several months google still does not pickup these pages in local searches. Did anybody have success with getting a website just with currency parameter ranked locally using the alternate tag? Does it help to have also static links (not only dropdown links) to currency versions on the page? Any other thing that could help to have google pick these up? Below my code sample:0