Getting pages that load dynamically into the SE's
-
SEO'ers,
Am dealing with an issue I cannot figure out the best way to handle. Working on a website that shows the definitions of words which are loaded dynamically from an open source. Source such as: wiktionary.org
When you visit a particular page to see the definition of the word, say; www.example.com/dictionary/example/ the definition is there. However, how can we get all the definition pages to get indexed in search engines? The WordPress sitemap plugin is not picking up these pages to be added automatically - guess because it's dynamic - but when using a sitemap crawler pages are detected.
Can anybody give advice on how to go about getting the 200k+ pages indexed in the SE's? If it helps, here's a reference site that seems to load it's definitions dynamically and has succeeded in getting its pages indexed: http://www.encyclo.nl/begrip/sample
-
I see what you mean there - thanks for sharing your expertise and views on this issue. Much appreciated
-
The only way I'd let those pages be indexed is if they had unique content on them AND/OR provided value in other ways besides just providing the Wiki definition. There are many possibilities for doing this, none of them scalable in an automated fashion, IMHO.
You could take the top 20% of those pages (based on traffic, conversions, revenue...) and really customize them by adding your own definitions and elaborating on the origin of the word, etc... Beyond that you'd probably see a decline in ROI.
-
Everett, yes that's correct. I will go ahead and follow up on what you said. I do still wonder what the best way would be to go about getting it indexed - if I wanted to do that in the future. If you could shed some light on how to go about that, I'd really appreciate it. Thanks so much in advance!
-
It appears that your definitions are coming from wiktionary.org and are therefore duplicate content. If you were providing your own definitions I would say keep the pages indexable, but in this case I would recommend adding a noindex, follow robots meta tag to the html header of those pages.
-
Hi Everett, I've been looking at the index for word definitions and there's so many pages that are very similar to each other. It's worth giving it a shot I think. If you can provide feedback please do. Here's the domain: http://freewordfinder.com. The dictionary is an addition to users who'd like to see what a word means after they've found a word from random letters. You can do a search at the top to see the results, then click through to the definition of the word. Thanks in advance
-
Ron,
We could probably tell you how to get those pages indexed, but then we'd have to tell you how to get them removed from the index when Google sees them all as duplicate content with no added value. My advice is to keep them unindexed, but if you really want them to be indexed tell us the domain and I'll have a look at how it's working and provide some feedback.
-
Hi Keri, did you think that the site might get penalized because it would in essence be duplicate content from another site? Even though the source is linked from the page? Please let me know your thoughts when you can
-
No they currently do not have additional information on them. They are simply better organized on my pages compared to the 3rd party. The unique information is what drives visitors to the site and from those pages it links to the definitions just in case they're interested understanding the meaning of a word. Does that help?
-
Do the individual pages with the definitions have additional information on them, or are they just from a third party, with other parts of the site having the unique information?
-
Hi Keri, thanks for your response. Well, I see what you're saying. The pages that show the definition pulled from the 3rd party are actually supplementary to the solution the site provides (core value). Shouldn't that make a difference?
-
I've got a question back for you that's more of a meta question. Why would the search engines want to index your pages? If all the page is doing is grabbing information from another source, your site isn't offering any additional value to the users, and the search engine algos aren't going to see the point in sending you visitors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Important pages are being 302 redirected, then 301 redirected to support language versions. Is this affecting negatively the linking juice distribution of our domain?
Hi mozzers, Prior to my arrival, in order to support and better serve the international locations and offering multiple language versions of the same content the company decided to restructure its URLs focused on locale urls. We went from
International SEO | | Ty1986
https://example.com/subfolder to https://example.com/us/en-us/new-subfolder (US)
https://example.com/ca/en-us/new-subfolder (CAN)
https://example.com/ca/fr-ca/new-subfolder (CAN)
https://example.com/de/en-us/new-subfolder (Ger)
https://example.com/de/de-de/new-subfolder (Ger) This had implications on redirecting old URLs to new ones. All important URLs such as https://example.com/subfolder were
302 redirected to https://example.com/us/en-us/subfolder and then 301 redirected to the final URL. According to the devs: If you change the translation to the page or locale, then a 302 needs to happen so you see the same version of the page in German or French, then a 301 redirect happens from the legacy URL to the new version. If the 302 redirect was skipped, then you would only be able to one version/language of that page.
For instance:
http://example.com/subfolder/state/city --> 301 redirect to {LEGACY URL]
https://example.com/subfolder/state/city --> 302 redirect to
https://example.com/en-us/subfolder/state/city --> 301 redirect to
https://example.com/us/en-us/new-subfolder/city-state [NEW URL] I am wondering if these 302s are hurting our link juice distribution or that is completely fine since they all end up as a 301 redirect? Thanks.1 -
Website relaunched: Both old pages and new pages indexed
Hi all, We have recently made major changes to our website and relaunched it. We have changed URLs of some pages. We have redirected old URLs to new before taking website live. When I check even after one week, still the same old and new pages also indexed at Google. I wonder why still old pages cache is there with Google. Please share your ideas on this. Thanks
International SEO | | vtmoz0 -
How To Proceed With Int'l Language Targeting if Subfolders Not An Option?
I’m currently working with my team to sort out the best way to build out the international versions of our website. Any advice on how to move forward is greatly appreciated! Current Setup: Subdirectories to target languages - i.e. domain.com/es/. We chose this because… We are targeting languages not countries Our product offering does not change from country to country Translated site content is almost identical to the english version Current Problem: Our site is built on WordPress and our database can’t handle the build out of 4 more international versions of the site. The database is slowing down and our site speed is being affected for multiple reasons (WordPress multilingual plugin being one of them). **What to do next? **My developers have said that we cannot continue with our current subdirectory structure due to the technical infrastructure issues I’ve mentioned above (as well as others I’m yet to get full details on). Now I’m left with a decision: Change to a subdomain structure Change to a ccTLD structure Is there an option 3? From what I’ve read it does not make sense to build out language targeted sites on a ccTLD structure because that limits the ability for people outside of the targeted country to find the content organically. I.e. a website at www.domain.es is targeted to searchers in Spain so someone in Columbia is less likely to find that content through the engines. Is this correct? If so, how much can it hurt organic discovery? What’s the optimal setup to move forward with in this case? Thanks!
International SEO | | UnbounceVan0 -
How well does Google's "Locale-aware crawling by Googlebot" work?
Hello, In January of this year Google introduced "Locale-aware crawling by Googlebot." https://support.google.com/webmasters/answer/6144055?hl=e Google uses different crawl settings for sites that cannot have separate URLs for each locale. ......... This is basically for sites that dynamically render contend on the same URL depending on the locale and language (IP) of the visitor. If e.g. a visitor was coming from France, the targeted page would load in french. If a visitor was coming from the US the same page would load in English on the same URL. Does anyone have any experience with this setup and how well it works? How well do the different versions of a page get indexed, and how well do those pages rank? In the example above, does the french content get indexed correctly? Many thanks!
International SEO | | Veva0 -
Optimizing for 3 international sites, how to avoid getting into trouble
Hi Guys As a newbie, I want to avoid any penalties or mistakes as possible that will be due to unknown and have taken some steps to educate myself around international sites and multiple domains. our aim was to target new zealand first and then branch out. Whilst we are pondering the NZ site and writing fresh unique articles for the site and the blog. And besides making the currency, language more relevant to these domains, is there anything else I could work on? I thought about making the meta tags different for the home page and adding Australia etc If we are going to spend time growing the site organically I thought I would make the most of spending the time growing all three together.... Any recommendations on how to get started and optimize the 3 alot better? Thanks
International SEO | | edward-may1 -
Pages with Duplicate Page Title
Blog - FDM Group has duplicate page title for all blog posts. We also have multiple localized versions of pages, so the titles are seen as duplicate. Possible resolutions? Thanks in advance.
International SEO | | fdmgroup0 -
Does Google take into account the place where the server is hosted to rank the pages.
Does Google take into account the place where the server is hosted to rank the pages. What I mean is, if I have a server in USA and I am working for the Spain marketplace: Will Google rank better my pages for this market if the server were hosted in Spain?
International SEO | | NorbertoMM0 -
Multilingual Ecommerce Product Pages Best Practices
Hi Mozzers, We have a marketplace with 20k+ products, most of which are written in English. At the same time we support several different languages. This changes the chrome of the site (nav, footer, help text, buttons, everything we control) but leaves all the products in their original language. This resulted in all kinds of duplicate content (pages, titles, descriptions) being detected by SEOMoz and GWT. After doing some research we implemented the on page rel="alternate" hreflang="x", seeing as our situation almost perfectly matched the first use case listed by Google on this page http://support.google.com/webmasters/bin/answer.py?hl=en&answer=189077. This ended up not helping at all. Google still reports duplicate titles and descriptions for thousands of products, months after setting this up. We are thinking about changing to the sitemap implementation rel="alternate" hreflang="X", but are not sure if this will work either. Other options we have considered include noindex or blocks with robots.txt when the product language is not the same as the site language. That way the feature is still open to users while removing the duplicate pages for Google. So I'm asking for input on best practice for getting Google to correctly recognize one product, with 6 different language views of that same product. Can anyone help? Examples: (Site in English, Product in English) http://website.com/products/product-72 (Site in Spanish, Product in English) http://website.com/es/products/product-72 (Site in German, Product in English) http://website.com/de/products/product-72 etc...
International SEO | | sedwards0