How to detect where Google gets indexed URL's
-
Google index some kind of way some links that create duplicate content. We doesn't understand how these are created so we would like detect where Google robots find these links.
We tried:
- Moz Crawl Diagnostics but it shows 0 as Internal Link Count for these kind of links.
- Find some information from Google Analytics, that maybe there is trace (site content - all content) from visitors side. There wan't.
- We tried to find some information in Webmaster Tools under Internal link and HTML Improvements but didn't find any trace.
- Tried some search commands. Is there maybe some good one to search.
- TO search URL's form code with https://search.nerdydata.com.
-
It really isn't possible for an outsider to know why your website is generating those URLs in error; you would have to talk to your developer about that.
As far as canonicals, if your problem is page.com is getting duplicated by added parameters: page.com/?id=1, page.com/?id=2, page.com/?id=3, etc. as long as you have the canonical on page.com, all of the parameter pages will have the correct canonical on them as well. (But you are right, you should track down the source; your developer will know.)
-
Thanks you for your answer but yes I know that these are generated by our site. But problem is that I can use canonical tag for these that are indexed right now but later new ones will be created someway. Problem root isn't that we doesn't know how to use canonical, it's how to get to know where these URL's are find/indexed/detected by Google.
These kind of URL's have been there for months so we can't just hope that somehow these will be droped. We need to find some kind of solution and detect real problem.
-
If you found those URLs by doing a site: search, then those parameters are being generated by your site. (I am surprised that Google is even indexing them; I assume that pretty soon all but one will be dropped.) Here is an article that explains more about those types of duplicate pages: http://moz.com/blog/which-page-is-canonical
You can fix this by using a canonical tag on your homepage with the version that doesn't have the parameter.
-
Our front page has almost 50 duplicate versions. These are shown when we do site:oursite.com, there are /et?id=xx, /et?productId=xx, etc. In URL xx are different numbers.
-
Where are you seeing these duplicate content links? Does Webmaster Tools say that they are duplicate content? Or does this show up in your Moz crawl? What do these URLs look like?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Submitted URL marked 'noindex'
Search Console is giving this issue for near 100 pages of my website. I have checked the Yoast plugin settings. We haven't used any meta robots tag for these pages, neither have these pages been disallowed in robots.txt. Previosuly this issue was for some 20+ pages. I tried to reindex them by submitting the urls again. Now the count has risen to 100+. There is also this "Submitted URL blocked by robots.txt" issue for pages which are NOT disallowed in robots.txt. Can anyone please suggest me a solution here?
Reporting & Analytics | | Reema240 -
AMP Session Stitching - How to deal with Google's Client ID AMP Policy
Hello, I recently attended SMX East and the concept of 'session stitching' for AMP was brought up (https://www.stonetemple.com/amp-tech-guide/). I reached out to my development team and they told me they could do it, but that we would need to agree to the new TOS changes and making users aware of then... https://support.google.com/analytics/answer/7486055 Has anyone here done something like this? And if so how did you deal with the Google's Client ID AMP policy? Thank you all! -Margarita
Reporting & Analytics | | MargaritaS1 -
Is having the same URL in several sitemaps a problem for google?
We have 30 sitemaps, one for each language version of our site. About 5000 pages per sitemap.
Reporting & Analytics | | lcourse
To get a better idea on which pages google is not indexing, I thought about quickly generating sitemaps by page cagetories to see if there are any patterns. Any problems if I submit now new additional sitemaps dividing all our pages by product page, considering that the same pages are already in our existing sitemaps we submitted in the search console. So having same URL in more than 1 sitemap would be a problem? As a side note, we observed when adding a sitemap index that google search console in its count of total indexed pages, now counts every page twice since we submitted both the sitemap index and the individual sitemaps, so search console does not recognize in count that sitemaps in sitemaps index are identical to the ones we submitted individually in search console.0 -
Hey guys, any advice on why my 'search analytics' in Google Webmaster tools is blank with no data?
I have added all the website versions into Google web master tools and I have no crawl errors. When I click on Search traffic these pages are blank - search analytics, & Mobile usability. And When I fetch as Google the status is constantly saying me 'temporarily unreachable' - any help would be greatly appreciated.
Reporting & Analytics | | HLAS0 -
Moving data between Google Analytics Properties
Last summer we setup another Google Analytics property for us with Universal Analytics and have been running this alongside the old Google Analtyics property. is there a way of exporting all the old data from the old property into the new Universal Analytics property?
Reporting & Analytics | | ese0 -
'Search Queries Report' in Webmaster Tools Question
Hi, How much do you use the search queries report in webmaster tools to research current rankings/movements? It does look like a great tool but the data doesn't seem to be spot on. For example a keyword over a week might have flux in position so lets say 6.0 then 9.2 for 3 days then back to 6.0. But I check the serp's for this keyword everyday and didn't see any movement?!?! Is this a good tool for you?
Reporting & Analytics | | activitysuper0 -
Google Analytics Reports
Hi all , I do not have the skill and time to look into Google Analytics to derive actionable conclusion and inferences. I would like to outsource this activity , just like I outsource link building. But not sure how... Any guidance ?
Reporting & Analytics | | iamnew0 -
Has Google changed its algorithm? My traffic has almost doubled and I don't know why.
My traffic has nearly doubled over the past few days, and my conversion rate has doubled as well. It looks like our rankings haven't changed... We haven't done anything to the site recently, although we did submit a couple of press releases through the wire not too long ago (within the last few weeks). I do not think that this is a cyclical/seasonal jump in traffic because last year this did not occur. Any ideas?
Reporting & Analytics | | deuce1s0