URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does redirecting the existing URLs in the website without reducing our current rankings? The new website runs on the bubble, so it seems there is no provisions to redirect the existing URLs to this platform?
Hi Moz Fans, There are some clarification needed in a website revamping and loosing of current website rank. Please go through the questions and would be great if you like to share some insights on it. 1. We would like to revamp our existing website by joining hands with the bubble visual programming platform. Thus, kindly let us know if there are provisions to redirect the existing URLs to this platform. We would also like to know if this kind of redirecting affects the current website ranking. If yes, how can we redirect the existing URLs in bubble without reducing our current rankings? 2. As a part of the revamping of our website, we would like to enquire about the possibilities of its relaunch via bubble. Does it cause any changes for the current rankings of our website if we redirect the existing URLs via bubble? If yes, is there any provision to redirect the URLs without affecting the current ranking of the website?
Web Design | | OceanAirTravels0 -
Site move?
What happens in a site move from a subdomain to a new domain and how does that effect the root domain of the subdomain and whether or not the subdomain SEO would be transferred to the new domain?
Web Design | | Jjjay1 -
URL Designing and Site Architecture
Hi, 1. At the end of the URL, should we have "/" or not (for e.g. my website www.example.com/abc/) 2. What is the optimum level of site depth 3. Google crawler will go through the breadcrumb or url (If a page is at 6th level through breadcrumb and at 4th level through URL, at which level will google crawler recognize this.
Web Design | | adiez12340 -
How to change the entire contents and design in my site without getting troubles with google?
Hello everyone This is my first post over here. In the next few weeks we going to change the entire content and design in our site. The site has 240 pages with poor contents and design. Except 301 redirects for all the old url’s I wanted to consult with you what is the right way to do it without harm my organic traffic that come from google? How google refers to this kind of changes? Which steps should I need to take to do it properly? Hope to get your help in the issue. Tahnks in advance.
Web Design | | JonsonSwartz0 -
Time On Site and SEO?
Does time on site impact rankings? If a person visits your site from the serps or directly visits it by typing in your name in the search field and then leaves within a minute, will that impact your serps? What is the best way to increase time on site?
Web Design | | bronxpad0 -
Question About Site Redesign and Nav / Page Structure
Hey guys, i am currently redesigning our company's site, and have come across some things that I'm not quite sure of. We used to have individual service pages in our main navigation (design, video, marketing) before the redesign. In this new design, i had the idea of making just one "services" or "capabilities" page, where these three services would each be outlined, and each service would have a list of links to more specific landing pages. Obviously, breaking it up correctly with HTML5 using the andtags. What I'm wondering is that if i'm going to be penalized for having those three services that aren't necessarily related too closely on the same page as opposed to having the one page for each service (like we have now). Any help would be greatly appreciated, and let me know if i need to elaborate more. Thanks in advance!
Web Design | | RenderPerfect0 -
How can we improve our e-commerce site architecture to help best preserve Page Authority?
Today I installed the SEOMoz toolbar for Firefox (very cool, highly recommended). I was comparing our site http://www.ccisolutions.com to this competitor: http://www.uniquesquared.com For the most part, the deeper I go in our site the more the page authority drops. We have a few exceptions where the page authority of a subcategory page is actually better than the cat. page one level up. In comparison, when I was looking at http://www.uniquesquared.com I noticed that their page authority stays at "21" on every single category page I visit. Are you seeing what I'm seeing? Is this potentially a problem with the tool bar or, is there something significantly different about their site architecture that allows them to maintain that PA across all category and sub category pages? Is there something fundamentally wrong with our (http://www.ccisolutions.com) site architecture? I understand that we have longer URLs, but this is an old store with a lot of SKUs, so we have decided not to remove the /category/ and /product/ from the URLs because the 301 redirects that would result wouldn't pass all of the authority they've built up over the years. Interested to know viewpoints on the site architecture and how it might be improved. Thanks!
Web Design | | danatanseo0 -
Google Analtyics Conversion Tracking for Wordpress Life Coaching Site
Hello, How do I do conversion tracking for Google Analytics for this site: debidonner(dot)com She has a 'Thank You' page after you return from Paypal Thanks!
Web Design | | BobGW0