URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looking for list Pro's & Con's of removing Folder from URL?
Hi We have a sub-folder ("/shop-by-department/") which is pretty much useless on our site and I'm looking to remove it. But the team want a list of the Pro's & Con's in doing so. So for example I'll be changing www.example.ie/shop-by-department/furniture/beds/product-a to www.example.ie/furniture/beds/product-a I know there will be an intial hit as Google adjusts to the change but think it's definitely the way to go. I was lookng for a complete list of the Pro's & Con's to send onto the team. It'll be going to the traditional marketing (print, radio, etc.) too so can ve top-level points too. Hope you can help! Thanks
Web Design | | Frankie-BTDublin0 -
What’s the best tool to visualize internal link structure and relationships between pages on a single site?
I‘d like to review the internal linking structure on my site. Is there a tool that can visualize the relationships between all of the pages within my site?
Web Design | | QBSEO0 -
Second Store URL
Our store has been up and running for about 18 months and has been more successful than expected. Unfortunately we have run into a few minor issues with customers wanting to pick up in store but much of the online parts we sell are drop ship. We have made the decision to open a second store. One will support our brand (using current URL)and reflect products and pricing we offer at our B&M locations. The second will continue down the path we have been going but under a completely different brand (and URL) in no way tied to our B&M stores. My question is this: Would it be smarter to re brand the store we have now and change the URL.?We would then create the second site as our corporate branded site. Or do we adjust the currently site and then create a second site with the new brand. The only real hold up is that the re branded store will generate far more revenue and the current site is optimized very well. However if i change the URL the optimization will go out the window .
Web Design | | Rillik0 -
Footer links on my site... bad for passing page rank?
i've been told that it is possible that google discounts the weight or page rank passed in footer links of websites and my website has the navigation to many of my pages in the footer of each page. My whole website is about 20 pages so each page has links to the 5 most popular pages at the top and the rest of the links are in the footer of each page. Am i losing page rank by having these links in the footer? Should i make my navigation different? I have lots of articles on my site so i thought it might be not only helpful to my readers but give my pages an seo boost if i placed in context links in the body of my articles to other pages of my site. Does this sound like a good idea? Thanks mozzers! Thanks mozzers!
Web Design | | Ron100 -
Live website is an addon domain - Need site old development url inaccessable from live domain
Hi everyone, I have a website which is built in Joomla 2.5. The development site is located at www,abc.com/subdomain/. We have set the site live using an addon domain which is www.xyz.com. The problem is, www.abc.com/subdomain/ is still accessible and being crawled by Google. How is the best way to make the development url inaccessible? Any help would be appreciated!
Web Design | | DougHosmer0 -
How to create Google Section or Jump To Links
Hello all! i need create a jump to links on my site and when seach a keyword on google it will display jump to links http://techwyse.com/blog/wp-content/uploads/2009/12/google-jump-to-links.jpg i same with that images please help me how to do it, and have any plugin on wordpress can do that google-jump-to-links.jpg
Web Design | | ITVSEO.COM0 -
Will changing our URL's to MVC friendly URL's have a positive or negative affect on our rankings and link juice?
We've recently changed our site over to a new hosting system, we've got similar pages and are now looking at changing the URL's to ensure we do not loose our link juice from our previous site. My question is regarding the URL's, is it worth us changing our URL's to MVC friendly URL have a good or bad affect on our rankings and or link juice? Thanks
Web Design | | SimonDixon0 -
Question About Site Redesign and Nav / Page Structure
Hey guys, i am currently redesigning our company's site, and have come across some things that I'm not quite sure of. We used to have individual service pages in our main navigation (design, video, marketing) before the redesign. In this new design, i had the idea of making just one "services" or "capabilities" page, where these three services would each be outlined, and each service would have a list of links to more specific landing pages. Obviously, breaking it up correctly with HTML5 using the andtags. What I'm wondering is that if i'm going to be penalized for having those three services that aren't necessarily related too closely on the same page as opposed to having the one page for each service (like we have now). Any help would be greatly appreciated, and let me know if i need to elaborate more. Thanks in advance!
Web Design | | RenderPerfect0