URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site redesign - 301 Redirects
I've just overhauled a website, leaving lots of former posts in the dust. I've set up a 404 redirect to the home page so that if anyone goes to one of those old pages they land on the home page instead of a dead 404. But, there's a couple urls from the old site I'd prefer to redirect to similar pages. These urls have forward slashes and I don't know how to get the slashes in when I copy it over to the new site. This is probably something easy, but I'm baffled. This www.lawbarron.com/personal-injury/whip-its-nitrous-oxide/ becomes this when I copy it www.lawbarron.com/personal-injury-whip-its-nitrous-oxide Can someone help me out?
Web Design | | julie-getonthemap0 -
Why aren't Images in G+ product page posts showing up in SERPs for brand searches?
Before 1-2 weeks ago, our G+ posts containing links to our product pages would show up in in SERPs (when searching for our brand name) with a thumbnail of the product image. Now, they do not (see image below for visual). Our tech team confirmed there hasn't been any coding change that might be to blame and I see that this isn't happening to other sites. Any idea what may be the problem here? tcnhLgy
Web Design | | znotes0 -
Website Drops Some Traffic after Redesign. What's Happening?
What it is NOT: No Link was broken. I have used Moz, Screaming Frog, Excel, etc - there are not broken links. We have not added spammy links. We kept the same amount of links and content on the homepage - with an exception of 1 or 2. All the pages remained canonical. Our blog uses rel=prev rel=next, and each page is canonicalized to itself. We do not index duplicated content. Our tags are content="noindex,follow" We are using the Genesis Framework (we were not before.) Load time is quicker - we now have a dedicated server. Webmaster tools has not reported any crawl report problems. What we did that should have improved our rankings and traffic: Implemented schema.org Responsive design Our bounce rate is down - Average visit length is up. Any ideas?
Web Design | | Thriveworks-Counseling0 -
Is WP okay for E commerce sites?
Do any of you out there use wordpress for an ecommerce site? I'm getting some mixed reviews on it (but it's the internet, so that's bound to happen). Is there any sort of site traffic or page limit that would make using wordpress a bad idea? Thanks, Ruben
Web Design | | KempRugeLawGroup1 -
Live website is an addon domain - Need site old development url inaccessable from live domain
Hi everyone, I have a website which is built in Joomla 2.5. The development site is located at www,abc.com/subdomain/. We have set the site live using an addon domain which is www.xyz.com. The problem is, www.abc.com/subdomain/ is still accessible and being crawled by Google. How is the best way to make the development url inaccessible? Any help would be appreciated!
Web Design | | DougHosmer0 -
Magento URL Structure
I'm about to migrate to Magento and wanted to ask about the optimal URL structure for the following page: Knowledge Centre |-Videos |-Customer Testimonials |-Customer X Would it be better to use: Domain/knowledge-centre/videos/customer-testimonials/customer-x or Domain/customer-x Thanks in advance for any replies.
Web Design | | ssoneil0 -
Image sliders & site speed
We are having a new website designed using WordPress and the Genesis framework. We wanted to include header image sliders on a number of internal site pages, but our designer says that sliders on more than just the home page will slow down the site significantly. How much could they slow down the site, and what can be done to minimize their effect on site speed?
Web Design | | GordyH0 -
Any reaction to the announcement from Google that 'signed in' searches won't pass through search query info to analytics?
Seems like SEO is about to get that much harder: http://analytics.blogspot.com/2011/10/making-search-more-secure-accessing.html Any thoughts on this?
Web Design | | PaulM011