URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site redesign - 301 Redirects
I've just overhauled a website, leaving lots of former posts in the dust. I've set up a 404 redirect to the home page so that if anyone goes to one of those old pages they land on the home page instead of a dead 404. But, there's a couple urls from the old site I'd prefer to redirect to similar pages. These urls have forward slashes and I don't know how to get the slashes in when I copy it over to the new site. This is probably something easy, but I'm baffled. This www.lawbarron.com/personal-injury/whip-its-nitrous-oxide/ becomes this when I copy it www.lawbarron.com/personal-injury-whip-its-nitrous-oxide Can someone help me out?
Web Design | | julie-getonthemap0 -
Is it Bad to Break Up A Site into Multiple Sites?
I have a big cluttered website with endless pages. It's a non-profit that has content for patients, researchers, therapists, etc.. Would it be a bad idea to turn this cluttered site into 3 or more completely different sites, each focused on their specific demographic? Or should I just figure out how to organize the one site better? Thanks for your help!!!
Web Design | | bosleypalmer0 -
Did i got hit from some google updates.
Hello everybody, i got a problem and i hope someone can clear it up for me. my root domain authority is 42 and home page is 52 (jumped there only yesterday) ,while my google page rank is still PR2 (same for 3 month already). 1 month ago i changed my home page design (not the text) and since then my home page just disappeared from the search engines. can somebody look on my website www.kspiercing.com , and tell me if i got hit by some panda ,koala,penguin or some other sweet Google animal . thank you very much.
Web Design | | kspiercing0 -
Can anyone recommend a firm for cross-browser testing?
I'd like to outsource the cross-browser testing and get complete functionality testing across browsers too. I work for an online travel tour operator, so we take a lot of reservations online. I don't totally trust my internal resources using tools like Selenium because we're adding a lot of new functionality to our new site - saved itineraries, wish list, some upselling and cross selling functionality, selling insurance, etc. So, while of course I want to be sure everything renders properly across browsers, I also feel like the investment in having someone check booking process in all browsers/technologies is critical especially since many of our customers are accessing our sites using older browsers. Thoughts??
Web Design | | CityWonders0 -
Google penalty for links opening in new tab?
Our web services provided suggested that Google doesn't like in-text links that open the link in a new tab. Can anyone verify this? We often link to outside credible resources for our audience, though it seems smarter to open in a new tab rather than risk that the person will not navigate back to our site after finding us. Thank you in advance!
Web Design | | jhamlin0 -
Duplicate content on mobile sites
Hi Guys We are launching a mobile webshop later this year and have decided to use a subdomain for this. (m.domainname.xx). The content will be more or less identical with the one on the standard desktop site (domainname.xx), but im struggeling to find out if this will create dipplicate content between the mobile and desktop site. Does anyone have a solid answer for this one?
Web Design | | AndersDK0 -
Google indexing Quickview popups
Hi Guys I can't seem to find any info on this. Maybe you can help. We are using xcart as our shopping cart. When you land on a product page you have the option to "Quickview" the item. Google is picking up the quickview urls" and the vote on product urls. I have added the following to the robots.txt file but not sure if this will work. Any help on this would be great. Disallow: /?popup=Y Disallow: /?mode=add Undesired URL Examples: <colgroup><col width="735"></colgroup>
Web Design | | fasctimseo
| http://www.funlove.com/store/6_Pack_Shooter_Beer_Belt/?mode=add_vote&vote=60 | <colgroup><col width="735"></colgroup>
| http://www.funlove.com/store/6_pack_shooter_beer_belt/?popup=Y |0 -
Does listing my customer's address, phone number, and a contact form on "every page" count as duplicate content that they'd be penalized for?
I work with small local businesses (like Tree Farms, Feed Stores, Counselors, etc) doing web design, seo, etc. I encourage them to have their contact information visible at all times on their websites. I'm also delving into the world of contact forms. I want to have this info on every page - is this detrimental? Here's an example: http://www.trinityescape.net/marriage-couples-counselors-therapy-clermont-florida/ Thank you!
Web Design | | mikjgens1
