URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Mobile-First Indexing New Site monetized with Adsense AMP or not?
I am considering developing a new site monetized with Adsense. I am wondering if it's still worth bothering with AMP, it will take some work to get the functionality I have in mind working on these pages due to the inherent limitations. Has anyone got any insights in terms of current and future benefits of AMP in terms of ranking benefits and Adsense earning potential?
Web Design | | GrouchyKids0 -
What structured data would you recommend marking up for a companies 'service'?
Hi, I'm finally starting to use structured data in my clients sites now that Google has embraced it so much. I have a client that is a cosmetic physician and has associated services. I thought it would be beneficial to markup each service according to http://schema.org/Service I'll be doing a similar thing with a new local surveyors website too. I'm just wondering if it is worth doing and what Schema properties would you recommend using. For the cosmetic physician I'm thinking of: description
Web Design | | morktron
name
url
sameAs
provider
serviceArea
serviceType0 -
Why aren't Images in G+ product page posts showing up in SERPs for brand searches?
Before 1-2 weeks ago, our G+ posts containing links to our product pages would show up in in SERPs (when searching for our brand name) with a thumbnail of the product image. Now, they do not (see image below for visual). Our tech team confirmed there hasn't been any coding change that might be to blame and I see that this isn't happening to other sites. Any idea what may be the problem here? tcnhLgy
Web Design | | znotes0 -
Can the website pages have the site name like Title of the page | Sitename.com
Hi, Can the website pages have the site name like Title of the page | Sitename.com I have a site with 50K pages and all pages have | Sitename.com mentioned would that be a good practice or bad? Thanks Martin
Web Design | | mtthompsons0 -
Finding a good wordpress web developer for SEO fixes
I have gotten a good audit done of my site now it is time to get the fixes done. I have a developer I worked with for a long time that is good at coding and fixes on my site. However I am not confident in his abilities to execute some of the seo changes that need to be done. Their are some common stuff he can tackle but when it comes to GWT and proper handling of 404's and 301's and other seo tasks I am not sure if he is the right choice.Maybe due to a lack of experience of dealing with the issues I have or is just not his specialty or web devs just don't know seo...lol.... Is obvious a lot of SEO's don't make the changes themselves but leave it up to the devs to handle it their suggestions and fixes. But from my experience devs are not so well versed in seo and you have a hard time knowing if they are doing it correct or can even do it.(of course they will say they can they want your cash and i understand that ). In particular a good amount of wordpress devs claim to know seo but i find that far from the truth.Even when guided to issues some of them will often leave you disappointed. Sorry for my rant! Now to my question , obviously not many SEO's make the actual code changes themselves (how i wish i knew a one that did) are their ones out their that do? If not how do i find a good wordpress dev that can make proper seo changes and knows his stuff....example i need someone who can trouble shoot and track down some serious GWT I have and deal with some hardcore 404 & 301 issues . A lot claim to know but when push comes to shove I have been left disappointed. Thanks in advance for your help and suggestions or recommendations.
Web Design | | chrisyak0 -
Does anyone know how much a wordpress site can store (in terms of data) I want to put all my movies on it and use it as a personal global external hard drive! Thanks!!
So basically, I have about 500 GB of movies on my computer and I don't want to buy an external hardrive. I don't want to spend the money A website I could access anytime, and anywhere, without having to carry my external with me everywhere I go. Thanks in advance for any help/ references.
Web Design | | TylerAbernethy1 -
Suggestions For My Ecommerce Site
I am starting to work on an ecommerce site that I am part owner of. My partner who is the other owner started the site a while back and because he has no internet marketing experience the site didn't come out very well. I am currently overhauling the site and here is a list of things first on my list so I can at least get started on some seo and even ppc. I would really appreciate it if you could take a look at our store and see if you think I am missing anything or you could suggest anything else that really should be done immediately or something that is wrong. The site is www.clubfitnesswarehouse.com Update magento to newest version Fix url structure to be SEO friendly for homepage, category pages, product pages etc. For an example of a site that has great seo friendly urls please refer to examples on http://www.bigfitness.com/ site. For example these pages. http://www.bigfitness.com/treadmillstore.html , http://www.bigfitness.com/bosplincy.html , Remove top navigation menu and instead create left sidebar navigation menu created to navigate site and products. For example of these types of sidebars please refer to sidebar on http://www.bigfitness.com/ or http://www.americanfitness.net/ We will NOT be using same structure as what is in top navigation currently. Categories and keywords of categories will be changed and some will stay the same structure as is currently in navigation For header we would like to feature our shipping policy, returns, privacy policy, no sales tax etc. For example I have seen that I like refer to http://www.spiderofficechairs.com/ Obviously we do not want to copy but something similar for our own site. All images on homepage must be changed to include clickable txt so google can read text not just pictures. We would also like product pages modified to make more user friendly and conversion increased. Add to cart button needs to be changed, text needs to be brighter instead of dull grey color. Add to cart button must also be moved ABOVE THE FOLD! Also on product pages we would like to add sections in addition to Product Description, of Specs, and About section for each product page. We also need the ability to change this information as needed when we need to.
Web Design | | PEnterprises0 -
Recommendation for a company to make a site more mobile friendly?
Hiya, We have a client who uses us for SEO, and a separate company for web development. They have a fairly large site on a bespoke CMS. They're happy with the site, but the user experience on mobile devices is not right. Can anybody recommend a company specializing in that area? Preferably a UK company but not essential. Thanks 🙂
Web Design | | SteveOllington1