URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I'm doing a crawl analysis for a website and finding all these duplicate URLs with "null" being added to them and have no clue what could be causing this.
Does anyone know what could be causing this? Our dev team thinks it's caused by mobile pages they created a while ago but it is adding 1000's of additional URLs to the crawl report and being indexed by Google. They don't see it as a priority but I believe these could be very harmful to our site. examples from URL string:
Web Design | | julianne.amann
uruguay-argentina-chilenullnull/days
rainforests-volcanoes-wildlifenullnull/reviews
of-eastern-europenullnullnullnull/hotels0 -
Any second opinions as to why our organic search website traffic hasn't recovered from website rebrand (domain change, website redesign)?
I am hoping to see if anyone in the Moz community would be able to help troubleshoot or lend any advice on a major organic search traffic issue we've been experiencing over the last 8 months. In a nutshell, we decided our ~4.5-year-old business needed to undergo a rebrand in October 2015. After changing domains & redesigning our website (more below), our search-driven sessions have dropped 20% in 2016 v.s. 2015. We made quite a few on-site modifications (with some success) post-redesign but are still deep in a rut and not sure what more we can do to recover. I've listed my theories below as to why we're still suffering this hit. If anyone could weigh in on these and/or share any other troubleshooting ideas, I would greatly, greatly appreciate it (and owe you a lunch/beverage of your choice the next time I'm in your city!). ****Backlinks - despite our efforts to 301 all links, I sense we have lost many backlinks. According to Open Site Explorer, our old domain has 1,172 backlinks (some from some very authoritative pages domains), 1,068 of which are passing link equity. In contrast, our new domain has 367 backlinks, 321 are passing link equity, and very few overlap with our old domain. Domain Age - we may have lost much of our reputation with Google as our new domain is much younger than our old domain (1-year-old v.s. 5.5 years old). Domain Name - although I thought to have common keywords in one's domain was a myth, I am now questioning that belief. Our old domain contained a popular, topical keyword and our new domain is derived from a term that is topical, but very uncommon. New URLs - our developer has insisted all links were moved to the new domain, but I have a hunch they were not. When conducting a "site search" (i.e. "site:websitename.com"), the new domain returns 7,740 results. Prior to our switch, a site search with the old domain yielded 30,000+ results. 404s - we found and fixed 100-200 404'd links after the domain switch. We still see a few pop-up today and I'm wondering if this is a red flag in Google's eyes. For a little more background too, here are the nitty gritty details with a rough timeline: Pre-October 12, 2015 - registered new domain and designed the new website on Wordpress, while researching a range of articles and resources for a successful site migration (e.g. this and this Moz guide). October 12, 2015 - flipped the switch on the website design, domain, minor content reorganization, and social handles. We announced the change to our audience via an article, newsletter, and social; informed Google Webmaster Tools (GWT) of the new address, 301'd all links from the old to the new domain, and submitted new sitemap in GWT. October 12 - 16, 2015 - traffic is normal, everything seems to be okay. October 17, 2015 - search traffic drops by 54% v.s. the same day of week pre-rebrand. October 26, 2015 - search traffic rises, so now only down by 30% v.s. the same day of week pre-rebrand. November/December 2015 - re-added numerous elements from the old website such as category, tag, and page pagination and a few sidebar modules that linked to other important pages and tags. Search traffic rises slightly in November (down 27% year-on-year), dips again in December (down 31% year-on-year). January 2016 - today (June 17, 2016) - we published more content on a daily basis and search traffic fluctuates around the 20% versus the same period in 2015. January 2016 - down 23% year-on-year February 2016 - down 17% year-on-year March 2016 - down 20% year-on-year April 2016 - down 21% year-on-year May 2016 - down 21% year-on-year June 2016 (until the 17th) - down 23% year-on-year Thank you all in advance for your time and help, please let me know if you have any questions!
Web Design | | nick490 -
Internal linking for small site
I have a site with 13 pages, 6 are product pages, 5 are free tips pages (the other 2 are the home page and contact form). Currently I have the navbar at top of site with a "products" dropdown menu for the 6 product pages and a "Tips" dropdown menu for the 5 tip pages. All categories except the contact page are at the bottom as breadcrumbs, the homepage is "home" and the rest are relevant user friendly keyword anchor text. So I have 2 more pages to ad to "Tips" and am wondering whether to have a new 2nd level tips page that links to a 3rd level of 7 different tips pages, or keep it shallow as it is, with only 2 levels from the homepage to the other (now 13) pages, with a potential of 22 pages in the foreseable few years? (and some graphics work to make it user friendly like how Zappo's has categories to the side on each of its drop down navbar menu's and non-link text categories for its bottom of page breadcrumb links) Can those aforementioned pages linking to each other in the footer dilute link equity? (I think that's one of the primary reasons I'm curious). What do you think of this: http://www.dbswebsite.com/blog/2012/08/08/internal-linking-101-5-best-practices/ (I guess I should no follow my contact page), could it be better to have a 2nd level page for "Tips" to get more equity to that page rather than across all 7 tips pages? I have read around about this on here (hence how I found out about Zappo's) and elsewhere and wanted ask to make sure.
Web Design | | Zoolander0 -
Accordion Fold Ups Bad For Google
http://fandicoach.com/products Right now I have these accordion things on the website. Are they bad for google in terms of being an SEO best practice? I want to avoid doing anything black hat. Thanks!
Web Design | | OOMDODigital0 -
Any way of showing missed sales in Google Analytics?
Sit down, this might get a little complicated... I was approached by a design company to do some SEO work a client of theirs. Usually, this stuff is white label but I have direct contact with the client as the design agency felt it was easier for me to do this. The website is performing really well and looking at the sales funnel, I'm getting people wanting to buy. BUT, and here's the problem, everything falls apart because of the way the check out works. It's appalling. The customer has to register to buy a product, there's no guest check out or anything. The checkout button is right below the fold and you'd miss it completely if you didn't actually search for it. Basically, it's losing the client money. Last month alone there were 300~ people entering the conversion funnel and NONE of them complete it. I've been talking with the design company and they basically saying that it's too much work for them to change it, it's a signed off project blah blah. UI reports have been conducted and sent to them but still nothing. I have the client asking (a great client, obviously wondering why there is a lack of return on his investment) why he isn't making money. He's asking me because I'm the guy thats meant to be getting him the cash back. I keep saying to the design agency the problems and that it's never going to make money. The potential is massive. But thats my problem. Is there ANY way in GA to calculate the missed sales? I know that I can view the total amount made when the customer successfully checks out but I need figures to present that I'm leading the horse to water, but the check out system is preventing it from drinking. tl;dr I need to show client/design agency missed sales due to poorly built checkout system. Cheers!
Web Design | | jasonwdexter0 -
How can we improve our e-commerce site architecture to help best preserve Page Authority?
Today I installed the SEOMoz toolbar for Firefox (very cool, highly recommended). I was comparing our site http://www.ccisolutions.com to this competitor: http://www.uniquesquared.com For the most part, the deeper I go in our site the more the page authority drops. We have a few exceptions where the page authority of a subcategory page is actually better than the cat. page one level up. In comparison, when I was looking at http://www.uniquesquared.com I noticed that their page authority stays at "21" on every single category page I visit. Are you seeing what I'm seeing? Is this potentially a problem with the tool bar or, is there something significantly different about their site architecture that allows them to maintain that PA across all category and sub category pages? Is there something fundamentally wrong with our (http://www.ccisolutions.com) site architecture? I understand that we have longer URLs, but this is an old store with a lot of SKUs, so we have decided not to remove the /category/ and /product/ from the URLs because the 301 redirects that would result wouldn't pass all of the authority they've built up over the years. Interested to know viewpoints on the site architecture and how it might be improved. Thanks!
Web Design | | danatanseo0 -
Mobile Site Pages: Word Count Help
Hi there I am doing a mobile website for a client and they asked me what the dieal word count would be per page. They are SEO conciosu but we are not doing SEO on this site. I would just like to know a general rule of thumb. Regards Stef
Web Design | | stefanok0 -
Javascript changing URL - Thoughts?
So, our developer just created a player at the bottom of this site I work for. It's not really important what it is. The thing is, when you go to our home page now, the javascript changes the url from www.site.com to www.site.com/home It's not actually redirected or anything (no 301, it's just the javascript doing this), but I'm worried that if someone links back to our site they're going to surely pull that URL to point back to, which is wrong. Also, when you go to a category, the URL changes from www.site.com/category to www.site.com/home#category. Again, it's not a redirect but I'm still worried people will link back to this since it's on the entire site now... I'm suggesting that we turn off this new feature until we find a workaround. I just wanted to confirm with you guys that this is best. Thanks
Web Design | | poolguy0