URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any second opinions as to why our organic search website traffic hasn't recovered from website rebrand (domain change, website redesign)?
I am hoping to see if anyone in the Moz community would be able to help troubleshoot or lend any advice on a major organic search traffic issue we've been experiencing over the last 8 months. In a nutshell, we decided our ~4.5-year-old business needed to undergo a rebrand in October 2015. After changing domains & redesigning our website (more below), our search-driven sessions have dropped 20% in 2016 v.s. 2015. We made quite a few on-site modifications (with some success) post-redesign but are still deep in a rut and not sure what more we can do to recover. I've listed my theories below as to why we're still suffering this hit. If anyone could weigh in on these and/or share any other troubleshooting ideas, I would greatly, greatly appreciate it (and owe you a lunch/beverage of your choice the next time I'm in your city!). ****Backlinks - despite our efforts to 301 all links, I sense we have lost many backlinks. According to Open Site Explorer, our old domain has 1,172 backlinks (some from some very authoritative pages domains), 1,068 of which are passing link equity. In contrast, our new domain has 367 backlinks, 321 are passing link equity, and very few overlap with our old domain. Domain Age - we may have lost much of our reputation with Google as our new domain is much younger than our old domain (1-year-old v.s. 5.5 years old). Domain Name - although I thought to have common keywords in one's domain was a myth, I am now questioning that belief. Our old domain contained a popular, topical keyword and our new domain is derived from a term that is topical, but very uncommon. New URLs - our developer has insisted all links were moved to the new domain, but I have a hunch they were not. When conducting a "site search" (i.e. "site:websitename.com"), the new domain returns 7,740 results. Prior to our switch, a site search with the old domain yielded 30,000+ results. 404s - we found and fixed 100-200 404'd links after the domain switch. We still see a few pop-up today and I'm wondering if this is a red flag in Google's eyes. For a little more background too, here are the nitty gritty details with a rough timeline: Pre-October 12, 2015 - registered new domain and designed the new website on Wordpress, while researching a range of articles and resources for a successful site migration (e.g. this and this Moz guide). October 12, 2015 - flipped the switch on the website design, domain, minor content reorganization, and social handles. We announced the change to our audience via an article, newsletter, and social; informed Google Webmaster Tools (GWT) of the new address, 301'd all links from the old to the new domain, and submitted new sitemap in GWT. October 12 - 16, 2015 - traffic is normal, everything seems to be okay. October 17, 2015 - search traffic drops by 54% v.s. the same day of week pre-rebrand. October 26, 2015 - search traffic rises, so now only down by 30% v.s. the same day of week pre-rebrand. November/December 2015 - re-added numerous elements from the old website such as category, tag, and page pagination and a few sidebar modules that linked to other important pages and tags. Search traffic rises slightly in November (down 27% year-on-year), dips again in December (down 31% year-on-year). January 2016 - today (June 17, 2016) - we published more content on a daily basis and search traffic fluctuates around the 20% versus the same period in 2015. January 2016 - down 23% year-on-year February 2016 - down 17% year-on-year March 2016 - down 20% year-on-year April 2016 - down 21% year-on-year May 2016 - down 21% year-on-year June 2016 (until the 17th) - down 23% year-on-year Thank you all in advance for your time and help, please let me know if you have any questions!
Web Design | | nick490 -
I want to make changes, in my site's visual appearence
As we are getting more user to our site. We decided to improve its visual appearance. As of now our site ranked higher around 1 - 5 in google. Does the visual changes affect SEO rank and what about adding subdomains?
Web Design | | FhyzicsBCPL0 -
What To Do When Improved Site Speed & Layout Result In Higher Bounce Rates & Lower Time On Site
We launched a new Bootstrap 3.0 site template 2 weeks ago. The site loads 5x faster and has a much improved layout (utilizing most common above the fold recommendations ). It's only been two weeks, but our bounce rate has increased 5-10% and our avg time on site decreased by 10-18%. Here is the page for one of our most common products so you can see the general experience: <a>http://www.jwsuretybonds.com/surety-bonds/commercial-bonds/auto_dealer_bond.htm</a> (here is the old version: <a>http://199.119.123.134/surety-bonds/commercial-bonds/auto_dealer_bond.htm</a>) We spent two months implementing the new design and working on a speedy load time. We had anticipated a drastic improvement, not mild downturn in user behavior. I'm hopeful that the Analytics metrics aren't showing the true picture on the keywords we care about (can't see anymore due to "Not Provided" listed as most keywords now. Argh!) and perhaps some of the more important/accurate user behavior metrics that we can't see are improving. We know our industry and our clients needs VERY well. We THOUGHT our new content/layout was perfect so it will be tough for us to try to make improvements at this point. We believe our best plan of action now is to add more content on each page and A/B test it along with other subtle changes. The problem is that our new content is very concise and hits on all of the primary visitor intentions, so additions of content could be redundant and making concise answers more "fluffy", which is what we tried to get away from. What do you think? Is there reason for panic? What would your plan of attack be if your "sure shot" new design didn't provide the improvements you "knew" it would? 🙂
Web Design | | TheDude0 -
New Google SERPS design - What's Changed?
Has anyone noticed any fall out from the recent redesign of SERP pages by Google? I noticed that there appears to be one less organic result "above the fold" now, so if you were possibly in third or fourth position maybe slight dip in traffic? Any noticeable shift in click through rate with the new bigger font? Also, has anyone noticed if the new design has caused any shift in best practices for on-page meta data like Title tag and description tag counts? I know the Title tag was previously driven by the pixel width of the title in Google SERPS, just curious if that has changed with this redesign.
Web Design | | IrvCo_Interactive0 -
Google HTML, CSS and javascript styleguides ?
Who's following the Google style guides especially in HTML, CSS and javascript? What are the benefits of following the style guides? I am thinking of sending the style guides to our web development team before we launch our new site but I think there might be some conflicts. I'm an SEO and not programmer or web developer and I'm sure there are some "rules" that these web dev guys should follow and break as well. Thanks in advance! 🙂
Web Design | | esiow20130 -
Can anyone recommend a great programming company?
I have had terrible luck with programmers who seem to live in their own little world and never get things done on time. Can anyone recommend a great company here in the usa that you have used before that has done great work? I am looking at the nerdery. Anyone use them?
Web Design | | netviper0 -
Migrating a site to Wordpress
I've recently been converting our old website to a wordpress based website and been working on the new version of the site on a subdomain. Now at the stage when I am getting ready to let the site go live and just wondering exactly how to do this so I have minimal downtime? Looking in the wordpress control panel there is the setting to enter the address of the site if you want it to be different from the directory it has been installed within - is this a good idea (i.e. is it stable if I do this? good for seo, bad for seo or makes no difference?)? or should I manually install everything in the root myself (if I do this is there a way to direct people to the temp version of the site on the subdomain? Any tips, do and don't s would be appreciated as I want to do this right!
Web Design | | Jon-C0 -
Setup of three major retail sites.. need advice.
I recently have taken a new position responsible for three large national retail sites which are all owned by one parent organization. Through a series of acquisitions, these three major brands have been brought under one umbrella and a brand consolidation is likely not to happen within the next 2-4 years. I have a number of questions I’m hoping to get some feedback on, but first a little more background is necessary. A year ago (before my time) the three sites were over-hauled, but were designed to use one common custom CMS and all of the navigation and nearly all the content is the same (with some exceptions, such as tags, url, etc.). All of the brands have identical products and services; however, each one services a different demographic in the US. The design was intended for ease of management, but is terrible for seo. Additionally, without the geographic reference, they all compete for the same keywords. They have now begun a very large ecommerce project utilizing an ATG platform. The initial direction is to use one platform for all three brands, but keep them on separate domains and with the use of basic switching, replace nominal content such as logos and references of the brands for each of the domains. I’m concerned with this approach and would like to hear your feedback.. When optimizing a page for one keyword set, are they likely to be filtered due to dup content? The argument that management has is that all three current sites rank very well for one keyword on all three sites. They feel it won’t be an issue due to this. One option, that is currently still available, is to tri-band one ecommerce site, but it would have to be on an entirely new domain. The other three domains are very well established and are PR6s. Management, and even I, is afraid to abandon these other domains, but having a single domain would allow us to have unique content and really leverage all efforts to one domain. Thoughts? Any knowledge or thoughts what kind of impact having three domains on one ATG platform will be? Thanks much! John If you feel it will help, please message me and I can share the urls... Also, how would you handle a company blog in this case?
Web Design | | kavaliauskas0