Crawling issue
-
Hello,
I am working on 3 weeks old new Magento website. On GWT, under index status >advanced, I can only see 1 crawl on the 4th day of launching and I don't see any numbers for indexed or blocked status.
| Total indexed | Ever crawled | Blocked by robots | Removed |
| 0 | 1 | 0 | 0 |I can see the traffic on Google Analytic and i can see the website on SERPS when i search for some of the keywords, i can see the links appear on Google but i don't see any numbers on GWT.. As far as I check there is no 'no index' or robot block issue but Google doesn't crawl the website for some reason.
Any ideas why i cannot see any numbers for indexed or crawled status on GWT?
Thanks
Seda
| | | | |
| | | | | -
Thanks Davenport and Everett, I've got XML sitemap submitted already, checked robot and no index etc but no stats yet. I'll wait for a few weeks more but it just doesn't make sense to not get any stays after a month. Meanwhile, If i figure out anything, I'll reply here.
-
The data in GWT is not always updated regularly. Also, for a new site that has never been indexed before and has no, or few, external links, it would not be surprising to experience infrequent crawls. The more links you earn and the more of a history of fresh content and updated pages you develop, the more often and deeply you'll be crawled.
As Davenport-Tractor mentioned, an XML sitemap submitted to GWT will also help if you haven't done that already.
If most of your pages are indexed when you do a (site:yourdomain.com) search on Google I wouldn't worry about it too much. If they aren't indexed, you may have a problem, such as inadvertently blocking the crawlers via robots meta tag or robots.txt file. I'd have to see the site to know that though.
-
Seda,
Have you submitted a sitemap to GWMT?
That will greatly help the Google spiders crawl your site. Kind of like telling someone how to find your business vs providing them a road map. They will get there a whole lot quicker if you provide a map on how to find all the different locations.
There are quite a few different sitemap generator programs available. These programs will index your site and build the sitemap.xml file for you. Now you can save the file to your website root directory, then point GWMT to the sitemap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Client wants to show 2 different types of content based on cookie usage - potential cloaking issue?
Hi, A client of mine has compliance issues in their industry and has to show two different types of content to visitors: domain.com/customer-a/about-us domain.com/customer-b/about-us Next year, they have to increase that to three different types of customer. Rather than creating a third section (customer-c), because it's very similar to one of the types of customers already (customer-b), their web development agency is suggesting changing the content based on cookies, so if a user has indentified themselves as customer-b, they'll be shown /customer-b/, but if they've identified themselves as customer-c, they'll see a different version of /customer-b/ - in other words, the URL won't change, but the content on the page will change, based on their cookie selection. I'm uneasy about this from an SEO POV because: Google will only be able to see one version (/customer-b/ presumably), so it might miss out on indexing valuable /customer-c/ content, It makes sense to separate them into three URL paths so that Google can index them all, It feels like a form of cloaking - i.e. Google only sees one version, when two versions are actually available. I've done some research but everything I'm seeing is saying that it's fine, that it's not a form of cloaking. I can't find any examples specific to this situation though. Any input/advice would be appreciated. Note: The content isn't shown differently based on geography - i.e. these three customers would be within one country (e.g. the UK), which means that hreflang/geo-targeting won't be a workaround unfortunately.
Intermediate & Advanced SEO | | steviephil0 -
Manage category pages and duplicate content issues
Hi everybody, I am now auditing this website www.disfracessimon.com
Intermediate & Advanced SEO | | teconsite
this website has some issues with canonicals and other things. But right now I have found something that I would like to know your opinion. When I was checking parts of the content in google to find duplicate content issues I found this: I google I searched: "Chaleco de streck decorado con botones" and found First result: "Hombre trovador" is the one I was checking -> Correct
The following results are category pages where the product is listed in. I was wondering if this could cause any problem related with duplicated content. Should I no index category pages or should I keep it?
The first result in google was the product page. And category pages I think are good for link juice transfer and to capture some searchs from Google. Any advice? Thank you0 -
SEO issues with removing a forum from a site
I'm thinking of adding a Discourse discussion forum to one of my websites. I'm not sure if it's going to be something that works well or not for the site. So I'm thinking ahead and wondering what Google issues I could have if after a few months of having the forum, I decide to remove it. What would Google think about all the then non-existent pages it might have indexed? Would there be a simple wildcard redirect I could do in htaccess that would satisfy that? Or some other thing I should do?
Intermediate & Advanced SEO | | bizzer0 -
WMT Showing Duplicate Meta Description Issues Altough Posts Were Redirected
Dear Moz Community, Some time ago we've change the structure of our website and we've redirected the old URL's to the new ones. About 2,000 posts were redirected at that time. While checking Webmaster Tools a few days ago I've discovered that about 500 duplicate meta-description issues appear in the "HTML Improvements" area. To my surprise, altough the old posts were redirected to the new path, WMT sees the description of the old posts similar with the one of the new post. Moreover, after changing the structure all meta-descriptions were modified and they weren't the same used before the restructure. For example I've redirected /blog/taxi-transfer-from-merton-sw19-to-london-city-airport/ to /destinations/greater-london/merton-sw19/taxi-transfer-to-london-city-airport-from-merton/ Now they are shown as having duplicate content. I've checked the redirects and they are working. I get the same error from the redirected pages for about 150 titles. Did anyone else get this errors or can you please offer me some suggestions about how I can fix this? Thank you in advance! Tiberiu
Intermediate & Advanced SEO | | Tiberiu0 -
Hey guys i have this issues on my crawling report what should i do to exlude the pages? are d
Overly-Dynamic URL Overly-Dynamic URL Although search engines can crawl dynamic URLs, search engine representatives have warned against using over 2 parameters in a given URL. Search engines may also see dynamic versions of the same URL as unique URLs, creating duplicate content.
Intermediate & Advanced SEO | | adulter0 -
When crawls occur - when will my links show up in Open Site Explorer
Hello everyone, I've been building links for a while now and none of them show up in Explorer. My domain authority hasn't changed for about a month or so. When does Google do crawls and when does SEOMoz do crawls? Thanks
Intermediate & Advanced SEO | | Harbor_Compliance0 -
Massive drop in organic rankings - but strange issues
Hi guys, We experienced a massive drop in organic rankings on Saturday 29 October, a drop of around 80%.
Intermediate & Advanced SEO | | cashchampion
I have received no notifications in webmaster tools
The site still ranks for its name so we are not banned
The site has been around since 2007
My mozrank and trustrank are very high (higher than sites that now still rank while we don't)
The strange this is the homepage doesn't rank for the main keyword (which it always did), but now a deeper page ranks for the main keyword, albeit on page 4.
Quite a few of our sites were affected, even though they target different industries, different countries, with different hosts, different unique content, different links (for one we havent even started link building and it was getting 250 UV per day, and now 8). I am completely confused by this.
Any advice or direction would be extremely helpful. Thanks so much0 -
Are tags an issue in SEO
SEOMoz saw that my tags were duplicate pages. Are tags a serious issue in SEO? Should I remove it entirely to prevent the duplicate pages?
Intermediate & Advanced SEO | | visualartistics0