Crawling issue
-
Hello,
I am working on 3 weeks old new Magento website. On GWT, under index status >advanced, I can only see 1 crawl on the 4th day of launching and I don't see any numbers for indexed or blocked status.
| Total indexed | Ever crawled | Blocked by robots | Removed |
| 0 | 1 | 0 | 0 |I can see the traffic on Google Analytic and i can see the website on SERPS when i search for some of the keywords, i can see the links appear on Google but i don't see any numbers on GWT.. As far as I check there is no 'no index' or robot block issue but Google doesn't crawl the website for some reason.
Any ideas why i cannot see any numbers for indexed or crawled status on GWT?
Thanks
Seda
| | | | |
| | | | | -
Thanks Davenport and Everett, I've got XML sitemap submitted already, checked robot and no index etc but no stats yet. I'll wait for a few weeks more but it just doesn't make sense to not get any stays after a month. Meanwhile, If i figure out anything, I'll reply here.
-
The data in GWT is not always updated regularly. Also, for a new site that has never been indexed before and has no, or few, external links, it would not be surprising to experience infrequent crawls. The more links you earn and the more of a history of fresh content and updated pages you develop, the more often and deeply you'll be crawled.
As Davenport-Tractor mentioned, an XML sitemap submitted to GWT will also help if you haven't done that already.
If most of your pages are indexed when you do a (site:yourdomain.com) search on Google I wouldn't worry about it too much. If they aren't indexed, you may have a problem, such as inadvertently blocking the crawlers via robots meta tag or robots.txt file. I'd have to see the site to know that though.
-
Seda,
Have you submitted a sitemap to GWMT?
That will greatly help the Google spiders crawl your site. Kind of like telling someone how to find your business vs providing them a road map. They will get there a whole lot quicker if you provide a map on how to find all the different locations.
There are quite a few different sitemap generator programs available. These programs will index your site and build the sitemap.xml file for you. Now you can save the file to your website root directory, then point GWMT to the sitemap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Content Strategy/Duplicate Content Issue, rel=canonical question
Hi Mozzers: We have a client who regularly pays to have high-quality content produced for their company blog. When I say 'high quality' I mean 1000 - 2000 word posts written to a technical audience by a lawyer. We recently found out that, prior to the content going on their blog, they're shipping it off to two syndication sites, both of which slap rel=canonical on them. By the time the content makes it to the blog, it has probably appeared in two other places. What are some thoughts about how 'awful' a practice this is? Of course, I'm arguing to them that the ranking of the content on their blog is bound to be suffering and that, at least, they should post to their own site first and, if at all, only post to other sites several weeks out. Does anyone have deeper thinking about this?
Intermediate & Advanced SEO | | Daaveey0 -
Is there an issue with my site?
Been mostly hanging around top of page two for the last couple of years for “Liverpool Wedding photographer” although got myself on page 1 for “Liverpool photographer” I have split the title of the page to target these two keywords. I took the Liverpool photographer off the title to see if it was being detrimental to the “Liverpool wedding photographer” I didn’t see no increase in ranking so put it back as I get a bit of commercial work from it. Since last year I have got onto page 1 at least three times around position 5-6. Within a week or two I start sliding down again and end up back at top of page two. I could understand this slow push out if my competitors were busy SEO wise but from what I have seen they are not. There is a guy using the keywords in URL and calls himself “Liverpool wedding photographer” last time I checked he literally had no links but is in the first 5 positions. I have I think a better link profile than every one else. Although I am on and off with Facebook and Instagram, (more off) so that probably isn’t helping. Although I have a colleague in the video side of things and he doesn’t use social media at all and it hasn’t harmed him. A few years ago I was burned quite badly by a total charlatan. He sunk my home page to page 4. He talked the talk about creating landing pages but his methods were shoddy to say the least. I can’t believe I was taken in by him, although I was only with him for 2 months. He was still using spammy link techniques to generate lots of toxic links for me! I disavowed all of his links and put the keywords back on the home page and was back to my usual top of page 2 position within a week. Since then I have disavowed all directory links and anything not wedding related. I have an article which ranks 1st or second for “Nikon CLS”. I have also another article of 2000 words or so on another reasonable placed photography website. A few links from other vendors or people I have taken photographs for. I have about 10 featured weddings with a link on 4 good weddings blogs. I don’t think a massive amount of blog comments although I have stopped doing this. If I look at most of the competitors these are their main links, with directories as well! Last winter I put a quite substantial article about documentary wedding photography on my home page. I flew to number 2, although I photographed The World Transformed (the alternative labour conference in Liverpool). I got a lot of clicks to a gallery page (few thousand off social media} so I don’t know if that coincided with it. Same thing – watching the website go down a few positions every day until within just over a week or two I was about 4<sup>th</sup> on page 2! Its like my website is on a spring which can push into page 1 but rebounds back to top of page 2. I am staring to worry that my site has been marked as a bad character in some way because I get what seems to be rough treatment from google compared to my peers. I have written I think 4 or 5 (1500 word) articles the last couple of months talking about lenses and wedding photography related topics and Google pushed me back to page 1, peaking At position 5. I was there for a few weeks and then the slide happened again. Bit demoralised at the moment, what to do? Any help or pointers would be most appreciated. Best wishes. David.
Intermediate & Advanced SEO | | WallerD0 -
Wordpress sidebar dropdown url ?cat...number issue
Hi does anyone know how to fix wordpress sidebar dropdown url ?cat...number found this below, but still not sure the best way to fix https://wordpress.org/support/topic/how-to-display-category-name-in-url-not-cat-number https://wordpress.org/support/topic/category-drop-down-menu-shows-wrong-permalink-structure-1 any ideas?
Intermediate & Advanced SEO | | Taiger0 -
Duplicate/ <title>element too long issues</title>
I have a "duplicate <title>"/"<title> element too long" issue with thousands of pages. In the future I would like to automate these in a way that keeps them from being duplicated AND too long. The solution I came up with was to standardize these monthly posts with a similar, shorter, <title>, but then differentiate by adding the month and the year of the post at the end of each <title>. Hundreds of these come out every week, so it is hard to sit there and come up with a unique <title> every time. With this solution the <title> tags would undoubtedly be short enough, however my primary concern is, would simply adding the month and year at the end of each <title> be enough for Google/Moz to decide it is not a duplicate? How much variation is enough for it not to be deemed a duplicate <title>? </p></title>
Intermediate & Advanced SEO | | Brian_Dowd0 -
Duplicate content issue - online retail site.
Hello Mozzers, just looked at a website and just about every product page (there are hundreds - yikes!) is duplicated like this at end of each url (see below). Surely this is a serious case of duplicate content? Any idea why a web developer would do this? Thanks in advance! Luke prod=company-081
Intermediate & Advanced SEO | | McTaggart
prod=company-081&cat=20 -
How does the crawl find duplicate pages that don't exist on the site?
It looks like I have a lot of duplicate pages which are essentially the same url with some extra ? parameters added eg: http://www.merlin.org.uk/10-facts-about-malnutrition http://www.merlin.org.uk/10-facts-about-malnutrition?page=1 http://www.merlin.org.uk/10-facts-about-malnutrition?page=2 These extra 2 pages (and there's loads of pages this happens to) are a mystery to me. Not sure why they exist as there's only 1 page. Is this a massive issue? It's built on Drupal so I wonder if it auto generates these pages for some reason? Any help MUCH appreciated. Thanks
Intermediate & Advanced SEO | | Deniz0 -
Ecombuffet.com are offering a Rescue Review focused on Panda - Penguin and identifying issues. Has anyone used this service or aware of the organisation in general?
http://www.ecombuffet.com/rescue-review.htm . I have 2 sites that have definitely been hit by penguin and getting worse so am thinking of paying for this service as nothing I do seems to stop the slide (more like a plummet). Any comments welcome.
Intermediate & Advanced SEO | | Shaann1 -
Massive drop in organic rankings - but strange issues
Hi guys, We experienced a massive drop in organic rankings on Saturday 29 October, a drop of around 80%.
Intermediate & Advanced SEO | | cashchampion
I have received no notifications in webmaster tools
The site still ranks for its name so we are not banned
The site has been around since 2007
My mozrank and trustrank are very high (higher than sites that now still rank while we don't)
The strange this is the homepage doesn't rank for the main keyword (which it always did), but now a deeper page ranks for the main keyword, albeit on page 4.
Quite a few of our sites were affected, even though they target different industries, different countries, with different hosts, different unique content, different links (for one we havent even started link building and it was getting 250 UV per day, and now 8). I am completely confused by this.
Any advice or direction would be extremely helpful. Thanks so much0