Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?
-
Hi Everyone!
My website is not being crawled regularly by Google - there are weeks when it's regular but for the past month or so it does not get crawled for seven to eight days. There are some specific pages, that I want to get ranked but they of late are not being crawled AT ALL unless I use the 'Fetch As Google' tool! That's not normal, right? I have checked and re-checked the on-page metrics for these pages (and the website as a whole, backlinking is a regular and ongoing process as well! Sitemap is in place too! Resubmitted it once too! This issue is detrimental to website traffic and rankings! Would really appreciate insights from you guys! Thanks a lot!
-
Hi Farhan
Yes sure, we've definitely found this helps bring the crawlers back to the site more regular!
Hope it helps
Dan
-
Hi Dan! Thanks a lot for your views! This might be it! Content on the website is not updated very regularly owing to the nature of the site. Perhaps updating the blog more regularly or making tweaks in the meta script could help as per your suggestions?
-
Hi Anders! They are all available to the Googlebot! I’m afraid, it won’t be possible for me to share the URL on a public forum! Please check your inbox! Thank you so much for your insights and concern!
-
Hi Umar! There’s been no change in web hosting nor a penalty! We regularly monitor the Search Console and the speed is good too! Thanks a lot for your time though!
-
Hi
How regular are you updating content on the site?
We've found that regular updating and tweaks to content on pages to improve the pages content & user's experience helps the page indexing & frequency.
Looking at analytics and seeing what customers are searching for on your site can help you then improve the content on pages, by going back amending and improving page content we've found Google returns more often to re-index.
Dan
-
Hi!
Are all pages available for GoogleBot, or are they perhaps blocked by robots.txt or a meta tag?
Do you have a URL for the website so we can take a look?
Anders
-
Hello Farhan,
Yes, it's not normal. Did you change your web hosting in this period? or have you faced any kind of penalty in recent past? Usually this happens when you just recovered.
You also need to look out your Search Console for crawling problems, are there any crawling or server issues you're facing? Sometimes, your .htaccess creates crawling problems, make sure to check again. Also, see how fasts your pages are loading, if the speed is low it also cause the crawling delayed.
Hope this helps!
Umar
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to get google to forget my old but still working page and list my new fully optimized page for a keyword?
Hi There! (i am beginner in seo) I have dynamic and static pages on our site. I created a static page for a specific keyword. Fully optimized it, (h1, alt, metas, etc.....maybe too optimized). My problem is that this page is alive for weeks, checked it in GWT and it is in robots.txt, google sees it, and indexed it. BUT whenewer i do a search for that keyword, we still appear with the dynamically created link in the google listings. How could i "redirect" google, if sy make a search for that keyword than shows our optimized page? Is there a tool for that? I cant delete the dynamic page... Any ideas? Thx Andrew
Technical SEO | | Neckermann0 -
Keyword rankings for new website good in Yahoo and Bing but no movement in Google?
Hello, This past September I launched a new redesigned website for a client. His old website was a static html site that was many years old and the new website was created using WordPress. With the new design we made sure to use all the proper techniques for SEO. (h1 tages, image names, quality links, page titles, etc.) Plus, all the content is new content written for this site. I've actually launched new sites many times and after a few months usually start seeing keyword ranking improvements from the major search engines. With this particular website I'm seeing improvements in Yahoo and Bing but no movement in Google. I've used Google Webmaster Tools and made sure my sitemap is being submitted, etc. It all seems good, but I can't understand why Yahoo and Bing are working but nothing from Google. My page grades are all A's and B's and Moz isn't showing any big issues. Maybe I need to give it more time? This client is a lawyer and has many websites out there so maybe he's being penalized somewhere I don't know? As I mentioned, I've been doing SEO for about 8 years and have never had this much trouble with Google. I was wondering if you can look at the site and see if there are any glaring issues I'm missing. Website: http://www.arizonamedicalmalpractice.info/ The keyword phrases were are looking at are "Phoenix Medical Malpractice Lawyer", "Phoenix Medical Malpractice Attorney:, "Arizona Medical Malpractice Lawyer & Attorney", etc. I appreciate anyone who takes the time and does a quick look over. Thanks very much, Bill
Technical SEO | | Bill_K0 -
Can someone interpret this entry in my htaccess file into english so that I can understand?
There are a number of entries in my htaccess and I'd like to understand what they are doing so that I can understand if they need to be there or not. So, can someone tell me what this says...in plain english? RewriteCond %{HTTP_HOST} ^legacytravel.com$ [OR]
Technical SEO | | cathibanks
RewriteCond %{HTTP_HOST} ^www.legacytravel.com$
RewriteRule ^carrollton-travel-agent$ "http://www.legacytravel.com/carrollton-travel-agent" [R=301,L] Thank you a million times in advance.0 -
Local Google vs. default Google search
Hello Moz community, I have a question: what is the difference between a local version of Google vs. the default Google in regards to search results? I have a Mexican site that I'm trying to rank in www.google.com.mx, but my rankings are actually better if I check my keywords on www.google.com The domain is a .mx site, so wouldn't it make more sense that this page would rank higher on google.com.mx instead of the default Google site, which in theory would mean a "broader" scope? Also, what determines whether a user gets automatically directed to a local Google version vs. staying on the default one? Thanks for your valuable input!
Technical SEO | | EduardoRuiz0 -
Can a Novice Fix Parallelize Issues?
I was working yesterday on making my WP site quicker (sellingwarnerrobins.com) and after updating the htaccess file to solve some "Leverage Browser Caching" issues I re-ran a scan on Pingdom Tools and am now getting a zero for "Parallelize downloads across hostnames" with a list of 34 items to fix. I did some web searches and when the articles started talking about cnames, subdomains, and hostname distribution it went beyond my capabilities. Are these Parallelize "issues" something a novice like myself can easily fix? If so, how?
Technical SEO | | Anita_Clark0 -
Google does not show my website anymore
Hi All, We developed a new website for the domain: www.instral.com. Before we build the webste, the domain was indexed by google and was showed as first result on a search "Instral". Without a website! Only a super simple hostingprovider webpage. Now, with the new website, all the website pages are found in google from page 7... home page is not even in the results. When i search on "instral.com" in google, it will show my website on first result including homepage. Is there something wrong with the website or DNS settings? Or mabe some other webhosting setting... am i on a blacklist or something? Bing and Yahoo are showing better results (first page). I hope someone can help me out here...
Technical SEO | | extrememedia0 -
What I doing wrong when trying to search for links from external websites to my website
This is just the little frustrating question nothing important but I’m sure somebody will know the answer. In the white board Friday this week Rand suggested at one point that when you’re searching for results links to your website if you put a - followed by site followed by your url like –site:yourwebsite.com you get the results of pages with links on other websites but excluding your own webpages but it just doesn’t work I get no results just an error message, any idea why? If I remove the - I get tons of results but there on my own webpages……….
Technical SEO | | whitbycottages0 -
Duplicate content issues with australian and us version of website
Good afternoon. I've tried searching for an answer to the following question but I believe my circumstance is a little different than what has been asked in the past. I currently run a Australian website targeted at a specific demographic (50-75) and we produce a LARGE number of articles on a wide variety of lifestyle segments. All of our focus up until now has been in Australia and our SEO and language is dedicated towards this. The next logical step in my mind is to launch a mirror website targeted at the US market. This website would be a simple mirror of a large number of articles (1000+) on subjects such as Food, Health, Travel, Money and Technology. Our current CMS has no problems in duplicating the specific items over and sharing everything, the problem is in the fact that we currently use a .com.au domain and the .com domain in unavailable and not for sale, which would mean we have to create a new name for the US targeted domain. The question is, how will mirroring this information, targeted towards US, affect us on Google and would we better off getting a large number of these articles 're-written' by a company on freelancer.com etc? Thanks,
Technical SEO | | Geelong
Drew0