Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?
-
Hi Everyone!
My website is not being crawled regularly by Google - there are weeks when it's regular but for the past month or so it does not get crawled for seven to eight days. There are some specific pages, that I want to get ranked but they of late are not being crawled AT ALL unless I use the 'Fetch As Google' tool! That's not normal, right? I have checked and re-checked the on-page metrics for these pages (and the website as a whole, backlinking is a regular and ongoing process as well! Sitemap is in place too! Resubmitted it once too! This issue is detrimental to website traffic and rankings! Would really appreciate insights from you guys! Thanks a lot!
-
Hi Farhan
Yes sure, we've definitely found this helps bring the crawlers back to the site more regular!
Hope it helps
Dan
-
Hi Dan! Thanks a lot for your views! This might be it! Content on the website is not updated very regularly owing to the nature of the site. Perhaps updating the blog more regularly or making tweaks in the meta script could help as per your suggestions?
-
Hi Anders! They are all available to the Googlebot! I’m afraid, it won’t be possible for me to share the URL on a public forum! Please check your inbox! Thank you so much for your insights and concern!
-
Hi Umar! There’s been no change in web hosting nor a penalty! We regularly monitor the Search Console and the speed is good too! Thanks a lot for your time though!
-
Hi
How regular are you updating content on the site?
We've found that regular updating and tweaks to content on pages to improve the pages content & user's experience helps the page indexing & frequency.
Looking at analytics and seeing what customers are searching for on your site can help you then improve the content on pages, by going back amending and improving page content we've found Google returns more often to re-index.
Dan
-
Hi!
Are all pages available for GoogleBot, or are they perhaps blocked by robots.txt or a meta tag?
Do you have a URL for the website so we can take a look?
Anders
-
Hello Farhan,
Yes, it's not normal. Did you change your web hosting in this period? or have you faced any kind of penalty in recent past? Usually this happens when you just recovered.
You also need to look out your Search Console for crawling problems, are there any crawling or server issues you're facing? Sometimes, your .htaccess creates crawling problems, make sure to check again. Also, see how fasts your pages are loading, if the speed is low it also cause the crawling delayed.
Hope this helps!
Umar
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Internal link is creating duplicate content issues and generating 404s from website crawl.
Not sure what the best way to describe it but the site is built with Elementor page builder. We are finding out that a feature that is included with a pop modal window renders an HTML code as so: Click So when crawled I think the crawling is linking itself for some reason so the crawl returns something like this: xyz.com/builder/listing/ - what we want what we don't want xyz.com/builder/listing/ xyz.com/builder/listing/%23elementor-action%3Aaction%3Dpopup%3Aopen%26settings%3DeyJpZCI6Ijc2MCIsInRvZ2dsZSI6ZmFsc2V9/ xyz.com/builder/listing/%23elementor-action%3Aaction%3Dpopup%3Aopen%26settings%3DeyJpZCI6Ijc2MCIsInRvZ2dsZSI6ZmFsc2V9//%23elementor-action%3Aaction%3Dpopup%3Aopen%26settings%3DeyJpZCI6Ijc2MCIsInRvZ2dsZSI6ZmFsc2V9/ so you'll notice how that string in the HREF is appended each time and it loops a couple times. Could I 301 this issue, what's the best way to go about handling something like this? It's causing duplicate meta descriptions/content errors for some listing pages we have. I did add a rel='nofollow' to the anchor tag with JavaScript but not sure if that'll help.
Technical SEO | | JoseG-LP0 -
Google Sites website https://www.opcfitness.com/ title NOT GOOD FOR SEO
We set up a website https://www.opcfitness.com/home on google sites. but google sites page title not good for SEO. How to fix it?
Technical SEO | | ahislop5740 -
Can bad html code hurt your website from ranking ?
Hello,For example if I search for “ Bike Tours in France” I am looking for a page with a list of tours in France.Does it mean that if my html doesn’t have list * in the code but only that apparently doesn’t have any semantic meaning for a search engine my page won’t rank because of that ?Example on this page : https://bit.ly/2C6hGUn According to W3schools: "A semantic element clearly describes its meaning to both the browser and the developer. Examples of non-semantic elements: <div> and - Tells nothing about its content. Examples of semanticelements: <form>, , and- Clearly defines its content."Has anyone any experience with something similar ?Thank you, </form>
Technical SEO | | seoanalytics0 -
Issues with getting a web page indexed
Hello friends, I am finding it difficult to get the following page indexed on search: http://www.niyati.sg/mobile-app-cost.htm It was uploaded over two weeks back. For indexing and trouble shooting, we have already done the following activities: The page is hyperlinked from the site's inner pages and few external websites and Google+ Submitted to Google (through the Submit URL option) Used the 'Fetch and Render' and 'Submit to index' options on Search Console (WMT) Added the URL on both HTML and XML Sitemaps Checked for any crawl errors or Google penalty (page and site level) on Search Console Checked Meta tags, Robots.txt and .htaccess files for any blocking Any idea what may have gone wrong? Thanks in advance!
Technical SEO | | RameshNair
Ramesh Nair0 -
My website keeps getting hit every other month. What should I do?
Since April 2012, my website impressions has dropped about 88% according to GWT. Every other month or so, the impressions are dropping by about 30%. My total traffic (visitors, not impressions) has dropped by about 25% but now I am getting lots of junk traffic. A few of the major keywords I used to rank for are still ranking in the top 10 but only in the USA. Lots of the keywords have gone to page 2 or 3 in the US and are gone to hell in other countries. Now I know I'm mostly responsible for this mess. About 3 years ago, I hired a freelancer to write news for my blog and she did a great job for quite some time so I stopped monitoring her work for duplicated content. Unfortunately, she started to provide me with copied content after a while and did so for almost 9 months before I noticed it. I had also hired RankPoop - errr I mean RankPop - to build some backlinks and that eventually got me in trouble too. I got an unnatural links warning in GWT in July 2012. Since then, I had more than 50% of the bad links taken down. There are still lots of them but they sure account for way less than 50% of all the backlinks. I have not submitted a reconsideration request yet as I haven't compensate for all the links taken down yet. I also started adding LOTS of fresh, unique and useful content to the website. I've added near 400 articles (sometimes up to 7 or 8 articles a day) over the last 5 months. I've also set lots of the duplicated posts to "noindex" and when they're not indexed anymore, I completely removed them in order to avoid any copyright issues (some were 100% identical to the source). I keep doing this gradually to avoid 404 errors. In early March of this year, I did a complete redesign of the site. The navigation structure stayed the same and visually, the layout is quite the same although the graphical elements are much more professional and the site is much faster. As much as I would've like to avoid a complete redesign, major technical issues from the previous design (and development platform) was now forcing me to do so. Unfortunately, I updated the website design right before the last Panda update so now I don't know if the recent traffic lost is due to the new design or because of Panda... or both. Google is like a police officer who repeatedly give you tickets for the same offense, yet they won't tell you what that offense is. My website is located at http://www.thewebhostinghero.com/ Any advice is welcomed. P.S. sorry for my english, I speak french.
Technical SEO | | sbrault740 -
Index quickly a website? (Google,Bing..)
Hi, I would like to know what are the best practices in 2012 to index our website in less than 24 hours? (or less..) Thanks for your answer 😄
Technical SEO | | Probikeshop0 -
I just found something weird I can't explain, so maybe you guys can help me out.
I just found something weird I can't explain, so maybe you guys can help me out. In Google http://www.google.nl/#hl=nl&q=internet. The number 3 result is a big telecom provider in the Netherland called Ziggo. The ranking URL is https://www.ziggo.nl/producten/internet/. However if you click on it you'll be directed to https://www.ziggo.nl/#producten/internet/ HttpFox in FF however is not showing any redirects. Just a 200 status code. The URL https://www.ziggo.nl/#producten/internet/ contains a hash, so the canonical URL should be https://www.ziggo.nl/. I can understand that. But why is Google showing the title and description of https://www.ziggo.nl/producten/internet/, when the canonical URL clearly is https://www.ziggo.nl/? Can anyone confirm my guess that Google is using the bulk SEO value (link juice/authority) of the homepage at https://www.ziggo.nl/ because of the hash, but it's using the relevant content of https://www.ziggo.nl/producten/internet/ resulting in a top position for the keyword "internet".
Technical SEO | | NEWCRAFT0 -
How can I get Google to crawl my site daily?
I was wndering if there was a trick to getting google to crawl my website daily?
Technical SEO | | labradoodlelocator0
