How can I get Google to crawl my site daily?
-
I was wndering if there was a trick to getting google to crawl my website daily?
-
Great advice from everyone!!! Thanks!
Saibose - If I add a twitter, FB or blog feed to my site, and it updates daily, this will help the rate that google crawls my site? Does it matter that it is embeded content?
-
Blogs, tweets and other social mentions are always good for a fresh crawl.
-
The best way is to enable the creation of unique content daily. UGC and user participation if added to the unique content would add to the crawl ability. If you want to go a bit high tech, you can put the fresh content on a webcache to enable Google bots to visit you more frequently. ALso, make sure that your older/archived pages get some content cycled ( new comments, new tweets etc etc)
-
sitemap should show daily, but only for pages you intent to change daily. I got the idea from a Matt Cutts video, that they only honour the sitmap if they find it to be reliable. if you say daily but never update the content, then its not going to work.
-
Proper sitemap that tells the SE's to come back can help.
Write content daily. If you have a blog, ping google when you post a new piece of content. If you show Google that you write content daily, they'll come back.
I did SEO for a very successful conservative news website and Google crawled them 15-20 times a day and indexed things in sometimes seconds.
Make good use of pinging!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi! I first wrote an article on my medium blog but am now launching my site. a) how can I get a canonical tag on medium without importing and b) any issue with claiming blog is original when medium was posted first?
Hi! As above, I wrote this article on my medium blog but am now launching my site, UnderstandingJiuJitsu.com. I have the post saved as a draft because I don't want to get pinged by google. a) how can I get a canonical tag on medium without importing and b) any issue with claiming the UJJ.com post is original when medium was posted first? Thanks and health, Elliott
Technical SEO | | OpenMat0 -
Is there a way to get Google to index more of your pages for SEO ranking?
We have a 100 page website, but Google is only indexing a handful of pages for organic rankings. Is there a way to submit to have more pages considered? I have optimized meta data and get good Moz "on-page graders" or the pages & terms that I am trying to connect....but Google doesn't seem to pick them up for ranking. Any insight would be appreciated!
Technical SEO | | JulieALS0 -
Can you help with some concerns over mobile site in SERPS
Hi Guys I wonder if you can put me right on this query please? We have a mobile version of our site, on an m. domain. it is actually http://m.uniquemagazines.co.uk. I have created a mobile sitemap, and added the sitemap reference to the robots.txt file. I have setup the site in web master tools, and submitted the sitemap to web master tools. The sitemap can be found at http://m.uniquemagazines.co.uk/sitemap.xml. This is an index file, which in turn links to 3 sitemaps, one for pages, one for product pages and one for category pages. I have not blocked any crawlers in the robots.txt file. My query is we have some products showing up in the SERPS with both the mobile and normal website. Secondly, in web master tools I am told that our mobile sitemap contains 3675 submitted URLs. However, it states under the WEB banner that 3675 submitted, and 3675 indexed. But under MOBILE it states 3675 submitted, none indexed... I will attempt to attach a screenshot from WMT of this. Do you have any ideas what has gone wrong here? is it wrong, or is this what you would expect to see? Many thanks Paul ScreenHunter_18%20Feb.%2021%2010.55.jpg
Technical SEO | | TheUniqueSEO0 -
What does the Google Crawler see when crawling this page?
If you look at this page http://www.rockymountainatvmc.com/t/49/61/185/730/Batteries. You will see we have a vehicle filter on it. Right now you only see a picture of a battery and some bad text that needs to be updated ( We just hired a copywriter!). Our question is when google crawls this site will thy just see this or will they see all the products that appear after you pick a "machine type" "make" "model" and "year" Any help would be great. Right now we think it just sees this main page how we have set things up; however, we know that the crawler is also crawling some ajax. We just want to be sure of things.
Technical SEO | | DoRM0 -
How many times robots.txt gets visited by crawlers, especially Google?
Hi, Do you know if there's any way to track how often robots.txt file has been crawled? I know we can check when is the latest downloaded from webmaster tool, but I actually want to know if they download every time crawlers visit any page on the site (e.g. hundreds of thousands of times every day), or less. thanks...
Technical SEO | | linklater0 -
If I get a natural link for a great site and I have my keyword with anchor text in this link, how should I proceed?
If I get a natural link for a great site and I have my keyword with anchor text in this link, how should I proceed? I need to contact the site and ask to remove the link or request the removal of the anchor text and leave only the site URL? Or yet do not I need to worry about this issue?
Technical SEO | | soulmktpro0 -
Mobile site rank on Google S.E. instead of desktop site.
Hello, all SEOers~ Today, I would like to hear your opinion regarding on Mobile site and duplicate contents issue. I have a mobile version of our website that is hosted on a subdomain (m instead www). Site is targeting UK and Its essentially the same content, formatted differently. So every URL on www exists also at the "m" subdomain and is identical content. (there are some different contents, yet I could say about 90% or more contents are same) Recently I've noticed that search results are showing links to our mobile site instead of the desktop site. (Google UK) I have a sitemap.xml for both sites, the mobile sitemap defined as follows: I didn't block googlebot from mobile site and also didn't block googlebot-mobile from desktop site. I read and watched Google webmaster tool forum and related video from Matt Cutts. I found many opinion that there is possibility which cause duplicate contents issue and I should do one of followings. 1. Block googlebot from mobile site. 2. Use canonical Tag on mobile site which points to desktop site. 3. Create and develop different contents (needless to say...) Do you think duplicate contents issue caused my mobile site rank on S.E. instead of my desktop site? also Do you think those method will help to show my desktop site on S.E.? I was wondering that I have multi-country sites which is same site format as I mentioned above. However, my other country sites are totally doing fine on Google. Only difference that I found is my other country sites have different Title & Meta Tag comparing to desktop site, but my UK mobile site has same Title & Meta Tag comparing to desktop. Do you think this also has something to do with current problem? Please people~! Feel free to make some comments and share your opinion. Thanks for reading my long long explanation.
Technical SEO | | Artience0 -
Does google use the wayback machine to determine the age of a site?
I have a site that I had removed from the wayback machine because I didn't want old versions to show. However I noticed that in many seo tools the site now always shows a domain age of zero instead of 6 years ago when I registered it. My question is what do the actual search engines use to determine age when they factor it into the ranking algorithm? By having it removed from the wayback machine, does that make the search engines think the site is brand new? Thanks
Technical SEO | | FastLearner0