How to activate Page Fetching and Rendering
-
We have a coupons and deal website. Coupons are added and removed from the website on a daily basis. But crawler isn't crawling it that often. Lately we started fetching and rendering the page, but that is a time taking task as we have more than 500 stores with coupons.
So, I was looking for some API or some method using which the crawler would crawl the website as defined. Suppose "x" store page should be crawled every alternate day as we daily update the coupons there, whereas "y" store coupons are update fortnightly so , they can be crawled weekly.
Can somebody suggest me something..
-
Hi Anukul
As far as I am aware, it is not possible to fetch and render via an API - however you can double-check the documentation here: https://developers.google.com/webmaster-tools/
It sounds like what you really need to work on perhaps is Crawl Optimization. If Google thinks the site is important enough, they will crawl it fast enough to catch all of the changes.
AJ Kohn wrote and excellent post on it here which I highly encourage you check out. Here are a few other resources on getting Google to crawl your site faster:
- https://blog.kissmetrics.com/googlebot-optimization/
- http://moz.com/blog/seo-log-file-analysis-part-2
- http://googlewebmastercentral.blogspot.com/2009/08/optimize-your-crawling-indexing.html
Hope those help!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
3,511 Pages Indexed and 3,331 Pages Blocked by Robots
Morning, So I checked our site's index status on WMT, and I'm being told that Google is indexing 3,511 pages and the robots are blocking 3,331. This seems slightly odd as we're only disallowing 24 pages on the robots.txt file. In light of this, I have the following queries: Do these figures mean that Google is indexing 3,511 pages and blocking 3,331 other pages? Or does it mean that it's blocking 3,331 pages of the 3,511 indexed? As there are only 24 URLs being disallowed on robots.text, why are 3,331 pages being blocked? Will these be variations of the URLs we've submitted? Currently, we don't have a sitemap. I know, I know, it's pretty unforgivable but the old one didn't really work and the developers are working on the new one. Once submitted, will this help? I think I know the answer to this, but is there any way to ascertain which pages are being blocked? Thanks in advance! Lewis
Technical SEO | | PeaSoupDigital0 -
Last Part Breadcrumb Trail Active or Non-Active
Breadcrumbs have been debated quite a bit in the past. Some claim that the last part of the breadcrumb trail should be non-active to inform users they have reached the end. In other words, Do not link the current page to itself. On the other hand, that portion of the breadcrumb would won't be displayed in the SERPS and if it was may lead to a higher CTR. Foe example: www.website.com/fans/panasonic-modelnumber panasonic-modelnumber would not be active as part of the breadcrumb. What is your take?
Technical SEO | | CallMeNicholi0 -
From page 1th to page 18th @ Google
Hello Mozzers! I have a question, you may help.. How may it be possible that a page ranking well (1th result) goes from 1th result to the 18th page just in 1 day? It doesnt seem to be any kind of penalization.. I now had all suspicious outgoing links to be nofollow (they were not before), this may be a cause .. (?) Do you have any other suggestion? Thanks
Technical SEO | | socialengaged0 -
Should We Index These Category Pages?
Currently we have marked category pages like http://www.yournextshoes.com/celebrities/kim-kardashian/ as follow/noindex as they essentially do not include any original content. On the other hand, for someone searching for Kim Kardashian shoes, it's a highly relevant page as we provide links to all the Kim Kardashian shoe sightings that we have covered. Should we index the category pages or leave them unindexed?
Technical SEO | | Jantaro0 -
Google Page speed
I get the following advice from Google page speed: Suggestions for this page The following resources have identical contents, but are served from different URLs. Serve these resources from a consistent URL to save 1 request(s) and 77.1KiB. http://www.irishnews.com/ http://www.irishnews.com/index.aspx I'm not sure how to fix this the default page is http://www.irishnews.com/index.aspx, anybody know what need to be done please advise. thanks
Technical SEO | | Liammcmullen0 -
Inter-link Optimization for Pages
Hi Guys, How much interlinking is okay for between pages? Can i start link my internal pages like Wikipedia does? 20,30 interlink per page? OR is it all depends on Page Authority? Can i go inter-link crazy on 1 year old legitimate site? Big Thanks?
Technical SEO | | DigitalJungle0 -
404 - page authority?
If in open site explorer my 404 pages have a higer page authority - what benefit would i see in rankings if I 301 redirected those pages to the right page. For example www.site.com/widget is a 404 but has authority according to open site explorer - but the page i see in the serps is www.site.com/widget/ with the / at the end. so what benefit would i see in rankings if I 301 redirected those pages to the right page?
Technical SEO | | DavidS-2820610 -
Lost page rank, am I in the sandbox?
Hi, Our page rank was 5 for the last 3 months. Before it was 4 for a long time. Today our page rank dropped to 0 and I have no idea why.. We didn't buy any links or did anything wrong.. I also checked every keyword rankings we are on but we didn't loose our spot in any keyword.. Is it a short term thing, or am I in the sandbox? How can I fix this problem or avoid from a worse situation..? www.gourmetmarketing.net We just changed the site template few times in last two months.. Thanks
Technical SEO | | onurkiyak0
