How to activate Page Fetching and Rendering
-
We have a coupons and deal website. Coupons are added and removed from the website on a daily basis. But crawler isn't crawling it that often. Lately we started fetching and rendering the page, but that is a time taking task as we have more than 500 stores with coupons.
So, I was looking for some API or some method using which the crawler would crawl the website as defined. Suppose "x" store page should be crawled every alternate day as we daily update the coupons there, whereas "y" store coupons are update fortnightly so , they can be crawled weekly.
Can somebody suggest me something..
-
Hi Anukul
As far as I am aware, it is not possible to fetch and render via an API - however you can double-check the documentation here: https://developers.google.com/webmaster-tools/
It sounds like what you really need to work on perhaps is Crawl Optimization. If Google thinks the site is important enough, they will crawl it fast enough to catch all of the changes.
AJ Kohn wrote and excellent post on it here which I highly encourage you check out. Here are a few other resources on getting Google to crawl your site faster:
- https://blog.kissmetrics.com/googlebot-optimization/
- http://moz.com/blog/seo-log-file-analysis-part-2
- http://googlewebmastercentral.blogspot.com/2009/08/optimize-your-crawling-indexing.html
Hope those help!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pricing value pages
We have the main pricing page here: https://www.eginnovations.com/product/pricing Then depending on what you click, you'll be taken to the appropriate form on one of these pages: https://www.eginnovations.com/product/request-quote?pricetype=audit https://www.eginnovations.com/product/request-quote?pricetype=saas https://www.eginnovations.com/product/request-quote?pricetype=perpetual https://www.eginnovations.com/product/request-quote?pricetype=subscription How should I handle these? Noindex, follow? Set a canonical? I keep getting notifications that these are duplicate content, but it's just a way to keep the form fills organized. Thanks for your help!
Technical SEO | | eGInnovations1 -
Overdynamic Pages - How to Solve it?
Hi everyone, I'm running a classified real estate ads site, where people can publish their apartment or house they want to sell, so we use multiple filters to help people find what they want. Lately we added multiple filters to the URL to make the search more precise, things like: Prices (priceAmount=###) Bedrooms (BedroomsNumber=2) Bathrooms (BathroomsNumber=3) TotalArea (totalArea=1_50) Services (Elevator, CommonAreas, security) Among other Filters so you see the picture, all this filters are on the URL so that people can share their search on multiple social media, that makes two problems for moz crawl: Overdynamic URLs Too long URLs Now what would be a good solution for this 2 problems, would a canonical to the original page before the "?" would be ok? Example:
Technical SEO | | JoaoCJ
http://urbania.pe/buscar/venta-de-propiedades?bathroomsNumber=2&services=gas&commonAreas=solarium The problem I have with this solution is that I also have a pagination parameter (page=2), and I'm using prev and next tags, if I use a such canonical will break the prev and next tag? http://urbania.pe/buscar/venta-de-propiedades?bathroomsNumber=2&services=gas&commonAreas=solarium&page=2 Also thinking if adding a noindex on pages with paramters could also be an option. Thanks a lot, I'm trying to address this issues.0 -
Blog Page Titles - Page 1, Page 2 etc.
Hi All, I have a couple of crawl errors coming up in MOZ that I am trying to fix. They are duplicate page title issues with my blog area. For example we have a URL of www.ourwebsite.com/blog/page/1 and as we have quite a few blog posts they get put onto another page, example www.ourwebsite.com/blog/page/2 both of these urls have the same heading, title, meta description etc. I was just wondering if this was an actual SEO problem or not and if there is a way to fix it. I am using Wordpress for reference but I can't see anywhere to access the settings of these pages. Thanks
Technical SEO | | O2C0 -
Is There Google 6th page penalty?
My site have keyword domain but my page doesnt up or down at 6th page on search results. And my main page doesnt show 6th page too my alt pages. So what can i do for this penaly? Thanks for your help
Technical SEO | | iddaasonuclari0 -
Banned Page
I have been using a 3rd party checker on indexed pages in google. It has shown several banned pages. I type the page in and it comes up. But it is nowhere to be found for me to delete it. It is not in the wordpress pages. It also shows up in the duplicate content section in my campaigns in moz.com. I can find the page to delete it. If it is banned then I do not want to redirect it to the correct page. Any ideas on how to fix this?
Technical SEO | | Roots70 -
Wordpress Page vs. Posts
My campaigns are telling me I have some duplicate content. I know the reason but not sure how to correct it. Example site here: Bikers Blog is a "static page" referencing each actual "blog post" I write. This site is somewhat orphaned and about to be reconstituted. I have a number of other sites with a similar problem. I'm not sure how to structure the "page" so it only shows a summary of the blog post on the page not the whole post. Permalinks is set as "/%postname%/" I've posted on Wordpress.org with no answer. Since this is an SEO issue I thought maybe someone with WP experience could chime in. Thanks, Don
Technical SEO | | NicheGuy0 -
Too many on page links
Yes this question again. I know it get's asked a lot and I know of a few fixes, but this one I'm having a problem with. So we have a fan gallery on our site which is not only causing duplicate page titles, which I'm thinking we can fix with a canonical, but also too many on page links. The issue is this is on drupal which I have very little experience with and it seems to just be located within the fan galleries section of the site. After looking at a few things I know that no-follow wont be an option since from what I read it wont really work anyway so I was wondering if anyone else has an asnwer. I just read through a million articles trying to find a simular situation and can't seem to find anyone with the same thing. It might have something to do with the plugins the programmers used, but my inexperience with drupal is making this difficult. Thanks guys.
Technical SEO | | KateGMaker0 -
Dealing with 404 pages
I built a blog on my root domain while I worked on another part of the site at .....co.uk/alpha I was really careful not to have any links go to alpha - but it seems google found and indexed it. The problem is that part of alpha was a copy of the blog - so now soon we have a lot of duplicate content. The /alpha part is now ready to be taken over to the root domain, the initial plan was to then delete /alpha. But now that its indexed I'm worried that Ill have all these 404 pages. I'm not sure what to do.. I know I can just do a 301 redirect for all those pages to go to the other ones in case a link comes on but I need to delete those pages as the server is already very slow. Or does a 301 redirect mean that I don't need those pages anymore? Will those pages still get indexed by google as separate pages? Please assist.
Technical SEO | | borderbound0
