Does Frequency of content updates affect likelyhood outbound links will be indexed?
-
I have several pages on our website with low pr, that also themselves link to lots and lots of pages that are service/product specific. Since there are so many outbound links, I know that the small amount of PR will be spread thin as it is. My question is, if I were to supply fresh content to the top level pages, and change it often, would that influence whether or not google indexes the underlying pages? Also if I supply fresh content to the underlying pages, once google crawls them, would that guarantee that google considers them 'important' enough to be indexed"
I guess my real question is, can freshness of content and frequency of update convince google that the underlying pages are 'worthy of being indexed', and can producing fresh content on those pages 'keep google's interest', so to speak, despite having little if any pagerank.
-
Hello Ilya,
There are several good responses here, and I think some of them would depend on how large your site is and what types of pages they are. Judging by your URL example below, I'm guessing it is real estate related or at least that you have localized pages in different geographic areas.
You have a few issues here. First, this video might help, but it is sort of outdated and misleading in some ways. There may not be a set limit (i.e. we're only going to index 10k pages) but how much of your site gets indexed, and how often it gets crawled is based largely on the quality of your site (assuming all other factors are there, such as sitemaps and crawlable navigation, etc...). And the quality of your site depends on many, many different factors. Of course the two most important for this discussion would probably be uniqueness/usefulness of the content, and the amount of links the site and sections of the site, as well as the deep pages have.
The more links you can get into those deep pages, the more likely it is that Google is going to crawl more often, and index those pages. You said you "can't" get links into those pages. If you can't get links into them, they probably aren't "quality" and therein lies your problem.
If by "can't" you just mean there isn't enough time in the day for you to build links into ALL of these pages, you can still build links into as many as you can. This will get the bots crawling down to that level of your site more often, and make it more likely that this level of your site will be indexed.
Here is another useful link, although it is dated as well:
http://www.seomoz.org/blog/googles-indexation-capHaving fresh content (with a fresh "last modified" date) usually does, in my experience, entice Googlebot to come back more often. Does that translate into "indexing" more pages? I don't know. But I do know that having better content and more links into those inner pages does translate into more indexation, and not just for the pages linked to externally, but for that entire section/folder/directory of your site.
Consider user-generated content on those pages if you can. A lot of VERY popular review and realestate sites' deep pages would go unindexed without it.
-
We shouldn't confuse a query that deserves freshness (QDF) with enticing Google to recrawl a page or set of pages by giving them fresh content. Maybe I read your response wrong, but those are two different things. QDF would apply, for instance, if you were writing an article right now about the nuclear disaster in Japan; not if you were updating a page from three years ago about how to lose weight after pregnancy, or how to optimize a webpage.
-
From my experience, adding fresh content on a regular basis, even when the pages are rather empty, will make Google crawl more and more your website. As crawl budget gets bigger, deeper pages will be crawled.
Although I never worked on a similar case to yours, I would suggest adding fresh content on a regular basis and link those new pages on the homepage to get them crawled ASAP. Put internal links to the pages you want to be crawled in those new pages if they are revelant.
-
Not as much. You may have to engineer some process for feed generation. The idea is to have the content in RSS and help it propogate through stuff like ping.
-
It can, as Rand has said in the past, results deserve freshness, that is, results seem to always include a few such pages.
-
saibose...do you think a service like linklicious? (link->rss) would work?
-
the 100 links is more of a guideline and not a strict rule as such. Your 1st objective should be to enable the page to be indexed. If Query Deserves Freshness(QDF) algorithms in Google will eventually index your URL. Its a matter of time with you linking to that page from atleast 1 page.
My advice would be to link it from more pages (if possible) and keep the content fresh.
Maybe you can even try the RSS idea as well.
-
I guess it would depend a little how you're doing it, however the best way to get Google to crawl your product pages is to get links directly to them from other sites that are being crawled often/ have authority. I would also suggest creating a (XML) sitemap and submit it to them if you haven't already.
If all your links are coming to your homepage (not uncommon in smaller sites) then Google's going to usually enter your site that way and if there's a lot of links on the homepage and the site only has a little authority then it has to prioritise how many and which pages to visit.
Having regular content updates may get Google to change which pages it crawls at any one time, though some of your other pages may then have longer cache dates.
Ultimately if your site structure is good enough then you really need to work on building links to the product pages to regularly 'convince' Google to crawl them. Though adding relevant content is one way of doing this
-
Thank you guys.
Anthony, I am not sure I agree; indexing and crawling are 2 different things. I guess that is really what I'm getting at here. I can force google to crawl my whole site daily (or almost daily) with rss feeds, sitemaps, proper structure, frequent updates, etc....but WILL that freshness of content force google to go hm....despite the page being very insignificant, it might be important enough to go into my index.
Saibose, unfortunately i'm well beyond the 100 link limit....I am noticing quite a bit of the pages that ARE indexed, ARE ranking since they're well optimized through on-page and they are targeting extremely long-tail keyphrases. So my main goal is to convince goal to index these pages because once I do, they will rank.
What I have done so far:
1. Made sure that the page is easily accessible from at least 1 page on the website
2. Create a sitemap (proper sitemap index and several underlying sitemap files).
3. Submitted the sitemaps and increase google crawl rate; (I noted google is crawling around 1700 pages/day on my site.
4. Made sure that the page is at most 3 levels deep. (site/state/city) (we'er talking about city level pages)
5. created proper urls (/site/state/city)
I think maybe I misspoke. I am not doubting that google will 'crawl' the page. What I am asking is if I can't link externally to it, and the internal page rank passed is very small, will adding fresh content and making google think that the page gets updated frequently convince google to index it? Does frequent crawling finally force indexing or is it possible google may say "no matter how often you update this page, its just NOT important enough for me to index it," if noone links to it outside your site.
-
I think you are getting at the concept of continually updating the content on a few pages of your site to make sure they are indexed by google. If the page is not indexed already, that means it likely isn't being crawled by google at all so changing the content on the page won't make much of a difference.
Instead, make sure the page you want indexed is easily found within the website's internal linking structure, preferably only a handful of clicks away from the homepage. An even better way to make sure the page is indexed is to get a few external links pointed at it. If you are simply trying to achieve indexation and not expecting the page to rank high in the SERPs, something as easy as bookmarking the site to a few websites and tweeting it once or twice will probably get the job done.
As for your comment on whether or not google will consider your page 'important' enough to be indexed, I don't think you will have a problem with that as long as you are writing unique content.
-
The problem is very common for content heavy websites where content lies somewhere way down the hiearchy.
I am considering or assuming a few things here:
1. The webpage you are referring to is already crawled atleast once.
2. It is accessible from atleast one link on your homepage
3. It does not have a huge number of outbound links ..that is, around 100(within and outside your domain).
Your 1st task should be to get Google to crawl the page (s)
1. get a tool like gsite crawler and crawl your entire website. Create and submit a XML sitemap of your website to Google webmaster tools. Create links from your pages that are already indexed to this page (pages). That way, Google bot will find its way eventually.
2. Update fresh content on the page. Create a RSS feed of the content updates very frequently and serve it up front on the homepage or an important page of your website (which ranks well in Google).
All said, you have to wait and watch. There is no way you can forcefully ask Google to crawl your webpage. Also, updating your homepage content (just text with no link to your deep pages) wouldnt help in speeding up the process. But, its a good practice to keep your homepage content fresh so that Google bots visit your website regularly and you get Google love.
Hope that answers your question.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Us eof outbound links and referencing sources
Hello everyone, I am writing weekly articles for a website. At times, I am referring to data/statistics from other articles/studies. My question is - how do you include links/references to this material? For example, I notice that some articles cite their sources at the end/bottom. But - some include outbound links to the original source throughout the main content of the article- so that when you click on the link it takes you to the correct web page. So far, I am including a 'Sources' section at the end of my article that includes title, author, date. I am doing this to stop people clicking off my article to check out the link! Do I need a specific sources section if I cite my source as I go along? For example, 'In their 2017 study of life insurance, the Association of British Insurers found that 70% of Brits are without life insurance'. (made up statistic! Different web writers seem to do different things and it's a little confusing!! Sorry if this is rather long -winded! Any help with this would be much appreciated. Thanks guys, Clare
On-Page Optimization | | ClareO0 -
Wordpress #comment links?
We just started our trial account and have the results from our first Moz Pro Site Crawl. It's showing that we have a large portion of our pages have 'Too Many Links' and I'm trying to determine exactly what this means and how to fix it. The article referenced is from 2011 and doesn't fully address what I'm looking for. Here are a few questions: 1. Can we lower the 'link count' by adding a 'no follow' or does too many links, count links regardless? The question being, is the only way to solve this by removing links or are their no follow or no index options that will prevent us from having this issue moving forward? 2. Comment Links: Our site is in Wordpress and I just recognized that each of the comment links are followed: https://screencast.com/t/b0CIKVafWw. These aren't links from our users, rather these are links within Wordpress and are structured like this: https://mysite.com/blog-post/#comment-6970. From my screenshot you can see they are highlighted as 'followed links'. Is there a setting within Wordpress to turn this off or is there another option I should consider? Should we just make these no index, no follow links? Will that solve the 'Too many links' problem? I searched through the Q&A's and couldn't find an answer directly to my question. Most were around people leaving links in the comments section, which isn't what I'm looking for. Thank you for any help you can provide.
On-Page Optimization | | FabulesslyFrugal0 -
Product content length & links within product description
Hello, I have questions regarding content length and links within descriptions. With our ecommerce site, we have thousands of products, each with a unique description. In the product description, I have links to the parent category and grandparent category (if it has one) in the main product text which is generally about 175 words. Then I have a last paragraph that's about 75 words that includes links to our main homepage and our main product catalogue page. Is the content length long enough? I used to use text that was 500 words, and shortening it I still rank when launching new products, so I don't think an increase in text length will have any additional benefit. I do see conflicting information when I do searches, with some people recommending a minimum of 300 words and some saying to try and go a 1000 for category pages. In regards to the links, I noticed a competitor has stopped following this format, so I'm unsure if I should keep going too. Is it too many links to have each of the products link back to the main catalogue and homepage? Is it good to have links with anchor text to the categories a product is in? There are breadcrumbs on the page with these links already. There are already have heaps of links on our pages (footer, and a right sidebar with image links to relevant categories), so my pages do get flagged for too many links. Thanks!
On-Page Optimization | | JustinBSLW0 -
Duplicate Content on our own website
Our website sells tickets for events. We also have an news articles section with information about events / artists / venues. From time to time we release a product page and a related news article on a separate page. Some of the content in the news article would be perfect for our product page. Essentially its our product page we want too rank. Would it harm our SEO if we had some of the same content on both of these pages?
On-Page Optimization | | Alexogilvie0 -
Duplicate links from forum what to do?
After a crawl it found over 5k errors and over 5k warnings. Those are: Duplicate page content; Duplicate page title; Overly-Dynamic URLs; Missing Meta descr; Title Element too long. All those come from domain.com/forum/ I don't need SEO on forum so what should I do? What could be an easy solution to this? No index? No follow? Please help
On-Page Optimization | | OVJ0 -
Google Indexing
Hi, We recently launched a new version of our site on the Magento platform. I submitted a new sitemap and on the first crawl only 7 pages out of 132 were indexed...a few days later and we now have 107 indexed (phew). My question is this....how on earth do i find out which pages are indexed and more importantly not indexed? For all i know they might be really important ones so I need to be able to identify the missing pages so i can work on getting them indexed. Nic
On-Page Optimization | | nicc19760 -
Internal linking structure
howdy SEO moz people, I had a quick question about internal linking. I am using the WP platform for my website and on one of the widgets I have a category widget that shows all the pages, (on every page). I wanted to know that when if i make a new post or page does it matter if the internal anchor texts are on the content itself or is it suffice to have it on the side bar within the widget? Basically what im asking is does it matter if i have anchor text links on the content itself or is it ok to have it anywhere on the page. and if it does matter to have it on the content, please help out on any suggestions on where to place them. thanks
On-Page Optimization | | david3050 -
Too many on page links
I'm having trouble interpreting this data. It says several of my blog pages have too many on page links, some as high as 140 and there is no example of a blog post that they are referring to. What am I missing? I never post more than a handful (5-7) in our 600-1000wd blogs. When I drill down, it doesn't give me very much information except "Found over 41 years ago" off to the right. When I click on the "too many on page links" URL, it provides a long list of website pages that are renamed with the blog name. huh? A lot of this stuff isn't very intuitive, SEOMoz.
On-Page Optimization | | amandahx20