Single Page on my client's website is not crawling and indexing new changes. What could be the possible reason?
-
I made several changes on client's website on different pages, changed titles, add content on few pages, moved blog from subdomain to sub directory. Everything is crawled but there is one page on the website (not part of the blog) that isn't getting crawled in Google and picking up changes. The last crawl of the website is 2 days back whereas that page was last crawled on 30th sep.
I just wanted to know the possible reasons and has anyone encountered this before?
-
Hi Moosa, do you have an update on this issue?
-
Things I check:
- header check (I've had pages appear 200 OK in webconfs and then massive errors in Browseo so I often check both.)
- Screaming Frog check - does the direct URL start a proper crawl?
- Make sure it's not blocked in WMT under removed pages (some website owners and/or bad SEOs try to demote sitelinks by removing the URL.)
- Meta tags & robots.txt both of which I assume you already checked.
- Duplicate content being indexed. Take a random sentence & throw it in quotes on Google. If a 100% dupe version comes up on their facebook page, archive.is, or some directory it may just need a content refresh to push through.
If you check all 5 of those off, then I'd definitely like to see the URL.
-
If that doesn't work, please share the page URL.
-
Thanks Drik,
I have double check the accessibility of page to crawlers and bots and pages does have decent amount of internal and external links pointing to the page.
Thanks for the Google fetch idea, i am going to try it to see if it works and update you accordingly.
-
Hi
I guess you have already checked that the page is accessible for crawlers and has sufficient internal links to be discovered by the bots (if not - try crawling the site with Screaming Frog)
If it's only one page that isn't picked up you could try a fetch like google & submit to index to force the page being indexed.
It could be that Google considers this page as less important and crawls with lower frequency compared to the other pages (to deep in the site, to few links, not enough content,...)
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New Page Not ranking?
One of this client's top keyword is "oak beams". They already rank well in the UK for other related terms like "reclaimed oak beams" at /reclaimed-oak-beams/ and "air dried oak beams" at /air-dried-oak-beams/ We have created a page at /oak-beams/ but this page ranks nowhere? Instead the reclaimed oak beams or air dried oak beams page ranks for the term "oak beams". Any ideas why Google is swapping between those pages and not choosing the /oak-beams/ page? A few notes are that the /oak-beams/ page is newest page on the site and yes I know there are no links pointing to it but there are no links pointing to the other pages either?
On-Page Optimization | | Marketing_Today0 -
Tips and Tricks on yoast's video seo upgrade!
Hello Guys, I am using Yoast for several websites, and videos. I have a video sitemap and for all videos i added: 1. Titles 2. Descriptions 3. Tags My question is: Is the there any way or do anything to rank these videos higher?
On-Page Optimization | | DexSmart0 -
Does Google use 302's to pass value to the target page?
Hi, I've received the below advice, is this correct? Throughout the site, the 302 (moved temporarily) status code is used for redirects, which Google will use to pass value to the target page. Is this correct? I was under the impression a 301 was used to pass value to the target page? Could someone explain the difference between a 301 and a 302, I'm not 100% sure. Thanks, Nathan
On-Page Optimization | | Heehaw0 -
When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following: www.mywebsite.com/blog/title-of-post www.mywebsite.com/blog/tag/tag1 www.mywebsite.com/blog/tag/tag2 www.mywebsite.com/blog/category/categoryA etc My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
On-Page Optimization | | williammarlow0 -
On page "F" and I changed my tags long ago to match everything and still F?
My on-page analysis shows that my title tags are not supporting my keyword..........but they are! My title tag has my EXACT keyword phrase. What gives? Keyword is "San Diego Party Bus" Title is "San Diego Party Bus | xoxoxox | xoxoxoxo" F grade! ????????
On-Page Optimization | | DrewSpinoso0 -
Index.php getting Duplicate page content.
I am quite new to SEO and have now got my first results. I am getting all my index.php pages returned as Duplicate page content. ie: blue-widgets/index.php
On-Page Optimization | | ivoryred
blue-widgets/ green-widgets/large/index.php
green-widgets/large/ How do solve this issue?0 -
Creating Authority and choosing URL's
Creating Domain Authority and choosing URL's: A: What is better if you want to get higher Domain Authority? Choose keyword.domain.com or www.domain.com/keyword when other sites link to it? B: And for Page Authority? Choose keyword.domain.com or www.domain.com/keyword? Thanks!
On-Page Optimization | | HMK-NL0 -
Using meta robots 'noindex'
Alright, so I would consider myself a beginner at SEO. I've been doing merchandising and marketing for Ecommerce sites for about a year and a half now and am just now starting to attempt to apply some intermediate SEO techniques to the sites I work on so bear with me. We are currently redoing the homepage of our site and I am evaluating what links to have on it. I don't want to lose precious link juice to pages that don't need it, but there are certain pages that we need to have on the homepage that people just won't search for. My question is would it be a good move to add the meta robots 'noindex' tag to these pages? Is my understanding correct that if the only link on the page is back to the homepage it will pass back the linkjuice? Also, how many homepage links are too many? We have a fairly large ecommerce site with a lot of categories we'd like to feature, but don't want to overdo the homepage. I appreciate any help!
On-Page Optimization | | ClaytonKendall0