Any idea why pages are not being indexed?
-
Hi Everyone,
One section on our website is not being indexed. The product pages are, but not some of the subcategories. These are very old pages, so thought it was strange. Here is an example one one:
https://www.moregems.com/loose-cut-gemstones/prasiolite-loose-gemstones.html
If you take a chunk of text, it is not found in Google. No issues in Bing/Yahoo, only Google. You think it takes a submission to Search Console?
Jeff
-
So I am testing removing some of the restrictions in the robots.txt file and see if that helps as I still can't get it to be indexed.
-
Yeah...it's very close to what I have. I also checked other websites I own with the same category structure and robots.txt file...no issues.
I even checked other subcats on www.moregems.com, and no issues. It seems to be all the pages under "GEMSTONES" that are not being indexed. Any thoughts there?
-
I usually do robots.txt for Magento sites custom. But I did find a good example to use. Check out this site:https://www.magikcommerce.com/blog/set-up-robots-txt-in-magento/
I would edit anything that doesn't fit your site.
Hope this helps!
-
So I submitted https://www.moregems.com/loose-cut-gemstones/prasiolite-loose-gemstones.html and fetched it in Google Search Console a few hours ago. Still not being indexed. I don't see any issues in the robots.txt file. Any thoughts?
-
Hi Nicholas,
I asked him this as well, but do you have any resources for a "good" Magento specific robots.txt file? I want to try updating it, as it has been the same for about 7 years.
The strange thing is the deeper product pages are indexed, but not the subcats.
-
Hi Christian,
Do you have any resources for a recommended Magento robots.txt file? I added this probably 6-7 years ago, and have not updated it since. I can definitely try that.
Jeff
-
Hi Jeff,
In addition to Christian's recommendation (which I would do first), use Google Search Console's Fetch & Render Tool to request your non-indexed pages to Google's index. Sometimes this tool in GSC will have them indexed immediately.
It is not uncommon for deep links or internal pages of internal pages to not be immediately indexed. It is definitely important to use new pages to link our to other pages on your website, and if possible go in and link to your new pages from older (already indexed) pages on your website
Nick White
SEO Services KC
https://seo-kansas-city.com/ -
Hey Jeff,
I just ran a quick scan of the site, it looks like you have a lot of links, pages, and directories being blocked by your robots.txt file: https://www.moregems.com/robots.txt
I would make sure the pages you want to be indexed by search engines are not being blocked in your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Follow no-index
I have a question about the right way to not index pages: With a canonical or follow no-index. First we have a blog page: **Blogpage **
Technical SEO | | Happy-SEO
URL: /blog/
index follow Page 2 blog:
URL: /blog?=p2
index follow
rel="prev" /blog/
el="next" ?=p3 Nothing strange here i guess. But we also have other pages with chance on duplicate content: /SEO-category/
/SEO-category/view-more/ Because i don't want the "view-more" items to be indexed i want to set it on: follow no-index (follow to reach pages). But now the "view-more" also have pagination. What is the best way? Option 1:
/SEO-category/view-more/
Follow no-index /SEO-category/view-more?=p2
Follow no-index
rel="prev" /view-more/
el="next" ?=p3 Option 2: /SEO-category/view-more/
Canonical: /SEO-category/ /SEO-category/view-more?=p2
rel="prev" /view-more/
el="next" ?=p3 Option 3: Other suggests? Thanks!0 -
Is it better to use XXX.com or XXX.com/index.html as canonical page
Is it better to use 301 redirects or canonical page? I suspect canonical is easier. The question is, which is the best canonical page, YYY.com or YYY.com/indexhtml? I assume YYY.com, since there will be many other pages such as YYY.com/info.html, YYY.com/services.html, etc.
Technical SEO | | Nanook10 -
Help! Pages not being indexed
Hi Mozzers, I need your help.
Technical SEO | | bshanahan
Our website (www.barnettcapitaladvisors.com) stopped being indexed in search engines following a round of major changes to URLs and content. There were a number of dead links for a few days before 301 redirects were properly put in place. And now, only 3 pages show up in bing when I do the search "site:barnettcapitaladvisors.com". A bunch of pages show up in Google for that search, but they're not any of the pages we want to show up. Our home page and most important services pages are nowhere in search results. What's going on here?
Our sitemap is at http://www.barnettcapitaladvisors.com/sites/default/files/users/AndrewCarrillo/sitemap/sitemap.xml
Robots.txt is at: http://www.barnettcapitaladvisors.com/robots.txt Thanks!0 -
Lots of Pages Dropped Out of Google's Index?
Until yesterday, my website had about 1200 pages indexed in Google. I did lots of changes: removed low quality content, rewrote passable content to make it better, wrote high quality content, got lots of likes and shares on social networks, etc. Now this morning I see that out of 1252 pages submitted, only 691 are indexed. Is that a temporary situation related to the recent updates? Anyone seeing this? What should I interpret about this?
Technical SEO | | sbrault740 -
Home page indexed but not ranking...interior pages with thin content outrank home page??
I have a Joomla site with a home page that I can't get to rank for anything beyond the company name @ Google - the site works fine @ Bing and Yahoo. The interior pages will rank all day long but the home page never shows up in the results. I have checked the page code out in every tool that I know about and have had no luck....by all account it should be good to go...any thoughts/comments/help would be greatly appreciated. The site is http://www.selectivedesigns.com Thanks! Greg
Technical SEO | | DougHosmer0 -
Should I index my search result pages?
I have a job site and I am planning to introduce a search feature. The question I have is, is it a good idea to index search results even if the query parameters are not there? Example: A user searches for "marketing jobs in New York that pay more than 50000$". A random page will be generated like example.com/job-result/marketing-jobs-in-new-york-that-pay-more-than-50000/ For any search that gets executed, the same procedure would be followed. This would result in a large number of search result pages automatically set up for long tail keywords. Do you think this is a good idea? Or is it a bad idea based on all the recent Google algorithm updates?
Technical SEO | | jombay0 -
Do pages that are in Googles supplemental index pass link juice?
I was just wondering if a page has been booted into the supplemental index for being a duplicate for example (or for any other reason), does this page pass link juice or not?
Technical SEO | | FishEyeSEO0 -
On Page 301 redirect for html pages
For php pages youve got Header( "HTTP/1.1 301 Moved Permanently" );
Technical SEO | | shupester
Header( "Location: http://www.example.com" );
?> Is there anything for html pages? Other then Or is placing this code redirect 301 /old/old.htm http://www.you.com/new.php in the .htaccess the only way to properly 301 redirect html pages? Thanks!0
