Sitemap & noindex inconstancy?
-
Hey Moz Community!
On a the CMS in question the sitemap and robots file is locked down. Can't be edited or modified what so ever.
If I noindex a page in the But it is still on the xml sitemap... Will it get indexed?
Thoughts, comments and experience greatly appreciate and welcome.
-
The site map is an indication to Google to crawl those pages, there are instances where people have meta tags with noindex, follow and would list them in their sitemaps so that Google will crawl all the links listed on the page but not index the page itself.
The meta tags or headers on your page will be the signal to Googlebot on how to handle that page regardless of your sitemap and whats on it.
-
It will get crawled, but it will not be indexed. Basically the crawler will get the page, see that and not index it. It could follow if you put noindex, follow. Or it cannot follow if you put noindex, nofollow.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No Follow & Rel Canon for Product Filters
Our site uses Canonicals to address duplicate content issues with product/facet filtering. example: www.mysite.com/product?color=blue Relcanon= www.mysite.com/product However, our site is also using no follow for all of the "filters" on a page (so all ?color=, etc. links are no follow). What is the benefit of utilizing the no follow on the filters if we have the rel canon in place? Is this an effort to save crawl budget? Are we giving up possible SEO juice by having the no follow and not having the crawler get to the canonical tag and subsequently reference the main page? Is this just something we just forget about? I hope we're not giving up SEO juice by
Technical SEO | | Remke0 -
Sitemaps, 404s and URL structure
Hi All! I recently acquired a client and noticed in Search Console over 1300 404s, all starting around late October this year. What's strange is that I can access the pages that are 404ing by cutting and pasting the URLs and via inbound links from other sites. I suspect the issue might have something to do with Sitemaps. The site has 5 Sitemaps, generated by the Yoast plugin. 2 Sitemaps seem to be working (pages being indexed), 3 Sitemaps seem to be not working (pages have warnings, errors and nothing shows up as indexed). The pages listed in the 3 broken sitemaps seem to be the same pages giving 404 errors. I'm wondering if auto URL structure might be the culprit here. For example, one sitemap that works is called newsletter-sitemap.xml, all the URLs listed follow the structure: http://example.com/newsletter/post-title Whereas, one sitemap that doesn't work is called culture-event-sitemap.xml. Here the URLs underneath follow the structure http://example.com/post-title. Could it be that these URLs are not being crawled / found because they don't follow the structure http://example.com/culture-event/post-title? If not, any other ideas? Thank you for reading this long post and helping out a relatively new SEO!
Technical SEO | | DanielFeldman0 -
Recovering from Sitemap Issues with Bing
Hi all, I recently took over SEO efforts for a large e-commerce site (I would prefer not to disclose). About a month ago, I began to notice a significant drop in traffic from Bing and uncovered in Bing Webmaster Tools that three different versions of the sitemap were submitted and Bing was crawling all three. I removed the two out of date sitemaps and re-submitted the up to date version. Since then, I have yet to see Bing traffic rebound and the amount of pages indexed by Bing is still dropping daily. During this time there has been no issue with traffic from Google. Currently I have 1.3 million pages indexed by Google while Bing has dropped to 715K (it was at 755K last week and was on par with Google several months ago). I know that no major changes have been made to the site in the past year so I can't point to anything other than the sitemap issue to explain this. If this is indeed the only issue, how long should I expect to wait for Bing to re-index the pages? In the interim I have been manually submitting important pages that aren't currently in the index. Any insights or suggestions would be very much appreciated!
Technical SEO | | tdawson090 -
Sitemap Contains Blocked Resources
Hey Mozzers, I have several pages on my website that are for user search purposes only. They sort some products by range and answer some direct search queries users type into the site. They are basically just product collections that are else ware grouped in different ways. As such I didn't wants SERPS getting their hands on them so blocked them in robots so I could add then worry free. However, they automatically get pulled into the sitemap by Magento. This has made Webmaster tools give me a warning that 21 urls in the sitemaps are blocked by robots. Is this terrible SEO wise? Should I have opted to NOINDEX these URLS instead? I was concerned about thin content so really didnt want google crawling them.
Technical SEO | | ATP0 -
New pages need to be crawled & indexed
Hi there, When you add pages to a site, do you need to re-generate an XML site map and re-submit to Google/Bing? I see the option in Google Webmaster Tools under the "fetch as Google tool" to submit individual pages for indexing, which I am doing right now. Thanks,
Technical SEO | | SSFCU
Sarah0 -
Image & Video Sitemaps - Submitted vs. Indexed
Hi Mozzers, I have read all the relevant blogs from media indexing experts like Phil Nottingham and have followed Google's best practice as well as advice from similar discussions on here. We have submitted video and image sitemaps to WT, and the image sitemap has 33 indexed from 720 submitted images, and the video 170 indexed from 738 submitted. With the image sitemap the number (33) has remained steady while the submitted has grown by over 100 in the last month. The video has shown signs of indexing new videos however but still not the amount that were submitted. Thus far, I have followed the guidelines sitemap structure as per Google. We are using Cloudfront so I have added and verified our cloudfront server in the same WT account. If anyone has any advice, it would be most appreciated. There is no duplicate content and the robots.txt is not blocking anything within the sitemap. Image sitemap: view-source:http://www.clowdy.com/sitemap.images.xml
Technical SEO | | Morrreau0 -
Do I need a link to my sitemap?
I have a very large sitemap. I submit it to both Google and Bing, but do I need a link to it? If someone went there it would probably lock their browser. Is there any danger of not having a link if I submit it to Google and Bing?
Technical SEO | | EcommerceSite0 -
Videos for SEO & Profits
Hello, I'm in the middle of developing a website that will be a tutorial site for SEO, http://universityofseo.com. My plan is to do video tutorials & blog posts to help entry-level SEOs and SMB Owners to help them become familiarized with SEO through quick and easy to watch videos. I eventually want to turn this into a revenue stream through advertisements. I want to know for both SEO and profit reasons, if I should host the videos on youtube and then embed them on my site, or do something like Bits on the Run / Whistia and put ads in the videos that way? I'm not overly obsessed with monetizing the site, but it would be nice to do it, but first and foremost i'm concerned with optimizing the site, having great and actionable content, then monetizing it. I'd appreciate any help on this matter, Zach
Technical SEO | | Zachary_Russell0
