Our parent company has included their sitemap links in our robots.txt file - will that have an impact on the way our site is crawled?
-
Our parent company has included their sitemap links in our robots.txt file. All of their sitemap links are on a different domain and I'm wondering if this will have any impact on our searchability or potential rankings.
-
While that is kind of odd to do, it should not have any effect on your site's search engine rankings or visibility. Simply providing a sitemap file is not going to help or hurt rankings in any way. It will help them discover and crawl pages. That's all.
A sitemap file is not required at all, and in fact some sites don't have them--they rely on Google's spiders to crawl their site through the links on their site. The sitemap is only a tool to help them crawl.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is Link equity / Link Juice lost to a blocked URL in the same way that it is lost to nofollow link
Hi If there is a link on a page that goes to a URL that is blocked in robots txt - is the link juice lost in the same way as when you add nofollow to a link on a page. Any help would be most appreciated.
Intermediate & Advanced SEO | | Andrew-SEO0 -
Open Site Explorer - Spam analysis: need help with inbound links... from my site!
hallo, reading my spam analysis report from open explorer, I found somenthing I don't understand (please see attached image): The long list of links inside the red rectangle are inbound links with a spam score of 5 coming from my same site. How is that possible? Should I remove those links? Also , I see that many of those links are links present in the top navigation bar (about page, home page, service description etc.) or in the sidebar section of the website (categories, recent posts, recent comments). Should I treat them differently? Thank you for your time.
Intermediate & Advanced SEO | | micvitale0 -
What happens if one remove the disavow file from a non penalised site
What happens if one remove the disavow file from a site that has not received a manual penalty from Google. Although the site did suffer from a drop in traffic and rankings.
Intermediate & Advanced SEO | | Taiger0 -
What things, that we might overlook, help retain link juice on the site?
Since subscribing to Moz, I have been focussing alot on some of the more technical aspects of SEO. The current thing I am finding interesting is stopping link juice leaks. Here are a selection of some of the things I have done: I have cloaked my affiliate links - see http://yoast.com/cloak-affiliate-links/ Removed some html coded social share links within the theme, and replaced with javascript plugin (http://wordpress.org/plugins/flare/) Used the Moz toolbar to view as Google, to see what google is seeing. Removed some meta links at the bottom of blog posts (author etc) that were duplicated. Now, I don't intend to go over the top with this, as links to social accounts on each page are there to encourage engagement etc, but are there any things you may have come across \ tips that people may have overlooked but perhaps should look out for? As example as some of the things that might be interesting to discuss: Are too many tags, categories bad? Do you index your tag, date archive pages? Does it matter?
Intermediate & Advanced SEO | | Jonathan19790 -
If I only Link to Page via Sitemap, can it still get indexed?
Hi there! I am creating a ton of content for specific geographies. Is it possible for these pages to get indexed if I only put them in my sitemap and don't link to them through my actual site (though the pages will be live). Thanks!
Intermediate & Advanced SEO | | Travis-W
Travis0 -
Meta No INDEX and Robots - Optimizing Crawl Budget
Hi, Sometime ago, a few thousand pages got into Google's index - they were "product pop up" pages, exact duplicates of the actual product page but a "quick view". So I deleted them via GWT and also put in a Meta No Index on these pop up overlays to stop them being indexed and causing dupe content issues. They are no longer within the index as far as I can see, i do a site:www.mydomain.com/ajax and nothing appears - So can I block these off now with robots.txt to optimize my crawl budget? Thanks
Intermediate & Advanced SEO | | bjs20100 -
What's the best way to manage content that is shared on two sites and keep both sites in search results?
I manage two sites that share some content. Currently we do not use a cross-domain canonical URL and allow both sites to be fully indexed. For business reasons, we want both sites to appear in results and need both to accumulate PR and other SEO/Social metrics. How can I manage the threat of duplicate content and still make sure business needs are met?
Intermediate & Advanced SEO | | BostonWright0 -
Will this internal linking feature cause canonicalization issues?
This is a canonicalization type question, so I believe it should be a pretty straightforward answer. I just haven't had much experience with using the canonical tag so I felt I should ask so I don't blow up my site 🙂 Ok, let's say I have a product page that is at: - www.exampledomain.com/products/nameofproduct Now on that page I have an option to see all of the specs of the product in a collapsible tab which I want to link to from other pages - So the URL to this tab ends from other pages ends up being: - www.exampledomain.com/products/nameofproduct?=productspecs This will link to the tab and default it to open when someone clicks that link on another page. Correct me if I'm wrong, but if I understand canonicalization correctly I believe creating this link is going to cause a duplicate page that has the opportunity to be indexed and detract from our SEO to the main product page. My question is... where do I put the "rel=canonical" tag to point the SEO value back to the main page since the page is dynamically generated and doesn't have its own file on the server? - or do even need to be concerned with this? Feel free to correct me if I'm wrong on any of the above. Like I said - this is something I am fairly familiar with how it works, but I haven't had much experience with using. Thanks!
Intermediate & Advanced SEO | | CodyWheeler0