Our parent company has included their sitemap links in our robots.txt file - will that have an impact on the way our site is crawled?
-
Our parent company has included their sitemap links in our robots.txt file. All of their sitemap links are on a different domain and I'm wondering if this will have any impact on our searchability or potential rankings.
-
While that is kind of odd to do, it should not have any effect on your site's search engine rankings or visibility. Simply providing a sitemap file is not going to help or hurt rankings in any way. It will help them discover and crawl pages. That's all.
A sitemap file is not required at all, and in fact some sites don't have them--they rely on Google's spiders to crawl their site through the links on their site. The sitemap is only a tool to help them crawl.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unknown index.html links coming to my site.
I'm getting a lot of domain/index.html urls on my site which I didn't create initially. We recently transfered to a new site so those links could come from the old site. Does any know how to get a comprehensive list of all the urls that lead to 404?
Intermediate & Advanced SEO | | greenshinenewenergy0 -
What is the best way to add semantic linked data to WordPress?
As a recent Moz subscriber, I'm trying to up my game in terms of inbound marketing. One of the most pressing tasks is to add json-ld across all of my WordPress sites. What is the best way to do this? Should I use the technique set out here: https://moz.com/blog/using-google-tag-manager-to-dynamically-generate-schema-org-json-ld-tags Or should I use one of these plugins? https://en-gb.wordpress.org/plugins/schema/ https://en-gb.wordpress.org/plugins/wp-structuring-markup/ I want to get this right so any guidance would be gratefully received.
Intermediate & Advanced SEO | | treb0r0 -
Large robots.txt file
We're looking at potentially creating a robots.txt with 1450 lines in it. This will remove 100k+ pages from the crawl that are all old pages (I know, the ideal would be to delete/noindex but not viable unfortunately) Now the issue i'm thinking is that a large robots.txt will either stop the robots.txt from being followed or will slow our crawl rate down. Does anybody have any experience with a robots.txt of that size?
Intermediate & Advanced SEO | | ThomasHarvey0 -
Disallow URLs ENDING with certain values in robots.txt?
Is there any way to disallow URLs ending in a certain value? For example, if I have the following product page URL: http://website.com/category/product1, and I want to disallow /category/product1/review, /category/product2/review, etc. without disallowing the product pages themselves, is there any shortcut to do this, or must I disallow each gallery page individually?
Intermediate & Advanced SEO | | jmorehouse0 -
301 from a defunct site due to great link profile
Hi there Would really appreciate your help in dealing with the following scenario: My client is an authority brand in their sector. They were bought end 2011, and a new website was launched under the new owner's brand. For whatever reason no 301 redirects were put in place from the old site to the new site. I am now auditing the new site and the traffic is pitifully low, way lower than they used to enjoy on the old site. The old site is defunct and Google is no longer indexing it. However OSE shows that the link profile of the old site was very good with thousands of good quality links, whilst it is non-existent for the new site. I am thinking that even though Google does not index the old site, we should try and get access and put 301s in place on the old pages to help transfer across all the link juice to boost the new site. Do you agree or am I missing something here? Will page rank be transferred across even though the old site is dead? What else could we do? Would change of domain in WMT help? Although how would that work for a defunct site? We should probably 301 anyway as it would be good to ensure that folk following all those links can find my client's new site, but it would be great if page rank flowed too! All ideas appreciated! Many thanks
Intermediate & Advanced SEO | | Chammy
Wendy0 -
Will links to a subdomain help it rank?
I have an affiliate subdomain on a larger company's domain. (For example I have: www.victor.company.com on www.company.com). Would working to attain backlinks to the subdomain help it rank or will I just be putting forth my effort and helping the domain rank?
Intermediate & Advanced SEO | | VictorVC0 -
Affiliate Links Added and Site Dropped in only Google
My site was dropshipping a product and we switched to an affiliate offer. We had three 4 links to different affiliate products. Our site dropped the next day. I have been number 1 for 6 months, has a pr 6 and is 2 years old. It has been 2 weeks and the site hasn't jumped back. Any suggestions on how to handle this?
Intermediate & Advanced SEO | | dkash0 -
Dual Authority – Dual Inline Site Links
Ok, I have a quick question about these, i keep seeing them. There has been talk of Google showing dual inline sitelinks (the extra links it shows under the number 1 results). It used to show 8 links under many number 1 results. It was reported it was showing 2. Now it’s showing 3 …for example, for comparestore prices, compare the market and pricerunner (for a search on compare). How do I get these, or go about getting started with being able to attain them?
Intermediate & Advanced SEO | | TomBarker820