Our parent company has included their sitemap links in our robots.txt file - will that have an impact on the way our site is crawled?
-
Our parent company has included their sitemap links in our robots.txt file. All of their sitemap links are on a different domain and I'm wondering if this will have any impact on our searchability or potential rankings.
-
While that is kind of odd to do, it should not have any effect on your site's search engine rankings or visibility. Simply providing a sitemap file is not going to help or hurt rankings in any way. It will help them discover and crawl pages. That's all.
A sitemap file is not required at all, and in fact some sites don't have them--they rely on Google's spiders to crawl their site through the links on their site. The sitemap is only a tool to help them crawl.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt & Disallow: /*? Question!
Hi, I have a site where they have: Disallow: /*? Problem is we need the following indexed: ?utm_source=google_shopping What would the best solution be? I have read: User-agent: *
Intermediate & Advanced SEO | | vetofunk
Allow: ?utm_source=google_shopping
Disallow: /*? Any ideas?0 -
Any idea why Google Search Console stopped showing "Internal Links" and "Links to your site"
Our default eCommerce property (https://www.pure-elegance.com) used to show several dozen External Links and several thousand Internal Links on Google Search Console. As of this Friday both those links are showing "No Data Available". I checked other related properties (https://pure-elegance.com, http:pure-elegance.com and http://www.pure-elegance.com) and all of them are showing the same. Our other statistics (like Search Analytics etc.) remain unchanged. Any idea what might have caused this and how to resolve this?
Intermediate & Advanced SEO | | SudipG0 -
Client wants to remove mobile URLs from their sitemap to avoid indexing issues. However this will require SEVERAL billing hours. Is having both mobile/desktop URLs in a sitemap really that detrimental to search indexing?
We had an enterprise client ask to remove mobile URLs from their sitemaps. For their website both desktop & mobile URLs are combined into one sitemap. Their website has a mobile template (not a responsive website) and is configured properly via Google's "separate URL" guidelines. Our client is referencing a statement made from John Mueller that having both mobile & desktop sitemaps can be problematic for indexing. Here is the article https://www.seroundtable.com/google-mobile-sitemaps-20137.html
Intermediate & Advanced SEO | | RosemaryB
We would be happy to remove the mobile URLs from their sitemap. However this will unfortunately take several billing hours for our development team to implement and QA. This will end up costing our client a great deal of money when the task is completed. Is it worth it to remove the mobile URLs from their main website to be in adherence to John Mueller's advice? We don't believe these extra mobile URLs are harming their search indexing. However we can't find any sources to explain otherwise. Any advice would be appreciated. Thx.0 -
Hreflang tag on links to alternate language site
Hey everyone! In the interest of trying to be brief, here's the situation in my favorite form of communication, bullet points! Client has two sites; one is in English and one is in Japanese Each site is a separate URL, no sub-domains or sub-pages Each main page on the English version of the site has a link to the homepage of the Japanese site Site has decent rankings overall, with room for improvement from page 2 to page 1 No Hreflang tags currently used in links to the Japanese version from the English version Given that the site isn't really suffering for most rankings, would this be helpful to implement on the English version? Ideally, I'd like each link to be updated to the corresponding subject matter of the Japanese, but in the interim it seems like identifying to Google that the link on the other side is a different language might be helpful to both the user and to maybe help those rankings on page two creep a little higher to page one. Thanks for reading, I appreciate your time.
Intermediate & Advanced SEO | | Etna0 -
Robots.txt: Syntax URL to disallow
Did someone ever experience some "collateral damages" when it's about "disallowing" some URLs? Some old URLs are still present on our website and while we are "cleaning" them off the site (which takes time), I would like to to avoid their indexation through the robots.txt file. The old URLs syntax is "/brand//13" while the new ones are "/brand/samsung/13." (note that there is 2 slash on the URL after the word "brand") Do I risk to erase from the SERPs the new good URLs if I add to the robots.txt file the line "Disallow: /brand//" ? I don't think so, but thank you to everyone who will be able to help me to clear this out 🙂
Intermediate & Advanced SEO | | Kuantokusta0 -
Lowercase Company Name - Impact on Rankings?
A company who spells their name in all lower case, uses their name spelled exactly this way in titles & meta descriptions & throughout all of their site's content pages - will it affect rankings (particularly since the company name isn't a recognised word)? For example, let's say the company is called "twintrest" Which <title>would be better:</p> <ul> <li><span style="background-color: initial;">A) </span><span style="background-color: initial;">Twintrest - It's Like a Twitter & Pinterest Cocktail or</span></li> <li><span style="background-color: initial;">B) twintrest - It's Like a Twitter & Pinterest Cocktail</span></li> </ul> <p><span style="background-color: initial;">I am leaning towards "A" because the crawler would recognise "Twintrest" at least as a noun, where as in B it could be misconstrued as a spelling mistake.</span></p> <p>Also, in content it would need to be italicised right? <span style="background-color: initial;">So both these would be ok:</span></p> <ul> <li><span style="background-color: initial;">You should use Twintrest because it rocks. or</span></li> <li><span style="background-color: initial;">You should use </span><em style="background-color: initial;">twintrest </em><span style="background-color: initial;">because it rocks.</span></li> </ul> <p>But not:</p> <ul> <li><span style="background-color: initial;">You should use twintrest because it rocks</span><span style="background-color: initial;">.</span></li> </ul> <p>Is it worth pointing out?</p> <p>I've noticed that jcpenny spell their name in lowercase and use this format in their meta tags, however, they are a well established brand. Is there any evidence to suggest that lowercase may even be advantageous since a lot of searches are carried out in lowercase?</p></title>
Intermediate & Advanced SEO | | wojkwasi0 -
The speed of my site do you agree with my hosting company
Hi my site is www.in2town.co.uk and i have to be honest i am not really happy at the moment. i took a huge gamble because my site was getting around 10,000 visitors per day but because the site was getting really slow as i was using the old joomla my hosting company suggested to me to use the modern version of joomla as i was using joomla 1.0 Now for months i said no but they said the site would remain slow until i updated it. I decided to take a gamble and as i was going to now move it to the new joomla i decided that i would redo the whole site and make it better which i am doing at the moment. But even though i am paying for a dedicated server and have done what they have said and using the new joomla and according to them they have refreshed my database the site is still running slow. If anyone could do a test on the site and let me know what speed they are getting. Below is the email i have sent to my hosting company and below is also the resonce that i am getting from them.- I would love to hear your feedback as i have taken a huge gamble and when i get back from spain i will be continuing the finish the site which will take me another week to complate and get it the way i want it. here is the feedback hi can you please look into my site www.in2town.co.uk as i am getting lots of reports that it is slow. we redone the whole site which i am still working on because of it being slow and you said that you cleared everything in my database so there was nothing there so we can start from scratch. when i started building the new site you said that the database would be like a brand new one but it seems that this is not working. what i mean by this is, we listened to you and got a dedicated server as you said this would make the site faster, this did not work, you said about redoing the site and using the latest joomla as this would make the site faster and after a lot of work we done this although we are still continuing to build it and as a result of this we lost lots of traffic and links and rankings but even though we done this the site is still slow. i have to be very honest with you, i am not happy and feel disapointed as the site is slow and i need this sorting out. i am in spain at the moment meeting clients ref the site and it is embarrassing when they say the site is slow so please can you look into this for me Hello, Thank you for contacting TMDHosting Technical Support Team. We have checked your current server load and it is below 1: top - 11:53:54 up 123 days, 9:47, 1 user, load average: 0.24, 0.12, 0.11 this is shows that the system is running fine. You have close to 1GB RAM free which provides enough resources for your scripts to function. As the server state is healthy the optimization of your website should be applied on its coding and main functions. This is a development related optimization and we do not offer development services at this point. If your developer recommends a server with more resources provide us with his/hers requirements and we will provide you with suitable solution. Should you have any further questions or comments please do not hesitate to contact us. Best regards, Gabriel Reed from me., i will have to have someone look into this as it may be a case that i will have to get rid of my dedicated server and take the sites to another hosting company and see the difference with them and see if the site is faster as no one else using joomla has the problems that i am having. can you let me know what speed the site is then, and then compare this against some other sites. for example, female first, claire hegarty.co.uk etc as when i do speed checks i am getting back over ten seconds and the site which is very small at the moment should not be more than 2 seconds Hello, Please note that even if you move the website to another hosting provider this will not change the reason because of which its speed is not as per your requirements. However it is up to you to explore different options. What we have ensured is that your server is not overloaded and it has resources for the functions on it to run. The optimization of your website coding I am afraid are outside of our technical the scope of our technical support. Additionally note that the loading speed of your website depends on huge number of other factors on certain remote locations which are not under our control. Should you have any further questions or comments please do not hesitate to contact us. Best regards, Gabriel Reed
Intermediate & Advanced SEO | | ClaireH-184886
Senior Support Team
TMDHosting.com If anyone can please comment on this and look at my site www.in2town.co.uk and test it for me on speed and advise me as i have done everything they have said and it seems that the site is no faster. There is not much content on there at the moment as they managed to lose over 20,000 pages of content while transfering it all which i am not happy about. So there should not be an issue on speed and now i am working through and putting some of the old content back on while writing new content0 -
Negative impact on crawling after upload robots.txt file on HTTPS pages
I experienced negative impact on crawling after upload robots.txt file on HTTPS pages. You can find out both URLs as follow. Robots.txt File for HTTP: http://www.vistastores.com/robots.txt Robots.txt File for HTTPS: https://www.vistastores.com/robots.txt I have disallowed all crawlers for HTTPS pages with following syntax. User-agent: *
Intermediate & Advanced SEO | | CommercePundit
Disallow: / Does it matter for that? If I have done any thing wrong so give me more idea to fix this issue.0
