How to command Robots.txt to this:
-
Hi,
So for some reason I have this unexplained issues in webmaster tools. Check them out:
See that iSeeCars.com? How to remove it? Is it just disallow: iseecars.com?
Or should I disallow the search to be crawled?
Regards,
-
Nikola, are you using wordpress ?
What is the URL?
Web therapist,
Chenzo
-
Do update the thread when you have more data Nikola. I would be interested to see what else is found.
-Andy
-
Well it's weird, I went to this one article that Moz was pointing out, and I did saw a link to ISeeCars.com, which was added by my contributors probablt.
The thing is, the link is directing to Iseecars.com, but when you click on it, It redirects you to:
I don't know why?!
When I edited the post, I double checked that the link is linking only to ISEECARS...
...the issue remained, so I gave up and deleted the link.
I'll wait another week for crawl reports.
Thanks for the guidance though,
Cheers,
-
I haven't actually seen his issue before. It looks like your searches are producing an element in your RSS feed and I suspect this is being done to try and gain backlinks from you or is referrer spam.
Add the following to your robots.txt. It can do no harm.
disallow: /iseecars.com
Someone else might pop along with a little more advice on this but I suspect it is something automated.
-Andy
-
Nono, I haven't had any contact with this website before. I've seen it only as crawl issue, I see it on Moz Crawl Report Too.
-
Is this a page / article in your site anywhere Nikola? I'm wondering if it's some kind of referrer spam.
-Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Reason for robots.txt file blocking products on category pages?
Hi I have a website with thosands of products. On the category pages, all the products are linked to with the code “?cgid” in the URL. But “?cgid” is also blocked in the robots.txt file for some reason. So I'm thinking it's stopping all my products getting crawled by Google. Am I right here? Is there any reason why a website would want to limit so many URL's? I'm only here a week and the sites getting great traffic, so don't want to go breaking it!!! Thanks
Web Design | | Frankie-BTDublin0 -
Block parent folder in robot.txt, but not children
Example: I want to block this URL (which shows up in Webmaster Tools as an error): http://www.siteurl.com/news/events-calendar/usa But not this: http://www.siteurl.com/news/events-calendar/usa/event-name
Web Design | | Zuken0 -
Is anyone using Humans.txt in your websites? What do you think?
http://humanstxt.org Anyone using this on their websites and if so have you seen and positive benefits of doing so? Would be good to see some examples of sites using it and potentially how you're using the files. I'm considering adding this to my checklist for launching sites
Web Design | | eseyo1 -
Robots.txt - Allow and Disallow. Can they be the same?
Hi All, I need some help on the following: Are the following commands the same? User-agent: * Disallow: or User-agent: * Allow: / I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots. Is that correct? Many thanks, Aidan
Web Design | | Presenter0 -
How to fix and issue with robot.txt ?
I am receiving the following error message through webmaster tools http://www.sourcemarketingdirect.com/: Googlebot can't access your site Oct 26, 2012
Web Design | | skehoe
Over the last 24 hours, Googlebot encountered 35 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. The site has dropped out of Google search.0 -
Search directory - How to apply robots
Hi. On the site I'm working on, we use a search directory to display our search results. It displays as follows - Mydomain.com/search-results/# With the dynamic search results appearing after the hash tag. Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav. How should I handle this? Thanks.
Web Design | | Blenny0 -
Correct use for Robots.txt
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
Web Design | | EricVallee340
