How to command Robots.txt to this:
-
Hi,
So for some reason I have this unexplained issues in webmaster tools. Check them out:
See that iSeeCars.com? How to remove it? Is it just disallow: iseecars.com?
Or should I disallow the search to be crawled?
Regards,
-
Nikola, are you using wordpress ?
What is the URL?
Web therapist,
Chenzo
-
Do update the thread when you have more data Nikola. I would be interested to see what else is found.
-Andy
-
Well it's weird, I went to this one article that Moz was pointing out, and I did saw a link to ISeeCars.com, which was added by my contributors probablt.
The thing is, the link is directing to Iseecars.com, but when you click on it, It redirects you to:
I don't know why?!
When I edited the post, I double checked that the link is linking only to ISEECARS...
...the issue remained, so I gave up and deleted the link.
I'll wait another week for crawl reports.
Thanks for the guidance though,
Cheers,
-
I haven't actually seen his issue before. It looks like your searches are producing an element in your RSS feed and I suspect this is being done to try and gain backlinks from you or is referrer spam.
Add the following to your robots.txt. It can do no harm.
disallow: /iseecars.com
Someone else might pop along with a little more advice on this but I suspect it is something automated.
-Andy
-
Nono, I haven't had any contact with this website before. I've seen it only as crawl issue, I see it on Moz Crawl Report Too.
-
Is this a page / article in your site anywhere Nikola? I'm wondering if it's some kind of referrer spam.
-Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Has anyone ever tested to see if having an ads.txt file provided any SEO lift?
I know that the ads.txt system is designed to prevent ad fraud and technically has nothing to do with search. That said, the presence of such a file would seem to be an indicator of overall site quality because it would show that a site owner wants to participate in a fraud-free system. Has anyone ever tested that? If so, they don't seem to have published their results. Maybe it's a secret weapon that some pros are using and not sharing?
Web Design | | scodtt0 -
Disallow: /sr/ and Disallow: /si/ - robots.txt
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember. If any of you know what these directives do, then please let me know.
Web Design | | McTaggart0 -
Robots.txt being blocked
I think there is an issue with this website I'm working on here is the URL: http://brownieairservice.com/ In Google Webmaster tools I am seeing this in the Robots.txt tester: User-agent: *
Web Design | | SOM24
Crawl-delay: 1
Disallow: /wp-content/plugins/
Disallow: /wp-admin/ Also when I look at "blocked resources" in the webmaster tools this is showing to be blocked: http://brownieairservice.com/wp-content/plugins/contact-form-7/includes/js/jquery.form.min.js?ver=3.51.0-2014.06.20It looks like the form plug in is giving the issues but I don't understand this. There are no site errors or URL errors so I don't understand what this crawl delay means or how to fix it. Any input would be greatly appreciated. Thank you0 -
Is anyone using Humans.txt in your websites? What do you think?
http://humanstxt.org Anyone using this on their websites and if so have you seen and positive benefits of doing so? Would be good to see some examples of sites using it and potentially how you're using the files. I'm considering adding this to my checklist for launching sites
Web Design | | eseyo1 -
How to fix and issue with robot.txt ?
I am receiving the following error message through webmaster tools http://www.sourcemarketingdirect.com/: Googlebot can't access your site Oct 26, 2012
Web Design | | skehoe
Over the last 24 hours, Googlebot encountered 35 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. The site has dropped out of Google search.0 -
Search directory - How to apply robots
Hi. On the site I'm working on, we use a search directory to display our search results. It displays as follows - Mydomain.com/search-results/# With the dynamic search results appearing after the hash tag. Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav. How should I handle this? Thanks.
Web Design | | Blenny0 -
Correct use for Robots.txt
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
Web Design | | EricVallee340
