Robots.txt question
-
I want to block spiders from specific specific part of website (say abc folder).
In robots.txt, i have to write -
User-agent: *
Disallow: /abc/
Shall i have to insert the last slash. or will this do
User-agent: *
Disallow: /abc
-
I will do so. And hope to get that back.
-
If you contact the help desk, they can probably help you get your old account back.
-
I am the same person with the username seoug, but lost that account. So, had to start afresh ! I was a PR0 member, but accidently deleted that account ( it was not intentional ). And now , when i tried login in, i get a message that seoug name is already taken.
-
Thanks for clearing my doubts.
-
at least our answers agree, so no Atul is doubley sure of how to do it...
-
EGOL does it to me all the time!
-
Hi Atul,
Add the trailing slash.
/abc could be a page url. Where as /abc/ is definitely a folder.
http://www.robotstxt.org/robotstxt.html <-- Everything you ever wanted to know about robots.txt
Regards
Aran
[EDIT: Damn it, Ryan submitted whilst I was answering! Must type faster ]
-
Use the trailing slash.
More about robots.txt can be learned at this site: http://www.robotstxt.org/
The trailing slash indicates you are blocking a folder. Without the slash the object would be considered a file (i.e. page). I am not sure what the result would be if you tried to block a folder without the trailing slash. Even if it worked it would not be the correct code and may lead to various bots treating it differently.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Role of Robots.txt and Search Console parameters settings
Hi, wondering if anyone can point me to resources or explain the difference between these two. If a site has url parameters disallowed in Robots.txt is it redundant to edit settings in Search Console parameters to anything other than "Let Googlebot Decide"?
Technical SEO | | LivDetrick0 -
General Question: Linking Root Domains 0
Hi, we have several subpages with PA 1. We try to figure out why. The link metrics show 0 Linking root domains, but shouldnt there be at least the own domain like 1 linking root domain or are we getting it already wrong here? The subpages have following link metrics: 0 External followed Links. 8416 subdomain and 8421 Root Domain, Linking Root Domains 0, 299 Subdomain, 302 Root Domain (From Mozbar). The pages seem to be crawled. We are suspecting techincal reasons. What would be the impact of the linking root domain to the PA 1? Thank you in advance
Technical SEO | | brainfruit0 -
Will it be possible to point diff sitemap to same robots.txt file.
Will it be possible to point diff sitemap to same robots.txt file.
Technical SEO | | nlogix
Please advice.0 -
Rel Canonical question
Hi: I got a report indication 17 rel canonical notices. What does this mean in simple language and how do i go about fixing things?
Technical SEO | | Shaaps0 -
Schema Address Question
I have a local business with a contact page that I want to add schema markup to. However, I was wondering if having the address with schema info on the contact page instead of the home page has any adverse effects on the rich snippet showing up in search. There's no logical place to add schema for a local business on the home page, so having it on the contact page—not in the footer or sidebar—is the only option.
Technical SEO | | DLaw0 -
Blogger Blog URL Structure Questions
I'm starting to use my blog more and wanted to ask about an issue I've read about on SEOmoz in the past. I use blogger instead of wordpress. It's quick and simple - I have no interest in switching to wordpress for this particular blog. My blog is currently setup as blog.site.com. Is it still important (for seo reasons) to switch from blog.site.com to site.com/blog? If so, is there a way to do this in blogger? And if I do this, will my past posts lose their authority if their redirected to the new url structure? Rand mentions in this article: http://www.seomoz.org/blog/11-best-practices-for-urls "never use multiple subdomains" - This is an old article, but I've seen this mentioned several times. Does this still hold true? Am I losing out on links to my blog? Thanks in advance.
Technical SEO | | ChaseH0 -
Should I set up a disallow in the robots.txt for catalog search results?
When the crawl diagnostics came back for my site its showing around 3,000 pages of duplicate content. Almost all of them are of the catalog search results page. I also did a site search on Google and they have most of the results pages in their index too. I think I should just disallow the bots in the /catalogsearch/ sub folder, but I'm not sure if this will have any negative effect?
Technical SEO | | JordanJudson0