Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I am Using <noscript>in All Webpage and google not Crawl my site automatically any solution</noscript>
| |
Web Design | | ahtisham2018
| | <noscript></span></td> </tr> <tr> <td class="line-number"> </td> <td class="line-content"><meta http-equiv="refresh" content="0;url=errorPages/content-blocked.jsp?reason=js"></td> </tr> <tr> <td class="line-number"> </td> <td class="line-content"><span class="html-tag"></noscript> | and Please tell me effect on seo or not1 -
Do I need to 301 redirect www.domain.com/index.html to www.domain.com/ ?
So, interestingly enough, the Moz crawler picked up my index.html file (homepage) and reported duplicate content, of course. But, Google hasn't seemed to index the www.domain.com/index.html version of my homepage, just the www.domain.com version. However, it looks like I do have links going specifically to www.domain.com/index.html and I want to make sure those are getting counted towards my overall domain strength. Is it necessary to 301 redirect in the scenario described above?
Web Design | | Small_Business_SEO0 -
Traffic flow going through homepage, /default.aspx, and /index.htm - How do I fix this?
In the Google Analytics visitor flow section, most of my traffic is starting off on the homepage, but a significant portion somehow is being directed to /default.aspx, as well as a minor fraction at /index.htm - see the attached image for what this looks like. My questions are, why is this happening, is it negatively affecting the site, and should I just fix it through a filter in Analytics or consult IT? Navigating to the /default.aspx page in the address bar simply returns a blank screen. Any idea of what's going on here? The logo in the top left corner of the site directs to /index.htm, so I think I have that issue solved, but /default.aspx is definitely stumping me. The server currently runs on Apache, though I don't know if it always did. My standard method of simply Googling the problem didn't give me any definitive answers, so any help would be greatly appreciated. Q12QqpC.jpg
Web Design | | BD690 -
How keywords per page to keep from being "spammy"?
Hi all, I am currently doing a marketing internship for a B2B company that does all sorts of out-sourced recruiting work. I have some experience with SEO, but not completely confident. My first question is, I know Google sees websites that load up on keywords as "spammy", so what is the appropriate number of keywords per page? Currently, I was thinking about this setup: 1 keyword for the URL 1 keyword per alt tag (1 per page, at most) 2 keywords per each title tag (approximately 4 pages that I am going to follow internally, not following the "about us" page). After that, I was thinking of adding 2-3 more keywords in each meta description and 2-3 in the body copy. That would equate to 6-8 keywords on each page, is this too many and should keywords be repeated (on the same page or across multiple pages)? Since this website is brand new (zero links), would it make sense to nofollow all of the internal links so that they homepage can gain ranking as quickly as possible within Google?
Web Design | | wlw20090 -
Question About Site Redesign and Nav / Page Structure
Hey guys, i am currently redesigning our company's site, and have come across some things that I'm not quite sure of. We used to have individual service pages in our main navigation (design, video, marketing) before the redesign. In this new design, i had the idea of making just one "services" or "capabilities" page, where these three services would each be outlined, and each service would have a list of links to more specific landing pages. Obviously, breaking it up correctly with HTML5 using the andtags. What I'm wondering is that if i'm going to be penalized for having those three services that aren't necessarily related too closely on the same page as opposed to having the one page for each service (like we have now). Any help would be greatly appreciated, and let me know if i need to elaborate more. Thanks in advance!
Web Design | | RenderPerfect0 -
SEOMoz crawl report shows a duplicate content and duplicate title for these two url's http://freightmonster.com/ and http://freightmonster.com/index.html. How do I fix this?
What page is attached to http://freightmonster.com/ if it is not the index.html ? Should I do a redirect from the index page to something more descriptive?
Web Design | | FreightBoy1 -
Using "#" anchors to display different content
If I have a page that has an area on the page that acts like a widget and has three different tabs. These tabs provide 3 different types of information relevant to the page subject matter. By default when someone goes to the page one of the tabs is showing but you have to click on the others to see the info on them. Is it OK to use domain.com/topic#TAB1, domain.com/topic#TAB2, domain.com/topic#TAB3 to create shortcut links so that people can land on the page and have that predetermined tab showing. I'm wondering what search engines might think. Essentially all the content of all three tabs is there for people to see but they'd have to click to see the other tabs. I don't consider the content to be hidden. But I'd like to hear people's thoughts.
Web Design | | Business.com0 -
Need advice on diplaying content for Search
Hi every body, I am doing landing page redesign(s). Does any body know or can refer a content carousel that can rotate video and pictures? The "site with images" search option result is a compelling reason to showcase pictures if your space competeiveness (showroom, merchandise, etc) can be improved with a strong image presence. here is my main landing page http://www.shearerpainting.com I know there is alot of stuff, and confusing call to action, but I am looking for strategies to clean it up, clear fous on action (get bid, learn more), but allow users to see that they can dig for more content.
Web Design | | johnshearer0