Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website Form Doesn't Work for Visitors from Google Search App
I've got a strange issue where if visitor is viewing a website from within the Google Search App, the app doesn't seem to be able to handle online forms. The form relies users on entering data, which is then POSTed to an external booking engine site. Normally this works fine, except when Google Search App is acting as the browser, the post payload is empty and the URL breaks. We're a bit stumped as to how to move forward. So far the only lead is that Android users can override having the app behave as a browser - but doesn't seem like iOS users can. Any additional ideas/tips are welcome here - thanks. 🙂
Web Design | | mirabile0 -
Need to find a wordpress theme
Hi, while browsing i found a wordpress website (http://www.zoompondy.com/). This website was really nice. Explored about the theme and unable to find the theme name. Even, I Tried WP theme detector. No solution. Please assist me in finding this theme.
Web Design | | jkovalan0 -
Responsive image plugins and seo / crawlability
Note : For the background of this question please read the preface below. Ive been researching responsive image options the main issue i can see with them is that they are not semantic html so bots may not index them correctly. For instance many of the responsive image plugins use data-src for an image rather than src. Does any one have any experience with this and if it impacts on SEO ? Does any one know of a client side responsive image soltion that uses a normal img tag with the image stored in the src and with the option to set an alt attribute ? **Preface : ** Ive got a site we are currently developing, the site has a large full width responsive image slider. To serve images that wont be pixilated we are making the width of the images 1800px wide (which should cover most screens, but isn't actually big enough if the site was viewed on a 27" imac) these 1800px wide images weight about 350kb - 500kb per image and our image slider has about 20 of them. As you can see this would be a problem for anyone with a connection slower than c.10 mbps. This is especially true for mobile devices that will be downloading an image 1800px wide although only require a much smaller one, this coupled with a 3g connection will make the site really slow.
Web Design | | Sam-P0 -
May I know what's the "CSS Class" here?
ul.nav a#homelink { text-indent: -9999px; background: url('http://www.elegantthemes.com/preview/MyProduct/wp-content/themes/MyProduct/images/home-icon.png') no-repeat center 6px; padding: 0px; } ul.nav a#homelink span { display: block; padding: 5px 9px 8px 5px; width: 16px; } I'm having problems in adding back the "house" icon instead of a "Home" text in my menu. I don't what "CSS class" should I use to be added in the Home menu button (Appearance>Menu). I'm currently using a "MyProducts" Wordpress theme by ElegantThemes. Thanks in advance!
Web Design | | esiow20131 -
Schema - Has it worked for you?
Hi all! For a while now I've been trying to convince the higher ups that Schema (or related semantic markup languages) would be a plus for our sites. Predominantly we're in ecommerce and so far I still haven't noticed any of our competitors really using Schema. So I was asked to hold off on it until other projects had been handled. But now the coder I had been working closely with has moved to another job so I basically need to re-explain all of this to someone else. But that's not really the point of this discussion post. I recently read Barry Schwartz's post on the Google Structured Data Markup Helper Tool over on Search Engine Land [link] and thought it'd be a good time to bring up Schema to management again. But first, I wanted to ask how Schema has worked for the rest of you. Have you seen any increases in clickthroughs? Have rich snippets in Google made you more visible? Have you seen a decent ROI for the time spent on implementing it? Are you seeing an increase in people implementing it across your verticals? Have you found certain types of semantic data are less useful than others? Any tips, tricks, pointers concerning their use and implementation for those who are considering it? Any good case studies you can share? Thanks all!
Web Design | | MikeRoberts0 -
Do pull quotes affect SEO positively or negatively?
I like the design element of a pull quote to ad interest and highlight an important point. If I use an exact quote from the page in a pull quote on that page, does that negatively affect SEO as duplicate content? Are there formatting or tagging methods that could help pull quotes to boost SEO? For clarity, by "pull quote" I mean a stylized bit of text that floats on a page in such a way that the body text wraps around it. It is actual text (not text embedded in a graphic) but it behaves like an image with text wrapping around it. Here's an example (in red on the right side): http://www.21ct.com/resources/news-room/21ct-announces-its-latest-us-patent-for-advancing-big-data-security/
Web Design | | kyle21ct0 -
Need help to implement microdata/microformat for ecommerce site
**Can somebody please help me to implement microdata/microformats codes for our ecommerce product pages? **
Web Design | | EastEssence22
Please guide me if you have some CSS example for the same. Thanks.0 -
I NEED ADVICE: Need Web Software for customers to access backend
I have a prospect in the credit repair industry. Does anybody know of some software where the clients can log into their own accounts to see the progress? The feature I am looking for is like a Real Estate website where the brokers can create their own accounts and upload their own properties.
Web Design | | Francisco_Meza0