Robots txt is case senstive? Pls suggest
-
Hi i have seen few urls in the html improvements duplicate titles
Can i disable one of the below url in the robots.txt?
/store/Solar-Home-UPS-1KV-System/75652
/store/solar-home-ups-1kv-system/75652if i disable this
Disallow: /store/Solar-Home-UPS-1KV-System/75652
will the Search engines scan this /store/solar-home-ups-1kv-system/75652
im little confused with case senstive.. Pls suggest go ahead or not in the robots.txt
-
Hi Already there is some equity for duplicate links, wht is going to happen?
-
Actually, you have just one option to not index them - the second one. The first will, still keep them in index if google can find them. I currently have roughly 27k URLs indexed that were blocked via robots.txt from the start (generated with a time-based parameter; yeah: ouch.).
Those results do not usually appear in "normal" search but can be forced (currently you may try site:grimoires.de inurl:fakechecknr and showing skipped results to see the effect of that). So basically I'd advise against using robots.txt - it does not prevent indexing, only the visiting/reading of that page.
Regards
Nico
-
Hi Abdul,
Yes, it is case sensitive.
Remember that you must not have many pages like that.
The first thing you should do is elimiate those duplicate pages.In the case you can´t eliminate them, you have 2 way to ask the google bot not to index them:
1- By robots.txt with a 'Disallow:' instruction
2- By a meta tag with a_ '' _in theHope it helps.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Suggestion for improvement
I am working on the website and feel like we have a lot more content (original) also all of it is above the fold, however I don't seem to find the website ranked higher compared to other sites with similar keywords. My URL is: http://www.cypressindustries.com/Any suggestion for improvement or which areas I should be focusing?
Intermediate & Advanced SEO | | HasitR0 -
My Domain authority dropped 9 points... Does anyone have any suggestions to fix this significant drop.
My domain authority dropped by 9 points and I haven't done anything differently since the last scan. What is going on?
Intermediate & Advanced SEO | | infotrust20 -
Quotery.com Suggestions?
Hello, I'm hoping to find some advice on how to proceed with a site I started last year — Quotery. I've spent a ton of time and money on it, tried my best to build value to what is typically a copycat niche, and adhered to Google's SEO recommendations (I believe, anyway). Yet, subpar sites rank well above mine. Now I'm wondering if I should find a buyer for the site and cut my losses. Does anyone have any suggestions on what the next steps might be in order to rank higher? Here is what we've tried/accomplished so far: We were mentioned on Netted. We built a well-designed and easy to navigate site that works on all devices. We've added topic descriptions and images, unique author descriptions and pictures, and exclusive picture quotes. We built a well-designed and useful WordPress plugin, and kept the backlinks nofollow (note: our main competitor also has a subpar WordPress plugin with dofollow backlinks, yet they don't get penalized for it). We've published curated blog posts, along with infographics that have been shared millions of time #1, #2, #3. We've built up significant social media profiles on Facebook, Twitter, and elsewhere. We even tried hiring 97thFloor (highly recommended on Moz) for 4 months, although most of their efforts seemed spammy and/or very basic to me (and cost a fortune), so we decided to take SEO into our own hands afterwards. We've added sources, pictures, and relevant information for thousands of quotes (e.g. example seen here). We started working on user profiles, and had plans for much more down the road. However, despite these efforts, sites like BrainyQuote dominate Google's rankings. So is it truly value that will earn you rankings... or is it still all about gaming the system? Of course, any suggestions in my case specifically would be much appreciated.
Intermediate & Advanced SEO | | JasonMOZ0 -
Can't find X-Robots tag!
Hi all. I've been checking out http://www.unthankbooks.com/ as it seems to have some indexing problems. I ran a server header check, and got a 200 response. However, it also shows the following: X-Robots-Tag:
Intermediate & Advanced SEO | | Blink-SEO
noindex, nofollow It's not in the page HTML though. Could it be being picked up from somewhere else?0 -
Bing Webmaster Tools failed to reach sitemap any suggestions?
My sitemap has been submitted to Bing webmaster tools well over a year ago and I have never had any problems. Starting last week it showed failed, for some reason it can't reach it. I have resubmitted several times and it fails every time. I can go to the url with no problems, and Google Webmaster Tools does not have any problems. We have made no changes in over a year to how the sitemap is made and submitted. Anyone have any ideas?
Intermediate & Advanced SEO | | EcommerceSite0 -
SEM Logic Or SEO Scout: Softwares for writers - Any Suggestions?
I was curious if anyone has used SEM Logic or SEO Scout for adding related LSI keywords in their content. I just hired a few content writers and was wondering if anyone had any suggestions if these softwares are worth investing in or if anyone knows of any courses that are good for training writers to be able to maximize the seo value of content written. Any suggestions??? thx
Intermediate & Advanced SEO | | ao5000000 -
Meta No INDEX and Robots - Optimizing Crawl Budget
Hi, Sometime ago, a few thousand pages got into Google's index - they were "product pop up" pages, exact duplicates of the actual product page but a "quick view". So I deleted them via GWT and also put in a Meta No Index on these pop up overlays to stop them being indexed and causing dupe content issues. They are no longer within the index as far as I can see, i do a site:www.mydomain.com/ajax and nothing appears - So can I block these off now with robots.txt to optimize my crawl budget? Thanks
Intermediate & Advanced SEO | | bjs20100 -
New server update + wrong robots.txt = lost SERP rankings
Over the weekend, we updated our store to a new server. Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers). When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed. We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site. The problem is that many of our search rankings have changed. We were ranking #2 for some keywords, and now we're not showing up at all. Is there anything we can do? Google Webmaster Tools says that the next crawl could take up to weeks! Any suggestions will be much appreciated.
Intermediate & Advanced SEO | | 9Studios0