Robots.txt question
-
I notice something weird in Google robots. txt tester
I have this line
Disallow: display=
in my robots.text but whatever URL I give to test it says blocked and shows this line in robots.text
for example this line is to block pages like
http://www.abc.com/lamps/floorlamps?display=table
but if I test
http://www.abc.com/lamps/floorlamps or any page
it shows as blocked due to Disallow: display=
am I doing something wrong or Google is just acting strange? I don't think pages with no display= are blocked in real.
-
Yes - there is bug in your robots.txt. You should wrote some as:
Disallow: /?display=table
or:
Disallow: /?display=*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
[Question] 880 old back links suddenly disappeared from google console
Hi everyone Last month I bought a premium domain from an auction, right after I bought the domain, I added it into google webmaster tools. It showed me site traffic, and 893 backlinks. So until today, all those backlink count was there. And when I checked the console today, it shows only 13 new backlinks which I started building last week. what has happened to all those old 880 old backlinks?I checked from Moz, those backlinks are still there. Am i doing something wrong? Or is this another Google dance? URL https://goread.io
Intermediate & Advanced SEO | | gizmos120 -
Backlink Question
Hello. I have an eCommerce Flower Shop Website and I have created an Online Free Game so player can create flower arrangement and stuff like that. If I had this game directly on my website and provide Embed Code so other Online Gaming Portals can add the game to their website. Would it be good to create back links for my website and increase my domain authority and SEO of my Flower Shop website? I ask because I am also in the Gaming industry and when I release my games, they can be added to hundreds of other gaming portals. But I am just afraid that since this is not is the same domain activity, this might hurt my SEO for my flower website. What are your thoughts about that?
Intermediate & Advanced SEO | | 8860319canadainc0 -
Technical SEO Question: Why is our new platform showing a small decline in traffic?
Hi there! We are in the process of transitioning to a faster platform, and we recently moved a subset of URLs over. The subset that moved over saw a drop. We didn't change the URL pattern, or the content. The only thing that is different is the new platform. Here's a link to one of the URLs that is currently served from the new platform: http://bit.ly/1YjXD7H And, here is an example of a URL currently served from the older platform: http://bit.ly/1Jtx7Di Any ideas why the newer platform is seeing a decline in organic traffic?
Intermediate & Advanced SEO | | nicole.healthline0 -
SEO question
Hi there! I'm the SEO manager for 5 Star Loans. I have 2 city pages running. We are running our business in 2 locations: Berkeley, CA & San Jose, CA. For those offices we've created 2 google listings with separate gmail accounts. Berkeley (http://5starloans.com/berkeley/) ranks well in Berkeley in Gmaps and it shows on first page in organic results. However the second city page San Jose (http://5starloans.com/san-jose/) doesn't show in the Gmaps local pack results and also doesn't rank well in organic results. Both of them have authentic backlinks and reviews. It has been a year already and it's high time we knew the problem 🙂 any comment would be helpful. thanks a lot
Intermediate & Advanced SEO | | moonalev0 -
Quick Question: Is it Bad for SEO to paste from Word to your CMS?
Hey just a quick question I'm having trouble finding a definitive answer to: Is the markup that is transferred from Word docs bad for SEO? We are managing to paste it and it looks fine, but the developers are worried that the extra code will be bad for SEO. Does anyone have solution besides pasting into Text Editor and formatting in the CMS? Is this necessary or can we just leave the extra code? Thank you!
Intermediate & Advanced SEO | | keL.A.xT.o0 -
High level rel=canonical conceptual question
Hi community. Your advice and perspective is greatly appreciated. We are doing a site replatform and I fear that serious SEO fundamentals were overlooked and I am not getting straight answers to a simple question: How are we communicating to search engines the single URL we want indexed? Backstory: Current site has major duplicate content issues. Rel-canonical is not used. There are currently 2 versions of every category and product detail page. Both are indexed in certain instances. A 60 page audit has recommends rel=canonical at least 10 times for the similar situations an ecommerce site has with dupe urls/content. New site: We are rolling out 2 URLS AGAIN!!! URL A is an internal URL generated by the systerm. We have developed this fancy dynamic sitemap generator which looks/maps to URL A and creates a SEO optimized URL that I call URL B. URL B is then inserted into the site map and the sitemap is communicated externally to google. URL B does an internal 301 redirect back to URL A...so in an essence, the URL a customer sees is not the same as what we want google to see. I still think there is potential for duplicate indexing. What do you think? Is rel=canonical the answer? In my research on this site, past projects and google I think the correct solution is this on each customer facing category and pdp: The head section (With the optimized Meta Title and Meta Description) needs to have the rel-canonical pointing to URL B
Intermediate & Advanced SEO | | mm916157
example of the meta area of URL A: What do you think? I am open to all ideas and I can provide more details if needed.0 -
2013 Panda Update Question
Hi everyone, I'm new here 🙂 So far I've had wonderful success seo wise and none of the updates (Penguin nor Panda) affected any sites, until this one. For example, one site has 7 keywords I'm optimizing for. Out of those 7, all but 2 (and variations of the 2 - one word vs long-tail) completely tanked. These keywords were all on page 2/3. One of the two survivors never budged from page 2 (it's a brand keyword so I was sooo happy to finally get it to page 2) Now when I check rankings, the other terms show up in the 200-400 spots, but NOT for the URL I was optimizing for (category page) but instead for random products in the category. The only thing I've done differently with the 2 keywords that are still doing well, was focus - we did more link-building for those, but not an extreme amount. Never over-optimize. My question is, how did 2 survive and 5 are still floating up and down. Last night I saw one go up 122 spots, now today down 14. I'm really struggling with this. Thank you
Intermediate & Advanced SEO | | Freelancer130 -
Anchor text questions - What are your thoughts?
Hi, I want to talk about anchor text and the effect it has on search engines (Good & Bad). Here is a fictitious example we can talk about: www.supercoolrunningsneakers.com Title Tag: Running Sneakers - The Super Trendy Running Trainers Keywords targeted: Running Sneakers, Running Trainers How would you vary your anchor text to target these terms? If you had 50 unique articles to play with how would you vary the anchor text using the articles? Would you push 25 articles at 'Running Sneakers' and the other 25 at 'Running Trainers' or would you link some articles using the domain name anchor text? Q: I'm guessing running 50 articles using anchor text 'Running Sneakers' would benefit the SERP's moe for that term then mixing it up with say 'Running Sneakers', 'Running Trainers' & 'Domain Name Links'. Cheers
Intermediate & Advanced SEO | | activitysuper0