Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots blocked by pages webmasters tools

Intermediate & Advanced SEO

2

7

649

Log in to reply

mihoreis last edited by
- a mistake made in software. How can I solve the problem quickly? help me.
XTRjH
1 Reply Last reply
Reply Quote 0
mihoreis @GastonRiera last edited by

Ok,

Updated.

I need to wait 1 week then

Thank You, Gaston
1 Reply Last reply
Reply Quote 0
Gaston Riera @mihoreis last edited by

That sitemap line place it at the end of the file.
like this:

User-agent: *
Disallow:

Sitemap: https://site.com/sitemap.xml

Best luck!
1 Reply Last reply
Reply Quote 1
mihoreis @GastonRiera last edited by

OK,

I did,

can you check from the picture.

Thank You.

zZd9P
1 Reply Last reply
Reply Quote 0
Gaston Riera @mihoreis last edited by

its better to set it as I've discribed before:

User-agent: *
Disallow:

It would take over a week, depending in scale of the website.
1 Reply Last reply
Reply Quote 1
mihoreis @GastonRiera last edited by

Yes,

in the shape of the photo,

is it correct? re-scanning takes a long time?

c4ZCy
1 Reply Last reply
Reply Quote 0
Gaston Riera last edited by

Hello there!

Have you identified all pages that you want to be blocked?
If you just do not want any page to be blocked, just edit you robots.txt as:

User-agent: *
Disallow:

For more information, check these resources:
Cómo bloquear URLs con robots.txt - Google Webmasters
About /robots.txt - robotstxt.org

Best Luck!
GR
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Merging Pages and SEO

Hi, We are redesigning our website the following way: Before: Page A with Content A, Page B with Content B, Page C with Content C, etc
e.g. one page for each Customer Returns, Overstocks, Master Case, etc
Now: Page D with content A + B + C etc.
e.g. one long page containing all Product Conditions, one after the other So we are merging multiples pages into one.
What is the best way to do so, so we don't lose traffic? (or we lose the minimum possible) e.g. should we 301 Redirect A/B/C to D...?
Is it likely that we lose significant traffic with this change? Thank you,
Intermediate & Advanced SEO | | viatrading1

0
Duplicate Page Content

We have different plans that you can signup for - how can we rectify the duplicate page content and title issue here? Thanks. | http://signup.directiq.com/?plan=100 | 0 | 1 | 32 | 1 | 200 |
| http://signup.directiq.com/?plan=104 | 0 | 1 | 32 | 1 | 200 |
| http://signup.directiq.com/?plan=116 | 0 | 1 | 32 | 1 | 200 |
| http://signup.directiq.com/?plan=117 | 0 | 1 | 32 | 1 | 200 |
| http://signup.directiq.com/?plan=102 | 0 | 1 | 32 | 1 | 200 |
| http://signup.directiq.com/?plan=119 | 0 | 1 | 32 | 1 | 200 |
| http://signup.directiq.com/?plan=101 | 0 | 1 | 32 | 1 | 200 |
| http://signup.directiq.com/?plan=103 | 0 | 1 | 32 | 1 | 200 |
| http://signup.directiq.com/?plan=5 |
Intermediate & Advanced SEO | | directiq

0
Google Webmaster Tools Parameters

We have several large ecommerce websites, and we've added some tracking parameters to GWT for google to ignore. All pages are correctly canonicaled. Google has been ignoring the parameters and the canonicals, and still ranks many parametered pages for us. Has anyone run into this?
Intermediate & Advanced SEO | | AMHC

0
A/B Testing - Should I add product descriptions on my category landing pages as well as on product pages and if so . how to do this to avoid duplicate content

Hi All, I recently relaunched a new design on my tool hire eCommerce website and now display my products in grid form on my category landing pages as opposed to just a list view which we previously had on the old design. My bounce rates are alot higher than they use to be and my gut instinct is telling me maybe this is wrong . I want to do some a/b testing using a list view. My question is , previously in our list views we just showed the images and pricing and had on page content on the bottom of the page. The user would click on the product image and they would then we taken to the product page which has the product description , t&c, etc etc.. If I was to do this in my a/b testing but change it so we also displayed the product descriptions as well on the category landing pages . Is there a special way to do this as in effect, we would have duplicate content as the product descriptions are also on the product page?. Does anyone have any thoughts on this as to whether its a No No from an SEO point of view ?... Heres a short url link to one of my category pages - http://goo.gl/QJv5gw Historically we use to rank well for the category landing pages and not for the product pages.Our Rankings are down , bounce rates are higher so I am trying to sort both. We have good content on pages etc. Any advice greatly appreciated as always thanks Pete
Intermediate & Advanced SEO | | PeteC12

0
Incorrect cached page indexing in Google while correct page indexes intermittently

Hi, we are a South African insurance company. We have a page http://www.miway.co.za/midrivestyle which has a 301 redirect to http://www.miway.co.za/car-insurance. Problem is that the former page is ranking in the index rather than the latter. The latter page does index occasionally in the same position, but rarely. This is primarily for search phrases like "car insurance" and "car insurance quotes". The ranking was knocked down the index with Penquin 2.0. It was not ranking at all but we have managed to recover to 12/13. This abnormally has only been occurring since the recovery. The correct page does index for other search terms like "insurance for car". Your help would be appreciated, thanks!
Intermediate & Advanced SEO | | miway

0
Urgent Site Migration Help: 301 redirect from legacy to new if legacy pages are NOT indexed but have links and domain/page authority of 50+?

Sorry for the long title, but that's the whole question. Notes: New site is on same domain but URLs will change because URL structure was horrible Old site has awful SEO. Like real bad. Canonical tags point to dev. subdomain (which is still accessible and has robots.txt, so the end result is old site IS NOT INDEXED by Google) Old site has links and domain/page authority north of 50. I suspect some shady links but there have to be good links as well My guess is that since that are likely incoming links that are legitimate, I should still attempt to use 301s to the versions of the pages on the new site (note: the content on the new site will be different, but in general it'll be about the same thing as the old page, just much improved and more relevant). So yeah, I guess that's it. Even thought the old site's pages are not indexed, if the new site is set up properly, the 301s won't pass along the 'non-indexed' status, correct? Thanks in advance for any quick answers!
Intermediate & Advanced SEO | | JDMcNamara

0
Which page to target? Home or /landing-page

I have optimized my home page for the keyword "computer repairs" would I be better of targeting my links at this page or an additional page (which already exists) called /repairs it's possible to rename & 301 this page to /computer-repairs The only advantage I can see from targeting /computer-repairs is that the keywords are in the target URL.
Intermediate & Advanced SEO | | SEOKeith

0
What to do with WordPress generated pages?

I'm an SEOmoz Newbie and have a very specific question about the auto generated WordPress Pages. SEOmoz caught and labeled the auto generated WP pages as Crawl Warnings like: Long URL - 302 - Title Element to Long - Missing Meta Description Tag - Too Many On-Page Links So I have learned the lesson and have now made those pages "no follow" / "no idex." HOWEVER, WHAT DO I DO WITH THE ONES THAT HAVE ALREADY BEEN INDEXED? Do I... 1. Just leave them as is a hope they don't hurt me from an SEO perspective? 2. Redirect them all to a relevant page? I'm sure many people have had this issue. What do you think? Thanks Dominic
Intermediate & Advanced SEO | | amorbis

0