Disallow: /search/ in robots but soft 404s are still showing in GWT and Google search?

[esiow2013]

Hi guys, I've already added the following syntax in robots.txt to prevent search engines in crawling dynamic pages produce by my website's search feature: Disallow: /search/. But soft 404s are still showing in Google Webmaster Tools. Do I need to wait(it's been almost a week since I've added the following syntax in my robots.txt)? Thanks, JC

[timhatton]

You could also look at using the meta robots = noindex tag on /search/ pages, rather than just blocking it in robots.txt, as this will remove existing URLs from the index.

[intSchools]

Glad to help

[esiow2013]

Thanks a lot Dan!

[intSchools]

That is a good recommendation but ultimately search engines will make a final decision on crawl frequency. Take a look at your 'Crawl Stats' on GWTs and this will give you an idea of how often your site is crawled.

[esiow2013]

Is the time issue related in crawl frequency of the URLs in my sitemap?

Thanks Dan, appreciate it.

[intSchools]

You will probably need to wait a little longer - it depends how often your site usually gets crawled and indexed.

However, robots.txt does not always stop search engines from indexing your pages. It will stop them crawling a page on your site but it tells them that they can still index that page. If they find links from external sites then the URL may still appear in the SERP.