Issue with Robots.txt file blocking meta description

[HiddenPeak]

Hi,

Can you please tell me why the following error is showing up in the serps for a website that was just re-launched 7 days ago with new pages (301 redirects are built in)?

A description for this result is not available because of this site's robots.txt – learn more.

Once we noticed it yesterday, we made some changed to the file and removed the amount of items in the disallow list.

Here is the current Robots.txt file:

# XML Sitemap & Google News Feeds version 4.2 - http://status301.net/wordpress-plugins/xml-sitemap-feed/
Sitemap: http://www.website.com/sitemap.xml
Sitemap: http://www.website.com/sitemap-news.xml

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Other notes... the site was developed in WordPress and uses that followign plugins:

• WooCommerce All-in-One SEO Pack
• Google Analytics for WordPress
• XML Sitemap
• Google News Feeds

Currently, in the SERPs, it keeps jumping back and forth between showing the meta description for the www domain and showing the error message (above).

Originally, WP Super Cache was installed and has since been deactivated, removed from WP-config.php and deleted permanently.

One other thing to note, we noticed yesterday that there was an old xml sitemap still on file, which we have since removed and resubmitted a new one via WMT. 

Also, the old pages are still showing up in the SERPs.

Could it just be that this will take time, to review the new sitemap and re-index the new site?

If so, what kind of timeframes are you seeing these days for the new pages to show up in SERPs? Days, weeks?

Thanks,

Erin
```

[ThompsonPaul]

At the moment, it doesn't seem that rel=publisher is doing all that much for sites (aside from sometimes showing better info ion the knowledge graph listing on Brand searches)  but personally I believe it's functionality and influence are going to be greatly expanded fairly soon, so well worth doing. As far as it contributing anything to help speed up indexing... doubt it.

P.

[HiddenPeak]

Paul,

Thanks... you hit upon my hunch, that we will just have to wait.

Much of the information in the SERPs (metadescriptions, titles and urls) are still old,even though they redirect to the new pages when I click.

Thanks for the tip... and about social media.

Do you think it will help to get the rel=publisher link to the Google+ page on the site?

Erin

[Johnny4B]

A lot of people, especially WP users use modules that may block certain spiders crawling your site, but in your case, you don't seem to have any.

[ThompsonPaul]

If you just changed the robots.txt file yesterday, my guess is you're going to have to be patient while the site gets recrawled, Erin. Any of the pages that are in the index and were cached before yesterday's robots update will still include the directive not to include the metadescription (since that's the condition they were under when they were cached.)

I suspect the pages you're seeing with metadescriptions were crawled since the robots update. Are you seeing the same page change whether it shows metadescription or not?

As far as old pages showing in the SERPs, again they'll all have to be crawled before the 301 redirects can be discovered and the SEs can begin to understand they should be dropped. (Even then it can take days to weeks for the originals to drop out.)

Another very effective way to help get the new site indexed faster is to attract some good-quality new links to the new pages. Social Media can be especially effective for this, Google+ in particular.

Paul

[HiddenPeak]

Thanks!

What do I need to look for in the .htaccess file?

Here is what is there... and the rest (not shown) are redirects:

BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On RewriteBase / RewriteRule ^index.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L]</ifmodule> # END WordPress

BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On RewriteBase / RewriteRule ^index.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L]</ifmodule> # END WordPress

[HiddenPeak]

Thanks for the tips! Let me check it out.

[Johnny4B]

I'd also insure its not something to do with your .htacess file.

[irvingw]

Make sure the pages aren't blocked with meta robots noindex tag

Fetch as Google in WMT to request a full site recrawl.

Run brokenlinkcheck.com and see if their crawler is successfully crawling or if it's blocked.