Welcome to the Q&A Forum

Yes, its a wordpress site and I always had the all in one SEO plugin enabled.

I have an established blog.We update it on daily basis. In the past, when I would publish a new post, it would get indexed within a minute or so.

But since a month or so, its taking hours. Sometimes like 10-12 hours for new posts to get indexed. Only thing I have changed is robots.txt.

This is the current robots file.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-login.php
Disallow: /*wp-login.php*
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /author
Disallow: /category
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /login/
Disallow: /wget/
Disallow: /httpd/
Disallow: /*.php$
Disallow: /*?*
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

User-agent: TechnoratiBot/8.1
Disallow:

# ia_archiver
User-agent: ia_archiver
Disallow: /

# disable duggmirror
User-agent: duggmirror
Disallow: /

# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow: /wp-includes/
Allow: /*

# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

Sitemap: http://www.domainname.com/sitemap.xml.gz

Site has tons of backlinks. Just wondering if something is wrong with the robots file or if it could be something else.

Hehe. Some one on DP suggested that.

So are you saying just add this to htaccess

Options +FollowSymlinks

RewriteEngine on

RewriteBase /

RewriteCond %{SERVER_PORT} ^443$ [OR]

RewriteCond %{HTTPS} =on

RewriteRule ^(.*)$ http://www.sitename.com/$1 [R=301,L]

I am just trying to understand what it actually do? Also I am guessing i will have to deindex already indexed https pages from webmaster tools.

Barry,

I will try that also. Also wanted to ask you what you think about adding following to the template.

if ($_SERVER['HTTPS'] = "on") {
    echo 'content="noindex, nofollow">';
}

Its a wordpress site and actually I do not even have a single page that needs https.

I am also thinking of just getting rid of ssl from my site. I added it to have a unique IP for my site cause I had heard it helps with SEO but I think the issues outweigh the benefits.
What do u think is better, do these code changes or just get rid of SSL?

I added SSL to my wordpress blog because that was the only way to get a dedicated IP address for my site at my host. Now I am noticing Google has started indexing posts both as http and https. Can some one please help how to force google not to index https as I am sure its like having duplicate content. All help is appreciated. So far I have added this to top of htaccess file: RewriteEngine on

Options +FollowSymlinks

RewriteCond %{SERVER_PORT} ^443$

RewriteRule ^robots.txt$ robots_ssl.txt And added robots_ssl.txt with following: User-agent: Googlebot Disallow: /  User-agent: * Disallow: /

But https pages are still being indexed. Please help.