Correct robots.txt for WordPress

[AslanBarselinov]

Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?

I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.

Thank you so much.

FzSQkqB.jpg

[jasongmcmahon]

Me too, can you upload or screenshot the actual file that you are using

[AslanBarselinov]

I have edited it down to

User-Agent: *

Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /contact/
Disallow: /refer/

It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.

[jasongmcmahon]

Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/

Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml

[effectdigital]

Just seems overly complex and like there's way more in there than there needs to be

I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else

User-Agent: *

Disallow: /wp-content/plugins/

Disallow: /comments

Disallow: /*?s=

Disallow: /*&s=

Disallow: /search

---

See if that helps