Problem of indexing

[android_lyon]

Hello, sorry, I'm French and my English is not necessarily correct.

I have a problem indexing in Google.

Only the home page is referenced: http://bit.ly/yKP4nD.

I am looking for several days but I do not understand why.

I looked at:

• The robots.txt file is ok

• The sitemap, although it is in ASP, is valid with Google

• No spam, no hidden text

• I made a request for reconsideration via Google Webmaster Tools and it has no penalties

• We do not have noindex

So I'm stuck and I'd like your opinion.

thank you very much

A.

[android_lyon]

Hello Rasmus,

i think it's ok now.

Indexing is better http://bit.ly/yKP4nD

Thank you so much.

Take care

A.

[android_lyon]

Hi,

very interesting, good idea !!!

I think you're right.

I will tell you

Best regards

A.

[rasmusbang]

Ah!

I've found it!

You have a canonical link on each page?

| rel="canonical" href="http://www.syrahetcompagnie.com/Default.asp" /> |

This is not so good, as it is on http://www.syrahetcompagnie.com/vins-vallee-du-rhone-nord.htm AND http://www.syrahetcompagnie.com/PBHotNews.asp?PBMInit=1

If you remove that (and keep it on the start page) you should experience a whole lot of indexing in the following days

Best regards

Rasmus

[rasmusbang]

You are correct. I've just found this page:

http://www.robotstxt.org/robotstxt.html

It says:

User-agent: *

Disallow:

Allows all robots to all pages.So that was my mistake. I am truly sorry for the confusion.

I will have a look at it later to see if I can find a good explanation...

[mad2k]

Hi Rasmus,

User-agent: *
Disallow:

means that all robots can enter the site

---

User-agent: *
Disallow: /

block all robots to enter.

---

User-agent: WebCrawler
Disallow:

block WebCrawler robot, but other can enter

Always first line of robots.txt tells what robots can crawl a site and  * means all. Second and next lines are pointing specific catalogues on a server e.g. Disallow: /admin/

So I think that is not a robots.txt issue - please ensure me

[rasmusbang]

Hi again,

Do you use Google Webmaster tools?

In Webmaster tools you can see how many URLs on your site that has been restricted due to robots.txt file. Perhaps that could give you a clue.

I would recommend that you take a look at webmaster tools. All in all there are a lot of good information in there for optimizing your site.

Best regards

Rasmus

[android_lyon]

Thanks for your answer.

OK I will edit the file but I am not convinced that this is causing my problem because it was written that way.

Take care

[rasmusbang]

Actually your robots.txt is NOT ok. It says:

Sitemap: http://www.syrahetcompagnie.com/Sitemap.asp?AccID=27018&LangID=0 User-agent: * Disallow:  Which means that all pages are to be disallowed. You should have: User-agent: * Allow: /

If you change that, it should fix it!