Yes, everything looks good, Webmaster Tools gave me the expected results with the following directives:

allow: /directory/$

disallow: /directory/*

Which allows this URL:

http://www.mysite.com/directory/

But doesn't allow the following one:

http://www.mysite.com/directory/sub-directory2/...

This page also gives an update similar to mine:

https://support.google.com/webmasters/answer/156449?hl=en

I think I am good! Thanks

Thank you Cyrus, yes, I have tried your suggested robots.txt checker and despite it validates the file, it shows me a couple of warnings about the "unusual" use of wildcard. It is my understanding that I would probably need to discuss all this with Google folks directly.

Thank you for you answer... and, yes Keri, I know this is a old thread, but still useful today!

Thanks

What about something like:

allow: /directory/$

disallow: /directory/*

Where I want this to be indexed:

http://www.mysite.com/directory/

But not this:

http://www.mysite.com/directory/sub-directory/

Ideas?

Thank you Michael, it is my understanding then that my idea of doing this:

allow: /directory/$

disallow: /directory/*

Should work just fine. I will test it within Google Webmaster Tools, and let you know if any problems arise.

In the meantime if anyone else has more ideas about all this and can confirm me that would be great!

Thank you again.

Thank you Michael,

Google and other SEs actually recognize the "allow:" command:

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

The fact is: if I don't specify that, how can I be sure that the following single command:

disallow: /directory/*

Doesn't prevent SEs to spider the /directory/ index as I'd like to?

Hello here,

I am trying to figure out the correct way to tell SEs to crawls this:

http://www.mysite.com/directory/

But not this:

http://www.mysite.com/directory/sub-directory/

or this:

http://www.mysite.com/directory/sub-directory2/sub-directory/...

But with the fact I have thousands of sub-directories with almost infinite combinations, I can't put the following definitions in a manageable way:

disallow: /directory/sub-directory/

disallow: /directory/sub-directory2/

disallow: /directory/sub-directory/sub-directory/

disallow: /directory/sub-directory2/subdirectory/

etc...

I would end up having thousands of definitions to disallow all the possible sub-directory combinations.

So, is the following way a correct, better and shorter way to define what I want above:

allow: /directory/$

disallow: /directory/*

Would the above work?

Any thoughts are very welcome! Thank you in advance.

Best,

Fab.

Thank you Kurt, that's exactly what I thought but I wanted to have confirmation from the experts community.

Thank you again!

Thank you for your reply and suggestions, actually ahrefs returned quite a number of links... what I don't understand is how you can see 350 links inside the "Just-Discovered beta" of OSE... I see just 23 links for the following URL:

www.musicnotes.com/sheetmusic/ard.asp

Ideas?

Thank you guys for your replies and insights, I am more for keeping what we have lived with so far, which means leaving the system on our site disabling session IDs when bots request the pages, unless you tell me there is any downsides to do that... that's really what I am trying to find out here. Is there any downside to not serve pages with session IDs to search engines compared to users?

Thank you again.

Hello here,

I am trying to understand the best way to find sites that are affiliate of a competitor, through link research.

Typically our competitor's affiliates link to our competitor website via any of the following links:

http://www.musicnotes.com/sheetmusic/ard.asp?SID=[aff_id]&LID=[link_id]

http://click.linksynergy.com/link?id=[aff+id]&offerid=[off_id]&type=2&murl=http%3A%2F%2Fwww.musicnotes.com%2Fsheetmusic%2Fmtd.asp%3Fppn%3D[item_id]

The first link looks much easier to find, so I have tried to find the first kind of links with Google by using the "link:" clause as follows:

link:http://www.musicnotes.com/sheetmusic/ard.asp

Or, similarly, by using Open Site Explorer. But I always get 0 results! It is weird because I know there are thousands of affiliates out there with the same tracking code.

How's that possible? Why does it look impossible to find the sites I am looking for? Would you suggest any different approach?

Any ideas, suggestions and thoughts are very welcome!

Thank you in advance.

Fab.

Hello here.

When we setup our e-commerce website virtualsheetmusic.com to allow session IDs to be assigned to users back in 2001, we decided to not assign them if a bot called the page. We wanted to be sure that bots, which officially can't store cookies, wouldn't have found links containing every time different session IDs . Just to better clarify, the way session IDs are generated on our system, is the standard way: if users have cookies enabled, a cookie called PHPSESSID is created which stores the session cookie. If the cookies are not enabled, session IDs are added automatically by the system to any link URL included on the page which could potentially cause the bots to find every time different link URLs with the session ID appended to them.

Now, after 12 years, we are considering if this is still a valuable solution, or can it be detrimental or negative in some way? What are your thoughts about this issue?

Thank you in advance for any thoughts.

Fab.

Thank you very much guys for your answers, tips and insights! I will move accordingly...

If you use a MySQL database to store your website data, I think that to do this kind of automatic "archival" work by creating an automatic PHP script would take between 2 to 5 hours work. I don't see why it should take more than that.

If someone tells you that it is going to take more than that, I would be suspicious. Either the programmer is not good enough, or wants to cheat on you. That unfortunately happens more than you think!!

Be sure to ask for a step-by-step description of how they plan to complete the job. If you have doubts, please feel free to ask me, I am a pretty expert PHP programmer. I don't work for others, but just for myself (I built and keep tweaking my own websites virtualsheetmusic.com, musicianspage.com and others with very little help from external programmers).

Good luck!

Hello everyone,

here is another question for you: I have several forum postings on my websites that are pretty old and so they are sort of "dead discussion" threads. Some of those old discussion threads are still getting good views (but not new postings), and so I presume may be valuable for some users. But most of them are just answers to personal questions that I doubt someone else could be interested in. Besides that, many postings are just single, unanswered questions still waiting for an answer, forgotten, they are just sitting there, and will probably stay unanswered for years.... I don't think this may be good for SEO, am I right? How do you suggest to approach this kind of issues on forums or discussions sections on a website?

I am eager to know your thoughts on all this. Thank you in advance!

All the best,

Fab.

The way I would do it would be to make a simple PHP (or Perl) program that every day, week or month (as you may need it), archives your sitemap.xml on a specific directory on your server, and possibly zip it. As a PHP programmer myself, I can tell you that that's really simply to do. Just ask to a PHP programmer, I am sure it will make it in a couple hours!

Thank you for your reply, this is interesting and I could keep it as a "backup" solution.

Yeah, thank you James! That's wonderful! I didn't even notice the"switch to basic version" at the bottom of the page!

Thank you for your help

Hello here,

In the past I was able to find out pretty easily how many images from my website are indexed by Google and inside the Google image search index. But as today looks like Google is not giving you any numbers, it just lists the indexed images.

I use the advanced image search, by defining my domain name for the "site or domain" field:

http://www.google.com/advanced_image_search

and then Google returns all the images coming from my website.

Is there any way to know the actual number of images indexed? Any ideas are very welcome!

Thank you in advance.

Thank you Mark! Yes, I knew about that option and helps a big deal!

I appreciated your help. Thanks!

Thank you Lynn, yes sorry, I meant 50% not 0.50% (I got confused with the conversion rate). Yes, I didn't notice that the data under the graph was actually updated, looks like it is just the graph that doesn't show that kind of filtered data (too bad!).

Thank you again, I appreciated your help!

Hello here,

I am trying to find out how I can filter out pages in Google Analytics according to their bounce rate.

The way I am doing now is the following:

1. I am working inside the Content > Site Content > Landing Pages report

2. Once there, I click the "advanced" link on the right of the filter field.

3. Once there, I define to "include" "Bounce Rate" "Greater than" "0.50" which should show me which pages have a bounce rate higher of 0.50%.... instead I get the following warning on the graph:

"Search constraints on metrics can not be applied to this graph"

I am afraid I am using the wrong approach... any ideas are very welcome!

Thank you in advance.