Welcome to the Q&A Forum

Thanks again Ryan!

My home page has links to major cities.  If someone chooses a specific city I want to give them the choice to choose a suburb within the city,  With say 50 cities and 50 suburbs for each city that's 2500 links on the home page.  In order to avoid that many links on the home page (or any page) I would like to have just the 50 cities and pull up the suburbs as an ajax call that search engines would not read/crawl.  This would be better than clicking on a main city and then getting the city page which they then can choose a suburb.  Better to do it all at once.

Is it a bad idea to ajax the subregions on the home page and to code it so Google, Bing, other search engines don't crawl or even see anything on the home page related to the suburbs?  The search engines will still find the suburb links because they will be listed on each main city page.

Hi Ryan,
The sitemap has yet to be submitted as I've been cleaning up url issues first.  However, Google has crawled 15,000 pages anyway, and Bing several thousand.  Everything is lowercase on the site now, and there are only 9 backlinks (all no-follow) from just 2 domains, as I haven't yet started building them.  I'll wait a couple more months and see if the number gets closer to zero,. 
Thanks. Ted

I have a site that first got indexed about a year ago for several months.  I shut it down and opened it up again about 5 months ago.  2 months ago I discovered that the upper case in the pages:  www.site.com/some-detail/Bizname/product  was a no-no.  There are no backlinks to these pages yet, so for search engines I put in 301 redirects to the lower case version thinking after a few weeks Bing and Google would figure it out and no longer try to crawl them.  FYI there are thousands of these pages, and they are dynamically created.

Well, 2 months later google is still crawling the upper case urls even though it appears that only the lower case are in the index (from when I do a site:www.site.com/some-detail ) search.

Bing is also crawling the upper case although I'm not seeing any of the upper case pages and only a small percentage of the lower case ones show using a site:www.site.com/details.... command

Assuming there are no backlinks will they eventually stop crawling those uppercase pages?   If not, again assuming there are no backlinks, should I 410 the upper case pages, or will that remove any credit I am getting for the page having existed for over a year prior to changing the upper to lower?

I recently added some ajax pages to automatically fill in small areas of my site upon page loading.  That is, the user doesn't have to click anything.  Therefore when Google and Bing crawl the site the ajax is executed too.  However, my understanding is that does not mean Google and Bing are also crawling the ajax content.

I actually would prefer that the content  would be not be executed OR crawled by them.  In the case of Bing I would prefer that the content not even be executed because indications are that the program exits the ajax page for Bing because Bing isn't retaining session variables which that page uses, which makes me concerned that perhaps when that happens Bing isn't able to even crawl the main content..dunno..So, ajax execution seems potentially risky for normal crawling in this case.

I would like to simply have my program skip the ajax execution for Google and Bing by recognizing them in the useragent and using an If robot == Y skip ajax approach.  I assume I could put the ajax program in the robots.txt file but that wouldn't keep Bing from executing it (and having that exit problem mentioned above).  It would be simpler to just have them skip the ajax execution altogether.

Is that ok or is there a chance the search engines will penalize my site if they find out (somehow) that I have different logic for them than for the actual users?  In the past this surely was not a concern but I understand that Google is increasingly trying to become like a browser so may increasingly have a problem with this approach.

Thoughts?

Thanks Travis,

I have discovered that for some users the initial page loads up to 3 times nearly immediately without it being visual.  It never happens to me or any or the browser/system combinations I use on remote machine testing - even if I match someones setup who is getting the problem, but there is no question it is happening.  This was triggering the robot message I was giving.  I don't yet know the cause as the typical culprits don't apply.  I relaxed the rule by 1 more load in 30 seconds, which is why you didn't get a message.  Am going to use someone's computer tomorrow that gets the problem to try and narrow it down.

Agree on the human testing.  Thanks for the suggestions.

take care

Hi Travis.

Thanks for the info re Screaming Frog.

I didn't white list your IP.  I just changed the number in my files to something else, which would already be unblocked since a day has passed.

The scenario you gave would be quite rare and wouldn't create a block because that requires everything happening in 30 seconds or less (and the session wouldn't expire in that time frame), or that the same IP address also tried to crawl my site with a bot in the useragent (your scenario with screaming frog)..  But yours and Max's experience is looking more and more like it is commonplace, and I'm the fool who hasn't known that's what's happening because I can't distinguish between you and a robot (which doesn't keep sessions).

All I need is to verify that I have a sessions problem.  Here's what it takes:

1. use a desktop or laptop
2. remove all cookies related to qjamba
3. go to http://www.qjamba.com
4. choose an option (restaurants) and a location (Saint Louis) and click
5. dont do anything else - just close out the tab
6. If your ip address changes, let me know when you do this so I can find it in the logs

In all my usage it keeps the session between steps 3 and 4.  It looks like for you and Max it doesn't do that, which means many of my users would be having the same terrible experience as you, for the 3 months since it has been live.  It's a disaster.  But I have to first verify that it really is a problem and unfortunately I have to rely on strangers like you who are experiencing the problem to do that.

If you just do those steps I promise we'll be done

Thanks, Ted

Massimiliano, my guess of your path was the most logical conclusion based on the fact that I have 3 records of the urls you went to on my site, and showing that the program didn't keep any session variables between the 3 urls you came to.  You first went to wildwood.  Then you went to the home page.  This implies that you either did that in a new tab, or you hit the back key, or you modified the url and removed the wildwood part to go to the home page, as opposed to clicking on something on the page.  Telling me I'm wrong at least lets me know I may have a serious problem to fix, but you are mistaken to think that this is a robot problem.  It is a php session variable problem, apparently, that none of my extensive (hundreds of hours) testing has ever had.

This is a serious problem unrelated to the OP and about 100 times more important than the OP that I was hoping to get some help with because it is very difficult to diagnose without feedback from those having the experience that you had with my site,. However, that's my problem I'll have to deal with.  I don't know if you just don't remember or aren't telling me because you think it is a robot problem, but if you do happen to recall the steps (or at least tell me it was all done in the same tab or you hit the back key) I'd appreciate whatever it is you can tell me.  If I can't solve the problem it probably means I'll have to shut down my website which I've put more than 4 years of my life into.  Seriously.

Thanks for your various other responses though..  Take care. Ted

Massimiliano,

Can you tell me your steps that led to that error?  It looks like you went directly to www.qjamba.com/local-coupons/wildwood/mo/all   and then  you opened up a separate tab and went to www.qjamba.com  and then either refreshed the home page or opened the home page again in another tab -- all within 30 seconds.  That's the only way I have been able to reproduce this , because it looks for 3 searches without any existing session within 30 seconds by the same ip address, and the home page wipes out the session and cookies, and those are the urls the db table shows that you went to, and in that order.

Normally a user stays in the same tab, so with the 2nd search will have a session -- but your ip had no session with each search.  And, normally you can't go to the home page from a location page.  So, I'm confused as to what you did if it wasn't like what I wrote above.  If you didn't do this then I'm worried of a serious programming problem having to do with the php sessions getting dropped between pages.

I"ve put a lot of time into this website and a ton of testing of it too, and just went live a few months ago, so these kinds of problems are disheartening.  Ironically, your experience is almost identical to that of Travis, except that in your case you must have moved a little faster since you got a different message.  But, it would REALLY help me to get some feedback from both of you confirming what I wrote or setting me straight if you did something different.

Geez, I'm so pedantic sometimes.  Just need to understand what this means:

<<or, you="" make="" a="" new="" page.="" it="" gets="" crawled.="" <strong="">Checking if it's indexed... no, no, no, no, yes?! That's how long it takes.>></or,>

How do you do the bolded?  site:www.site.com/thepage "my content change on the page" ?

And, you did say one can change and not the other yet the page really has been indexed, right?

Here's an example:

I get a 404 error for this:

http://webcache.googleusercontent.com/search?q=cache:http://www.qjamba.com/restaurants-coupons/ferguson/mo/all

But a search for qjamba restaurant coupons gives a clear result as does this:

site:http://www.qjamba.com/restaurants-coupons/ferguson/mo/all

What is going on?  How can this page be indexed but not in the Google cache?

I should make clear that the page is not showing up with any kind of error in webmaster tools, and Google has been crawling pages just fine.  This particular page was fetched by Google yesterday with no problems, and even crawled again twice today by Google  Yet, no cache.

Great answer Monica -- thank you!

It occurs to me that a malicious program could set up thousands of links to dummy pages on a website:

www.mysite.com/dynamicpage/dummy123

www.mysite.com/dynamicpage/dummy456

etc..

How is this normally handled?  Does a developer have to look at all the parameters to see if they are valid and if not, automatically create a 301 redirect or 404 not found? This requires a table lookup of acceptable url parameters for all new visitors.

I was thinking that bad url names would be rare so it would be ok to just stop the program with a message, until I realized someone could intentionally set up links to non existent pages on a site.