Bingpreview/1.0b Useragent Using Adding Trailing Slash to all URLs

[friendoffood]

The Bingpreview crawler, which I think exists in order to take snapshots of mobile friendly pages, crawled my pages last night for the first time.  However, it is adding a trailing slash to the end of each of my dynamic pages.  The result is my program is giving the wrong page--my program is not expecting a trailing slash at the end of the urls.  It was 160 pages, but I have thousands of pages it could do this to.

I could try doing a mod rewrite but that seems like it should be unnecessary.  ALL the other crawlers are crawling the proper urls.  None of my hyperlinks have the slash on the end.  I have written to Bing to tell them of the problem.

Is anyone else having this issue?  Any other suggestions for what to do?

The user agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b

[friendoffood]

Will do.  Forgot to mention Bing is checking into it.  But for the reasons you mentioned I am still going to do the 301s.  Thanks again.

[Cyrus-Shepard]

Sounds like a plan. I'd also make every redirect a 301, just in case. Cheers.

[friendoffood]

Thanks for your reply Cyrus.  Wow, so much to learn.

I will put in logic via a mod redirect to basically remove the trailing slash and go to the resulting url because otherwise all the trailing slash urls will be a different page of basically a 'no-product' business and the like.

These are all dynamically generated pages, so I think as long as I resolve to the 'proper' no-slash version then I won't need to worry about anything else, like a rel=canonical tag because there wont be any identical content.

Does that sound right to you?

[Cyrus-Shepard]

On one hand I'd agree with you that you shouldn't have to rewrite those URLs on your end. On the other hand, it's usually best practice to make sure both versions of a URL (with slash and/or without) resolve to the same page. The reason for this is that:

• Search bots, including Google, will often "explore" variations of URLs for discoverability reasons - they want to make sure they are discovering all of your available content.
• People will link to you with and without trailing slashes. If they link to you with a trailing slash and your page breaks, you could be wasting link equity, to say nothing of the bad user experience of people visiting your site from the referral links
• For one reason or another it's common to append URLs with various parameters (for tracking reasons, campaings, etc) and often these URLs are generated by third party services when pointing at your site.

For all of these reasons, it's pretty common to either force redirect trailing slashes (via a 301) or make sure both versions resolve to the same content, and use a rel=canonical tag to indicate to search engines that these are indeed meant to be the same page.

On the other hand, if this is something not feasible and URLs ending in a slash are indeed different pages, you might want to carefully consider what those pages deliver to both humans and bots because it seems inevitable that both will eventually crawl and stumble upon them.

Perhaps not the answer you were looking for, but I hope it helps.