Bingpreview/1.0b Useragent Using Adding Trailing Slash to all URLs
-
The Bingpreview crawler, which I think exists in order to take snapshots of mobile friendly pages, crawled my pages last night for the first time. However, it is adding a trailing slash to the end of each of my dynamic pages. The result is my program is giving the wrong page--my program is not expecting a trailing slash at the end of the urls. It was 160 pages, but I have thousands of pages it could do this to.
I could try doing a mod rewrite but that seems like it should be unnecessary. ALL the other crawlers are crawling the proper urls. None of my hyperlinks have the slash on the end. I have written to Bing to tell them of the problem.
Is anyone else having this issue? Any other suggestions for what to do?
The user agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
-
Will do. Forgot to mention Bing is checking into it. But for the reasons you mentioned I am still going to do the 301s. Thanks again.
-
Sounds like a plan. I'd also make every redirect a 301, just in case. Cheers.
-
Thanks for your reply Cyrus. Wow, so much to learn.
I will put in logic via a mod redirect to basically remove the trailing slash and go to the resulting url because otherwise all the trailing slash urls will be a different page of basically a 'no-product' business and the like.
These are all dynamically generated pages, so I think as long as I resolve to the 'proper' no-slash version then I won't need to worry about anything else, like a rel=canonical tag because there wont be any identical content.
Does that sound right to you?
-
On one hand I'd agree with you that you shouldn't have to rewrite those URLs on your end. On the other hand, it's usually best practice to make sure both versions of a URL (with slash and/or without) resolve to the same page. The reason for this is that:
- Search bots, including Google, will often "explore" variations of URLs for discoverability reasons - they want to make sure they are discovering all of your available content.
- People will link to you with and without trailing slashes. If they link to you with a trailing slash and your page breaks, you could be wasting link equity, to say nothing of the bad user experience of people visiting your site from the referral links
- For one reason or another it's common to append URLs with various parameters (for tracking reasons, campaings, etc) and often these URLs are generated by third party services when pointing at your site.
For all of these reasons, it's pretty common to either force redirect trailing slashes (via a 301) or make sure both versions resolve to the same content, and use a rel=canonical tag to indicate to search engines that these are indeed meant to be the same page.
On the other hand, if this is something not feasible and URLs ending in a slash are indeed different pages, you might want to carefully consider what those pages deliver to both humans and bots because it seems inevitable that both will eventually crawl and stumble upon them.
Perhaps not the answer you were looking for, but I hope it helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How To Shorten Long URLS
Hi I want to shorten some URLs, if possible, that Moz is reporting as too long. They are all the same page but different categories - the page advertises jobs but the client requires various links to types of jobs on the menu. So the menu will have: Job type 1
Intermediate & Advanced SEO | | Ann64
Job type 2
Job Type 3 I'm getting the links by going to the page, clicking a dropdown to filter the Job type, then copying the resulting URL from the address bar. Bu these are really long & cumbersome. I presume if I used a URL shortener, this would count as redirects and alsonot be good for SEO. Any thoughts? Thanks
Ann0 -
Duplicate content on URL trailing slash
Hello, Some time ago, we accidentally made changes to our site which modified the way urls in links are generated. At once, trailing slashes were added to many urls (only in links). Links that used to send to
Intermediate & Advanced SEO | | yacpro13
example.com/webpage.html Were now linking to
example.com/webpage.html/ Urls in the xml sitemap remained unchanged (no trailing slash). We started noticing duplicate content (because our site renders the same page with or without the trailing shash). We corrected the problematic php url function so that now, all links on the site link to a url without trailing slash. However, Google had time to index these pages. Is implementing 301 redirects required in this case?1 -
Urls in Bilingual websites
1-I have a bilingual website. Suppose that I am targeting a page for keyword "book" and I have included it in that page url for the English version: English version: www.abc.com/book Can I use the translation of "book" in the second language of the website url instead of "book" ? Please let me know which of the following urls are right " French Verison: www.abc.com/fr/book or www.abc.com/fr/livre livre=Book in French 2- Does Google have any tool to check if the second language page of the website has exactly the same content as the English version. What I want to do is for example for a certain page in English version, my targeted keyword is "book" . So my content would be around books. But in the French version of this page, I want to focus on keyword "Pencil" in French instead of "book". Is it wrong or any consequences? That was the main reason for the question number one. Because if it is ok to do what I explained in item 2 then I will set my urls like: In English : www.abc.com/book In French: www.abc.com/fr/crayon crayon=Pencil in French
Intermediate & Advanced SEO | | AlirezaHamidian0 -
Using abbreviations in URL - Matching Keyword
We have a website that uses /us/, /ca/, /va/, etc for URLs of the different U.S. states. How much better is it (or is it at all better) to use /california/ or /virginia/ instead in our URLs to rank for searches that include the name of those states?
Intermediate & Advanced SEO | | Heydarian0 -
Keyword/Content Consistency
My question is: If you have a keyword that is searched more when it's spelled wrong then when it's spelled right - what do you do? Do you do the misspelled word or keep true to the spelling and say oh well to SEO? Also - Along the same lines of that question: What if you have a keyword that has a - in the middle of it. For instance: website and web-site (this isn't the keyword just an example). and drupal website is searched more then drupal web-site but wordpress web-site is searched more then wordpress website. Technically website is the correct spelling and way to write it, but people put web-site (again not the case in reality - just an example).
Intermediate & Advanced SEO | | blackrino0 -
Adding index.php at the end of the url effect it's rankings
I have just had my site updated and we have put index.php at the end of all the urls. Not long after the sites rankings dropped. Checking the backlinks, they all go to (example) http://www.website.com and not http://www.website.com/index.php. So could this change have effected rankings even though it redirects to the new url?
Intermediate & Advanced SEO | | authoritysitebuilder0 -
Trailing Slash: Lost in Redirection?
Question here, but first the lead in. As you all know, 301 redirects don't pass on 100% of link juice. I've set up my site using htaccess to redirect all non-ww to www and redirect all URLs to have a trailing slash. FYI, the preferred domain is selected in WMT and canonical URLs appear in the head section of all pages. So now what happens when sites that link to mine don't include either the www or the trailing slash, which is actually quite common? Of course, asking the site own to correct the link is ideal, but that's not always possible. So if thousands of links on external sites are linking to http://www.site.com instead of http://www.site.com/, won't lots of link juice get lost in redirection? I can't think of anything more I can do to the URLs to reduce duplicate content and juice dilution. Thoughts? Kevin
Intermediate & Advanced SEO | | kwoolf0 -
Google Maps results doesn't show my site url but rather the maps url, why is this?
For several of my clients landing pages that show up in the Maps results the website url has been overwritten by the maps url (maps.google.com). Even though on my places page I have the correct website set up. Does anyone have any idea why they would be doing this and how I can correct it? Thanks kinldy in advance, Aaron. maps-url.png
Intermediate & Advanced SEO | | afranklin0