Bingpreview/1.0b Useragent Using Adding Trailing Slash to all URLs
-
The Bingpreview crawler, which I think exists in order to take snapshots of mobile friendly pages, crawled my pages last night for the first time. However, it is adding a trailing slash to the end of each of my dynamic pages. The result is my program is giving the wrong page--my program is not expecting a trailing slash at the end of the urls. It was 160 pages, but I have thousands of pages it could do this to.
I could try doing a mod rewrite but that seems like it should be unnecessary. ALL the other crawlers are crawling the proper urls. None of my hyperlinks have the slash on the end. I have written to Bing to tell them of the problem.
Is anyone else having this issue? Any other suggestions for what to do?
The user agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
-
Will do. Forgot to mention Bing is checking into it. But for the reasons you mentioned I am still going to do the 301s. Thanks again.
-
Sounds like a plan. I'd also make every redirect a 301, just in case. Cheers.
-
Thanks for your reply Cyrus. Wow, so much to learn.
I will put in logic via a mod redirect to basically remove the trailing slash and go to the resulting url because otherwise all the trailing slash urls will be a different page of basically a 'no-product' business and the like.
These are all dynamically generated pages, so I think as long as I resolve to the 'proper' no-slash version then I won't need to worry about anything else, like a rel=canonical tag because there wont be any identical content.
Does that sound right to you?
-
On one hand I'd agree with you that you shouldn't have to rewrite those URLs on your end. On the other hand, it's usually best practice to make sure both versions of a URL (with slash and/or without) resolve to the same page. The reason for this is that:
- Search bots, including Google, will often "explore" variations of URLs for discoverability reasons - they want to make sure they are discovering all of your available content.
- People will link to you with and without trailing slashes. If they link to you with a trailing slash and your page breaks, you could be wasting link equity, to say nothing of the bad user experience of people visiting your site from the referral links
- For one reason or another it's common to append URLs with various parameters (for tracking reasons, campaings, etc) and often these URLs are generated by third party services when pointing at your site.
For all of these reasons, it's pretty common to either force redirect trailing slashes (via a 301) or make sure both versions resolve to the same content, and use a rel=canonical tag to indicate to search engines that these are indeed meant to be the same page.
On the other hand, if this is something not feasible and URLs ending in a slash are indeed different pages, you might want to carefully consider what those pages deliver to both humans and bots because it seems inevitable that both will eventually crawl and stumble upon them.
Perhaps not the answer you were looking for, but I hope it helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Removing the Trailing Slash in Magento
Hi guys, We have noticed trailing slash vs non-trailing slash duplication on one of our sites. Example:
Intermediate & Advanced SEO | | brandonegroup
Duplicate: https://www.example.com.au/living/
Preferred: https://www.example.com.au/living So, SEO-wise, we suggested placing a canonical tag on all trailing slash pointing to non-trailing slash. However, devs have advised against removing the trailing slash from some URLs with a blanket rule, as this may break functionality in Magento that depends on the trailing slash. The full site would need to be tested after implementing a blanket rewrite rule. Is any other way to address this trailing slash duplication issue without breaking anything in Magento? Keen to hear from you guys. Cheers,0 -
Trailing Slash and Non-Trailing Slash Inconsistency
Hi, Majority of the URLs on the site I'm working on are using non-trailing slash, https://example.com/page1
Intermediate & Advanced SEO | | nerdieb
https://example.com/page2
https://example.com/page3 But when it comes to the blog pages, they are using trailing slash: https://example.com/blog/specific-blog-post1/
https://example.com/blog/specific-blog-post2/
https://example.com/blog/specific-blog-pos3t/ Note that there is no duplication here since they have canonical tags. My only concern is that:
Is it okay to have this kind of structure where the blog page have trailing slash while the rest are using non-trailing slash? Keen to hear from you guys. Cheers!0 -
No content using Fetch
Wooah, this one makes me feel a bit nervous. The cache version of the site homepage shows all the text, but I understand that is the html code constructed by the browser. So I get that. If I Google some of the content it is there in the index and the cache version is yesterday. If I Fetch and Render in GWT then none of the content is available in the preview - neither Googlebot or visitor view. The whole preview is just the menu, a holding image for a video and a tag line for it. There are no reports of blocked resources apart from a Wistia URL. How can I decipher what is blocking Google if it does not report any problems? The CSS is visible for reference to, for example, <section class="text-within-lines big-text narrow"> class="data"> some content... Ranking is a real issue, in part by a poorly functioning main menu. But i'm really concerned with what is happening with the render.
Intermediate & Advanced SEO | | MickEdwards0 -
Image URL Change Catastrophe
We have a site with over 3mm pages indexed, and an XML sitemap with over 12mm images (312k indexed at peak). Last week our traffic dropped off a cliff. The only major change we made to the site in that time period was adding a DNS record for all of our images that moved them from a SoftLayer Object Storage domain to a subdomain of our site. The old URLs still work, but we changed all the links from across our site to the new subdomain. The big mistake we made was that we didn't update our XML sitemap to the new URLs until almost a week after the switch (totally forgot that they were served from a process with a different config file). We believe this was the cause of the issue because: The pages that dropped in traffic were the ones where the images moved, while other pages stayed more or less the same. We have some sections of our property where the images are, and have always been, hosted by Amazon and their rankings didn't crater. Same with pages that do not have images in the XML sitemap (like list pages). There wasn't a change in geographic breakdown of our traffic, which we looked at because the timing was around the same time as Pigeon. There were no warnings or messages in Webmaster Tools, to indicate a manual action around something unrelated. The number of images indexed in our sitemap according Webmaster Tools dropped from 312k to 10k over the past week. The gap between the change and the drop was 5 days. It takes Google >10 to crawl our entire site, so the timing seems plausible. Of course, it could be something totally unrelated and just coincidence, but we can't come up with any other plausible theory that makes sense given the timing and pages affected. The XML sitemap was updated last Thursday, and we resubmitted it to Google, but still no real change. Anyone had a similar experience? Any way to expedite the climb back to normal traffic levels? Screen%20Shot%202014-07-29%20at%203.38.34%20PM.png
Intermediate & Advanced SEO | | wantering0 -
Followup question to rand(om) question: Would two different versions (mobile/desktop) on the same URL work well from an SEO perspective and provide a better overall end-user experience?
We read today's rand(om) question on responsive design. This is a topic we have been thinking about and ultimately landing on a different solution. Our opinion is the best user experience is two version (desktop and mobile) that live on one URL. For example, a non-mobile visitor that visits http://www.tripadvisor.com/ will see the desktop (non-responsive) version. However, if a mobile visitor (i.e. iOS) visits the same URL they will see a mobile version of the site, but it is still on the same URL There is not a separate subdomain or URL - instead the page dynamically changes based on the end user's user agent. It looks like they are accomplishing this by using javascript to change the physical layout of the page to match the user's device. This is what we are considering doing for our site. It seems this would simultaneously solve the problems mentioned in the rand(om) question and provide an even better user experience. By using this method, we can create a truly mobile version of the website that is similar to an app. Unfortunately, mobile versions and desktop users have very different expectations and behaviors while interacting with a webpage. I'm interested to hear the negative side of developing two versions of the site and using javascript to serve the "right" version on the same URL. Thanks for your time!
Intermediate & Advanced SEO | | davidangotti0 -
Will ranking be improved or hurt by changing 1/5 of part numbers to key words
Note: I bold major content for your quick skim for your convenience. Does this help you decide if its a fit for your response? My site has been devastated by the Panda or unknown reasons so I need to think outside the box. I distribute industrial products with average brand recognition. I only have about 5 competitors selling this same brand. My other brand competitors are billion dollar companies that pay a lot for PPC and have sites with 10 times the product offering. Since my brand recognition is not as important as the function.I'm thinking about changing the part numbers to reflect function. This will affect about 1/5of the parts ( about 500 out of 3,000 parts) . My concern is will ranking be hurt or helped by changing these parts with these strong keywords in front of the part for such a high % of the site. The strong keywords cost $10 for a chance at a $200 sale with repeat business. Example: Current part is: 10-10 Black Plastic; which is a Big Red Truck with my brand part # as 10-10 and comes in different colors of plastic. . Keyword is Big Red Truck. I would like to put my manufactures brand in the description. My same brand competitors sell 10,000 parts and my logic is that if I have the brand in 1/5 of my parts ranking would be improved because of the % of brand per the site versus my same brand competitors.. So I would change the part # to : **Brand 10-10-Black Plastic Big Red Truck ** In conversation I would state the part as: Brand: 18 characters, Part #: 8, Material:12, Keyword: 27 If the keyword should be first I could change to: K,B P,M. Which is recommended?
Intermediate & Advanced SEO | | Wales0 -
Removing dashes in our URLs?
Hi Forum, Our site has an errant product review module that is resulting in about 9-10 404 errors per day on Google Webmaster Tools. We've found that by changing our product page URLs to only include 2 dashes, the module stops causing 404 errors for that page. Does changing our URL from "oursite.com/girls-pink-yoga-capri.html" to "oursite.com/girlspink-yoga-capri.html" hurt our SEO for a search for "girls pink yoga capri"? If so, by how much (assuming everthing else on the page is optimized properly) Thanks for your input.
Intermediate & Advanced SEO | | pano0 -
Is it fine to use an iframe for video content? Will it still be indexed on your URL?
If we host a video on a third party site and use an iframe to display it on our site, when the video is indexed in SERPs will it show on our site or on the third party site?
Intermediate & Advanced SEO | | nicole.healthline0