Why do i get multiple variations of my url with ?order=asc and ?view=list at the end of it in my crawl report?
-
I just did a crawl for one my clients to validate any error in the structure. Next thing I know is that the website have multiple variation of the same url with query like ?order=asc and ?view=list at the end of it.
I am wondering why these url variations appears in the crawl I just did since bots aren't suppose to go further thant the ? normally.
Just to show you a couple of url's of my crawl test.
<colgroup><col width="484"></colgroup>
| https://test.com/exemple/?per_page=9 |
| https://test.com/exemple/?per_page=15 |
| https://test.com/exemple/?per_page=30 |
| https://test.com/exemple/?orderby=popularity |
| https://test.com/exemple/?orderby=date |
| https://test.com/exemple/?orderby=price |
| https://test.com/exemple/?orderby=price-desc |
| https://test.com/exemple/?order=asc |
| https://test.com/exemple/?order=desc |
| https://test.com/exemple/?view=list |Thank you Guys
-
Thank you Samantha your answer is very useful !
-
Hey there!
Sam from Moz's Help Team here! As far as I'm aware, Google and other crawlers do crawl past the '?', unless certain parameters are disallowed within the robots.txt. If the URL is: https://test.com/exemple/?per_page=9, a search engine will see something like test com exemple 'search' etc. Google recommends blocking all Internal Search Results in the Robots.txt file - for Rogerbot, it would look something like this
User-agent: Rogerbot
Disallow: ?utmHere is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt
I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Cannot Crawl ... 612 : Page banned by error response for robots.txt.
I tried to crawl www.cartronix.com and I get this error: 612 : Page banned by error response for robots.txt. I have a robots.txt file and it does not appear to be blocking anything www.cartronix.com/robots.txt Also, Search Console is showing "allowed" in the robots.txt test... I've crawled many of our other sites that are similarly set up without issue. What could the problem be?
Moz Bar | | 1sixty80 -
Sorry, but that URL is inaccessible?
Hi, I am trying to grade some pages and keywords using the "On-page grader" tool but for each URL that I try, the tool returns me a "Sorry, but that URL is inaccessible". The thing is that I have already used previously and without any problem some of these URLs. In fact, I have just realized that, while the same URL (for example: www.lacasadelaaldea.com) works in the "On-page optimization tool", it doesn't in the "On-page grader" right now. I have looked if someone could have experienced the same issue and I have found some other threads talking about it... so I have checked with my hosting provider that there is no firewall or any other thing causing this problem but they can't find anything. How do you make the call to the server? What could be happening? Thanks in advance, Juan
Moz Bar | | lcdla0 -
I'm getting, "you're not using the rel="canonical" META attribute" in my crawl diagnotic
I'm running a campaign crawler through Moz on this particular page: http://www.henley.ac.uk/executive-education/leadership-and-management-programmes/ but I'm getting a notifcaiton from Moz saying, "you're not using the rel="canonical" META attribute" I don't understand what this means!! Has anyone else had this problem, or can they help me understand what this means and how to fix it? Oh, and Happy Thanksgiving from the UK! Virginia
Moz Bar | | blackboxideas0 -
URL inaccessible for On Page Grader
I am trying to use the on page grader however it is not working with my website. The URL is as follows: https://capbeast.com. I have been trying to read in older posts to see if https is now supported or not but have not found anything. I know there is no robots.txt issue as I am able to run the crawl test on our website fine. Is the issue on my end in regards to configuration or is due to DDos attacks? Any help would be appreciated. Thanks
Moz Bar | | MisterStitches0 -
Moz Crawl Test Trying to Crawl Contact Form Submit Button Location?
Moz Crawl Test for some reason is trying to Crawl a contact form Widget Submit Location. My obvious guess is that obviously the crawl cannot submit to the required fields…..I believe this because they're only kicking back these errors on the pages I have a contact form widget on. http://crawfordspest.com/pest-control/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
Moz Bar | | Funk-Creative-Media
http://crawfordspest.com/tree-services/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/lawn-care/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/specialty-services/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404 Can you shed any insight to this? I'm a bit worried that I'll have to complete gut the contact form which was one of the major requests my client requested. Or in a worse scenario make all fields not required. It would let so much spam in. I have never seem anything like this at all. But I've learned a lot from Moz, and with major errors like 404 damage Domain Authority greatly. I've fixed 404 issues with newly acquired clients existing sites and tracked through Moz and the domain authority flies up once these errors are fixed. Along with fixing what Webmaster Tools through Google reports back. ..... Let me know if you have any expertise on this matter.0 -
Moz Report: Number of Domains linking to Domain
In the Moz report, what does the "number of domains linking to domain" indicate? I'm analyzing our competitors link profile and noticed some numbers as high as 2369232? Is there a form that explains the reports a little more in detail? Thanks,
Moz Bar | | WebRiverGroup0 -
Does Moz Pro generate similar keyword phrases in a list (preferably showing their difficulty %) or is it only one phrase at a time with no similar words/phrases suggested?
I just signed up for Moz Pro but the keyword research seems to only let you try one keyword phrase at a time. Is there a way for it to give related keywords along with their difficulty % info, etc. It is far too slow and inconvenient doing one at a time.
Moz Bar | | SavingSpotlight0 -
How do I get Moz to crawl my site?
If I'm not mistaken, Moz will crawl my site once a week and let me know of any errors, notices etc.. But can I ask them to crawl once I have updated all the errors? or do I have to wait for them to do this automatically?
Moz Bar | | david.smith.segarra0