Why do i get multiple variations of my url with ?order=asc and ?view=list at the end of it in my crawl report?
-
I just did a crawl for one my clients to validate any error in the structure. Next thing I know is that the website have multiple variation of the same url with query like ?order=asc and ?view=list at the end of it.
I am wondering why these url variations appears in the crawl I just did since bots aren't suppose to go further thant the ? normally.
Just to show you a couple of url's of my crawl test.
<colgroup><col width="484"></colgroup>
| https://test.com/exemple/?per_page=9 |
| https://test.com/exemple/?per_page=15 |
| https://test.com/exemple/?per_page=30 |
| https://test.com/exemple/?orderby=popularity |
| https://test.com/exemple/?orderby=date |
| https://test.com/exemple/?orderby=price |
| https://test.com/exemple/?orderby=price-desc |
| https://test.com/exemple/?order=asc |
| https://test.com/exemple/?order=desc |
| https://test.com/exemple/?view=list |Thank you Guys
-
Thank you Samantha your answer is very useful !
-
Hey there!
Sam from Moz's Help Team here! As far as I'm aware, Google and other crawlers do crawl past the '?', unless certain parameters are disallowed within the robots.txt. If the URL is: https://test.com/exemple/?per_page=9, a search engine will see something like test com exemple 'search' etc. Google recommends blocking all Internal Search Results in the Robots.txt file - for Rogerbot, it would look something like this
User-agent: Rogerbot
Disallow: ?utmHere is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt
I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a way to view difficulty for related keyword suggestions?
I'd really like to view the difficulty/organic CTR/priority of suggested related keywords without having to click on each individually - is there a way to do this?
Moz Bar | | avfundstack2 -
Crawl Test : Error attempting to request HTTPS page
Hallo When I launch the crawl report I get csv file with this error : 804 : HTTPS (SSL) error encountered when requesting page.
Moz Bar | | micvitale
Error attempting to request page; see title for details. Website is https://bastabollette.it0 -
Crawl Test Takes Long Time
Hi Moz, I have submitted our website for a crawl test. Usually it would only take a few hours to do the crawl. However this time, it takes quite long time and the result still shows in progress 😞 This is a small website which only contains less than 10 pages. Just wondering if this is our website setting issue or it is a technical issue at your end? Many thanks in advance. sFjAERG.png
Moz Bar | | russellbrown0 -
Should I be getting an 'A' Grade for basic words in a page title?
It seems as long as a word in the page title matches a word(s) within the page content you will get an 'A' grade. Should I be replacing these keywords with higher monthly searches and lower difficulty? It appears deceptive when you test a new website and all the pages get an 'A' for a word as basic as 'about.' Please advise.
Moz Bar | | Joseph.Lusso0 -
Moz Crawl Showing Duplicate Content But It's Not?!
Unfortunately I can't give out the URL, but here's the deal... I have two URL's which have completely different content on them but are being crawled as duplicate content. Any Idea how that would happen? I'm not seeing any errors in WMT's. Has anyone seen this before? Is the duplicate content reporting based on a % of the page content matching as the same?
Moz Bar | | Swarm-SEO0 -
How Can I print a report that shows the general results in the dashboard view?
I would like to print a report for a client, that looks like the dashboard report, more visual. Is there a print to PDF for the general info, which the dashboard provides? moz-screen-shot.png
Moz Bar | | jlmarketing1 -
Moz just did my weekly "crawl" and claims my meta titles and descriptions are "missing," but they aren't. What's up with this?
I can check for my titles and descriptions in the source. They are there, always have been. what in the hell?
Moz Bar | | LifeisRosie0