Can I exclude pages from my Crawl Diagnostics?
-
Right now my crawl diagnostic information is being skewed because it's including the onsite search from my website. Is there a way to remove certain pages like search from the errors and warnings of the crawl diagnostic? My search pages are coming up as:
- Long URL
- Title Element Too Long
- Missing Meta Description
- Blocked by meta-robots (Which is how I want it)
- Rel Canonical
Here is what the crawl diagnostic thinks my page URL looks like:
website.com/search/gutter%25252525252525252525252525252525252525252525252525252525
252525252525252525252525252525252525252525252525252525252525252
525252525252525252525252525252525252525252525252525252525252525
252525252525252525252525252525252525252525252525252525252525252
52525252525252525252525252525252525252525252525252Bcleaning/
Thank you,
Jonathan
-
Thanks! That was easy.
-
You can do so in robots.txt
SeoMoz bot is called rogerbot
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page with "Missing Title Tag" isn't a page
Hello, I am going through the various errors that the Moz Pro Crawl report and some non-existent pages keep coming up in the report. For example, one error category is "Missing Title Tag" with one page identified. But this page http://www.immigroup.com/news/“http%3A/crs.yorku.ca”?page=2 isn't real. It would have been a 404 were there not a redirect for everything that is /news/gobbledygook to /news. So my question is: when moz (or GA for that matter) identifies these pages as "real" and having errors, do I need to take this seriously? And what do I do about it? Thanks! George
Moz Pro | | canadageorge0 -
Clearing our on-page ranking reports?
Is there a way to "bulk delete" on-page ranking reports which are no longer relevant? I know we can delete them one at a time, but the reason I ask is that I've done a fair bit of work changing URL's, so the reports are often for old URL's which no longer exist. (yes, I made sure to do 301 redirects to the new ones!) Thanks in advance for any help!
Moz Pro | | koalatm0 -
On-page grader question
Hi there, Getting to know the Pro tools and can't find an answer to this. Can someone explain for me please? Using on page grader, I found a couple pages with an F. I scrolled downWTO where it shows the keyword phrases and under each, the URL. Clicking on the first keyword "Building site alarms"it tells me off essentially for not optimising the page for that term. The URL is "construction site security systems" which are different to building site alarms which also have their own page. I don't understand why is Moz associating this keyword with this page? I certainly haven't told it to. Please he
Moz Pro | | DaddySmurf0 -
My moz only one page was crawled
I recently moved my shopping cart from one provider to another and today moz only crawled one page, could this be because maybe google has not indexed it yet or should i be concerned? I pointed the DNS at the new cart monday night if that helps. I would have expected it to be indexed by now
Moz Pro | | SmartVapes0 -
Initiate crawl
Anyway to start the crawl of a site immediately after changes have been made? Or must you wait for the next scheduled crawl? Thanks.
Moz Pro | | dave_whatsthebigidea.com0 -
Crawl Diagnostics Shows thousands of 302's from a single url. I'm confused
Hi guys I just ran my first campaign and the crawl diagnostics are showing some results I'm unfamiliar with.. In the warnings section it shows 2,838 redirects.. this is where I want to focus. When I click here it shows 5 redirects per page. When I go to click on page 2, or next page, or any other page than page 1 for that matter... this is where things get confusing. Nothing shows. Downloading the csv reveals that 2,834 of these are all showing: URL: http://www.mydomain.com/401/login.php url: http://www.mydomain.com/401/login.php referrer: http://www.mydomain.com/401/login.php location_header: http://www.mydomain.com/401/login.php I guess I'm just looking for an explanation as to why it's showing so many to the same page and what possible actions can be taken on my part to correct it (if needed). Thanks in advance
Moz Pro | | sethwb0 -
Seomoz on-page analysis, how strict to be
Hello, In a competitive niche, how important is it to be strict with the seomoz on-page analysis? If it gives a page/keyword an A, am I good to go? Or do I need to be more strict in that. We've had some competition move above us and we want to make sure we're on-site optimized well. site: nlpca(dot)com Thanks.
Moz Pro | | BobGW0 -
Where can I view all of the Questions that I have Answered?
AFAICT there's no way to filter questions to the ones that you have answered. Is there any plan to implement a filter like this? I realise there are email notifications available but no easy way to just go back and check on your answers. No doubt I'm just missing something here so if I'm completely wrong, please point me in the right direction 🙂 Thanks very much!
Moz Pro | | NickPateman811