Crawl Diagnostics Summary Problem
-
We added our website a Robots.txt file and there are pages blocked by robots.txt. Crawl Diagnostics Summary page shows there is no page blocked by Robots.txt. Why?
-
Hey there,
Thanks for the question. How you have your robots.txt set is actually preventing all bots from even touching on those pages, not just the engines.
If you had a directive allowing RogerBot access to those pages it would be able to touch on them and register that they are blocked from the Search Engines in the robots.txt.
Since our crawler strictly adheres to the robots.txt file you won't have anything populated there.I hope that makes sense. Feel free to reach out if you need more information.
Cheers,
Joel. -
Thanks Federico,
Can we use meta robots noindex and robots.txt together?
-
I am guessing here, but Moz crawler does not respect your robots.txt file. Instead, if you want pages not to be crawled, try using the meta robots noindex for a change and see what happens.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
moz crawl is stopped?
moz stopped indexing the links due to some updates? can some one confirm me thanks
Moz Pro | | 42409300125323700 -
I got an 803 error yesterday on the Moz crawl for most of my pages. The page loads normally in the browser. We are hosted on shopify
I got an 803 error yesterday on the Moz crawl for most of my pages. The page loads normally in the browser. We are hosted on shopify, the url is www.solester.com please help us out
Moz Pro | | vasishta0 -
Problem with Forums
I am trying to respond in the forums, but keep getting an error. 1 error prohibited this form from being saved: Response can't be blank anyone else having issues?
Moz Pro | | netviper0 -
How to make a Crawl Report readable?
Hi! I am trying to find out how I make my CSV report neat, so I can interprete the report. I know have a CSV report with just numbers and text all in one column. I tried the button text to columns but that doesn't work because when I do that at column A it overwrites column B in which I have the same problem! Thanks
Moz Pro | | HetCommunicatielokaal0 -
Problem after opensiteexplorer update
i have problem every time put external link in linkdetective.com its always give me no 17.500 no profile back link but when i post old list from open site explorer that list i got it before update it work and give me result but from like 2 week every list i generate from open site explorer it always give me the same result 17.500 no back link profile any help ???
Moz Pro | | osama_evearabia.com0 -
My Campaign only crawled 3 pages on my site
On my first crawl of a new campaign, the software only crawled 3 pages. XXXaceXXXscholarships.org any ideas?
Moz Pro | | Santaur0 -
Opensite Explorer CSV Problems
I'm getting a few formatting errors in the CSVs I export from Opensite Explorer, after I've opened them in Excel. Specifically, in an Inbound Links export, there are several merged fields creating new rows. This seems to be caused by line returns in the Title column. At the moment I'm sorting these all out by hand, but it's a bit of a chore when there are lots of records. However it has to be done, because you can't sort of filter properly otherwise. Has anyone else had similar problems. If so. I'd really appreciate any tips to a) stop it happening, of b) speed up / automate the process of cleaning up the CSVs. Many thanks Ben
Moz Pro | | atticus70