Why my site not crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Has the Crawl Test gone?
Just checked the new Moz, am I right in thinking the super useful crawl test functionality has gone? I use it for existing sites to download all the title tags and meta name descriptions, is there more to come??
Moz Pro | | Karen_Dauncey0 -
Does Open Site Explorer purposefully not crawl some sites?
I use both SEOmoz's Open Site Explorer and Web Master Tools to find backlinks when conducting link audits. WMT always finds more links than OSE; I understand Google's database is bigger. But what is interesting to me is that it seems that a large percentage of the links WMT finds that OSE does not are real crappy links that I don't want. That makes me wonder if SEOmoz decides not to crawl certain, low quality, sites? Just curious.
Moz Pro | | ILM_Marketing0 -
Crawl Errors from URL Parameter
Hello, I am having this issue within SEOmoz's Crawl Diagnosis report. There are a lot of crawl errors happening with pages associated with /login. I will see site.com/login?r=http://.... and have several duplicate content issues associated with those urls. Seeing this, I checked WMT to see if the Google crawler was showing this error as well. It wasn't. So what I ended doing was going to the robots.txt and disallowing rogerbot. It looks like this: User-agent: rogerbot Disallow:/login However, SEOmoz has crawled again and it still picking up on those URLs. Any ideas on how to fix? Thanks!
Moz Pro | | WrightIMC0 -
Is there a way to pull ALL rankings for a site for a set of keywords?
I'm fairly new to SEOMoz, and am looking for a way to pull all rankings for a site with the keywords I've listed, not just the top ranking. So for instance if my site ranks for the keyword "widget maker in MO" in two spots. The current rank report will only show me the top result. I know I can click on each keyword individually and see a full list of the top 50, but I'm looking for a more automated way of doing this (I'm the only SEO at my agency and time is an issue). I previously used Advanced Web Ranking and the tool would like me pull a CSV of the top 30 sites for each keyword I was tracking and then I could just search for my site and pull those results easier. Anyone know of a shortcut or report I should use? Thanks!
Moz Pro | | OsbornBarr0 -
Finding page authority for a list of sites
I went to Distilled's linklove conference earlier this week. Many of the presenters showed spreadsheets where they had pulled in Page Authority for a long list of sites. What tool can you use to quickly do this? For instance, I want to be able to copy a list of 100 sites into a tool and have it spit out the PA for every site...anybody know how to do this?
Moz Pro | | znotes0 -
Crawl Diagnostics Summary
Sorry if I am not asking in the right place. On Crawl Diagnostics Summary it says this right..?? : "To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days.". so it's passed a day and it still doesn't show nothing. It says "Processing Crawl Data for 358 pages" How much should i wait??
Moz Pro | | Dussk0