Seo moz has only crawled 2 pages of my site. Ive been notified of a 403 error and need an answer as to why my pages are not being crawled?
-
SEO Moz has only crawled 2 pages of my clients site. I have noticed the following.
- A 403 error message
- screaming frog also cannot crawl the site but IIS can.
- Due to the lack of crawling ability, im getting no feed back on my on page optimization rankings or crawl diagnostics summary, so my competitive analysis and optimization is suffering
Anybody have any idea as to what needs to be done to rectify this issue as access to the coding or cms platform is out of my hands.
Thank you
-
Hi Joel,
I have indeed!
I'll take all this information to the development team and hopefully have this issue resolved asap.
Thanks once again to everybody for their input.
Regards,
Andy
-
Hey Andy,
You've gotten a lot of good info here and in your email with Maura. I just wanted to add in here that the crawl trouble you're having (at least with our system) is with the http://www.inoncology.com/ home page itself.
You've got a 301 redirect from the root to the www subdomain, the www then returned a 403 forbidden to our crawler.
It may have something to do with the verification you require, it may be something more obvious on the server level. It's tough to say.
Either way, you'll want to get this addressed to make sure your site is crawlable.
I hope that helps.
Cheers,
Joel. -
Hi Lynn
Thank you for taking a look into this issue, your feedback has given me food for thought.
I am going to forward on your feedback to the powers that be on the web development side of things and see if this is an answer to the problem.
I've just looked at http://www.inoncology.com/home/ and from first review, it seems that this is where the crawlers are breaking down, as it comes to a dead end.
Hopefully this is the reason.
Thank you once again
-
Hi Andy,
Not sure why you are seeing a 403 error (I don't see this in the headers), but you seem to have an issue with your rel=canonical tag on your homepage. It is showing a relative link to "/home" which leads you to this page: http://www.inoncology.com/home/ which is actually showing a 200 OK status, but is obviously not!
Digging into the headers a bit more it seems you might have a circular redirect going on before that, I am seeing /home redirect to /content/internet/pm/inoncology/com_EN which seems to redirect back to /home/.
No 403 in all of those, but maybe the combination of wrong rel canonical tag plus the mixed up redirects is causing the moz crawler and screaming frog to choke?
-
Not at all Jesse. Thank you for taking a look
Brief summary - the pharmaceutical industry has to have a disclaimer on their pages for legal reasons. Forcing a selection/disclaimer is required by law. If you select the first option, it'll take you to the web address intended.
In terms of it being related to the disclaimer, SEO Moz was indexing it with this disclaimer up until a few months ago. There's no robot txt file as you stated, so that also cannot be the reason for this.
hhhhmmm........the mystery continues!!!
-
no robots file so it's not that. I do notice that your landing page has an annoying popup that forces a decision and then redirects you upon choosing it... I'm not 100% sure but that might have something to do with it. I can't seem to get past that and onto the site itself honestly. Everytime I click a selection I get to another page with a popup.
If I were an average internet browser *(which in this instance I am) I would leave your site and not return. Poor user experience hurts ultimately, and that was what I experienced.Not trying to be rude, just trying to be helpful. Sorry if I came off cross.
G'luck!
-
the url is http://www.inoncology.com/
-
Please share the URL of your website so we can check.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why did Moz crawl our development site?
In our Moz Pro account we have one campaign set up to track our main domain. This week Moz threw up around 400 new crawl errors, 99% of which were meta noindex issues. What happened was that somehow Moz found the development/staging site and decided to crawl that. I have no idea how it was able to do this - the robots.txt is set to disallow all and there is password protection on the site. It looks like Moz ignored the robots.txt, but I still don't have any idea how it was able to do a crawl - it should have received a 401 Forbidden and not gone any further. How do I a) clean this up without going through and manually ignoring each issue, and b) stop this from happening again? Thanks!
Moz Pro | | MultiTimeMachine0 -
Duplicate pages coming from links from the login page - what should we do about them?
This is a follow on to an earlier question which was well answered by Dirk Ceuppens regarding abnormal crawl issues. We are seeing that the issues relating to Duplicate Pages are coming from links from the login page which shows information about where the user was redirected from. For example, if the visitor is not logged on and wishes to wish-list an item, they will be redirected to the login page, with the item code and intended action in the url; which can then continue on to the desired page once logged on. The MOZ crawler is seeing these pages as having Duplicated Content whilst they are all the same apart from a piece of information in the URL. Should we be blocking these duplications? Are they a risk to us? What should we be doing? Many thanks, Sarah
Moz Pro | | Mutatio_Digital0 -
How to Avoid Duplicate Page Content errors when using Wordpress Categories & Tags?
I get a lot of duplicate page errors on my crawl diagnostics reports from 'categories' and 'tags' on my wordpress sites. The post is 1x link and then the content is 'duplicated' on the 'category' or 'tag' that is added to the page. Should I exclude the tags and categories from my sitemap or are these issues not that important? Thanks for your help Stacey
Moz Pro | | skehoe1 -
Only One page crawled..Need help
I have run a website in Seomoz which have many URLs with it. But when I saw the seomoz report that showing Pages Crawled: 1. Why this is happen my campaign limit is OK Tell me what to do for all page crawling in seomoz report. wV6fMWx
Moz Pro | | lucidsoftech0 -
SEO-Experts help answer my newbie questions!
Hello! Thanks for coming to my rescue; I really appreciate it. I am a newbie at SEO and I still haven't got the full picture of everything in my mind that covers linkbuiling, on page optimization, domain names, website creating, etc. I've really pretty much all the available guides on WarriorForum and BHW without much help on the topics above since most of the information was either provided by some person who didn't know what they were talking about, the information was outdated, or I didn't know what they meant. I have so many questions that I hope some SEO-Expert could answer my newbie questions so that I won't get penalized for my two sites that I currently own. One of them was slapped by Yahoo!, Bing, AOL, and most other search engines with the exception of Google (It's only a matter of time of course). The other site I own is in the making currently without much on-page optimization. Before I start doing some on-page depletion t to my two sites so they both get de-indexed and such because of keyword stuffing, fail link building, and other factors, I wanted to cover the problems I currently have about SEO and this site. 1) Using the SEO-MOZ 'keyword analysis tool', what is the ideal percentage difficulty a newbie like me could take easily with just a few links here and there including 5 posts/pages of great content? (I usually only write 5 posts that cover 500+ words each). 2) Do I need hosting to become successful or to get more on-page optimization for my websites? I currently use Go-Daddy forwarding with masking so that when my site shows up in google the content comes from blogger. 3) What is "Self Cannibalization"? I ALWAYS get that error for both my sites when I use the 'on page optimzation tool' from SEO-MOZ. 4) Using Blogger, What exactly are labels and does using this add any SEO value to my website? 5) I've read so many damn articles about ALT text and Title text for blogger; nothing explained what I put in it though. What am I supposed to put in it that will help me with my on page optimization? (Stuff like do I use spaces or dashes, do I put my keyword in there, how many characters should I not exceed, do I put one word or two words?) 6) On my website, I 'accidentally' didn't know that copy and pasting images from paint straight to blogger would be a bad idea because in the html I saw it was f'ed up because there was literally random characters everywhere for the file name. Since my site is a tutorial site, I have over 50+ images in all my posts combined that I've copied and pasted from paint.exe to the Blogger post. Should I reupload all of them to blogger or keep it the way it is? Will I be penalized for this or nothing would happen? Is there a benefit to fixing it? 7) For a new website that is less than 1 week old, when can I start building backlinks? If I can start building backlinks now, what is the ideal number per day that I should create? 😎 Does post every single page or post of content I have on my website to other Web 2.0s like Squidoo, Wordpress, or Blog penalize my website for duplicate content or what? If I do this, does it give me any SEO benefit? 9) How much stronger is a .gov backlink than a .edu backlink? How much stronger are both of those compared to a regular extension backlink? 10) If I am building backlinks, should I link just my main home page URL or all internal URLs to? 11) I see several people that build backlinks to other websites by typing in a comment and throwing their link in there. Like question 10, should I post ALL the links to my website (Internal + Home) or just my main page link? 12) What is better; .com, .net, or .org? What about .info, .us, and .biz in terms of SEO benefit? 13) I have several pages on my site in which Google indexed them and I deleted the post. Now when I search my site on Google and I click on the link to the post, it is pretty much a 'dead link' because the post cannot be found. Does this harm my site in any way, and how can I take these dead pages off? 14) One of my sites is already indexed by Google but not Yahoo! What should I do to get my site indexed? It's already been about a week. Thanks for attempting to answers my questions. I don't have anything to give, but, I can choose 3 'good answers' for helping me!
Moz Pro | | 6786486312640 -
Is there a way to get the SEO campaign crawl to run from outside the US?
We are working on a international site that uses IP redirection. I am currently trying to crawl and monitor the Australian version of the site, however the pro-campaign crawler is running from the US and only giving me US data. Is there a way to tell the crawler to run through a local proxy so we can track international versions? Thanks!
Moz Pro | | MBASydney0 -
I need a tool/tools to extract keywords from say 50 sites in one niche and then check the rank tor those sites
This is for telemarketing of seo services I want to have some insight into an industry before I call them could this be done with the adwords keyword tool api and then exported to excel. It would also be nice to have data on backlinks say from seo moz opensite explorer.. Its just that the research you do before you even call a potential client is so time consuming and you can never really check to see how they are ranking for there main keywords manually. We are trying to automate as much of this initial research as possible... Any Idea's Thanks
Moz Pro | | duncan2740 -
Is this an error in SEO Moz Pro Web App?
In the rankings section if I add these two terms to monitor They are both monitored: Fresh fish fresh fish Do search engines give more weight to capitals on some small scale or is this an error in the app? Thanks, Daniel.
Moz Pro | | iSenseWebSolutions0