Seo moz has only crawled 2 pages of my site. Ive been notified of a 403 error and need an answer as to why my pages are not being crawled?
-
SEO Moz has only crawled 2 pages of my clients site. I have noticed the following.
- A 403 error message
- screaming frog also cannot crawl the site but IIS can.
- Due to the lack of crawling ability, im getting no feed back on my on page optimization rankings or crawl diagnostics summary, so my competitive analysis and optimization is suffering
Anybody have any idea as to what needs to be done to rectify this issue as access to the coding or cms platform is out of my hands.
Thank you
-
Hi Joel,
I have indeed!
I'll take all this information to the development team and hopefully have this issue resolved asap.
Thanks once again to everybody for their input.
Regards,
Andy
-
Hey Andy,
You've gotten a lot of good info here and in your email with Maura. I just wanted to add in here that the crawl trouble you're having (at least with our system) is with the http://www.inoncology.com/ home page itself.
You've got a 301 redirect from the root to the www subdomain, the www then returned a 403 forbidden to our crawler.
It may have something to do with the verification you require, it may be something more obvious on the server level. It's tough to say.
Either way, you'll want to get this addressed to make sure your site is crawlable.
I hope that helps.
Cheers,
Joel. -
Hi Lynn
Thank you for taking a look into this issue, your feedback has given me food for thought.
I am going to forward on your feedback to the powers that be on the web development side of things and see if this is an answer to the problem.
I've just looked at http://www.inoncology.com/home/ and from first review, it seems that this is where the crawlers are breaking down, as it comes to a dead end.
Hopefully this is the reason.
Thank you once again
-
Hi Andy,
Not sure why you are seeing a 403 error (I don't see this in the headers), but you seem to have an issue with your rel=canonical tag on your homepage. It is showing a relative link to "/home" which leads you to this page: http://www.inoncology.com/home/ which is actually showing a 200 OK status, but is obviously not!
Digging into the headers a bit more it seems you might have a circular redirect going on before that, I am seeing /home redirect to /content/internet/pm/inoncology/com_EN which seems to redirect back to /home/.
No 403 in all of those, but maybe the combination of wrong rel canonical tag plus the mixed up redirects is causing the moz crawler and screaming frog to choke?
-
Not at all Jesse. Thank you for taking a look
Brief summary - the pharmaceutical industry has to have a disclaimer on their pages for legal reasons. Forcing a selection/disclaimer is required by law. If you select the first option, it'll take you to the web address intended.
In terms of it being related to the disclaimer, SEO Moz was indexing it with this disclaimer up until a few months ago. There's no robot txt file as you stated, so that also cannot be the reason for this.
hhhhmmm........the mystery continues!!!
-
no robots file so it's not that. I do notice that your landing page has an annoying popup that forces a decision and then redirects you upon choosing it... I'm not 100% sure but that might have something to do with it. I can't seem to get past that and onto the site itself honestly. Everytime I click a selection I get to another page with a popup.
If I were an average internet browser *(which in this instance I am) I would leave your site and not return. Poor user experience hurts ultimately, and that was what I experienced.Not trying to be rude, just trying to be helpful. Sorry if I came off cross.
G'luck!
-
the url is http://www.inoncology.com/
-
Please share the URL of your website so we can check.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I mprove site visibility and keyword ranking for new product site
Hi, Sorry if this is a ridiculous post as I am really new to SEO, but I haven't had this problem with other sites! We had a website www.r-dna.co.uk that was never promoted or used very much as it was early days in the product lifecycle. The product (is called R-DNA or Remote Data Network Analysis) is now live so we re-branded and re-launched the site - it has now been live since the beginning of September but we still only have 0.35% visibility and very little ranking in our keywords. We are also using Google Adwords to try and generate business and have registered with numerous online business directories. I have been blogging to update content, tweeting and updating our facebook page, but we still aren't getting the traffic or visibility increases that we have experienced with our other sites. The MOZ site crawl shows 5 medium priority issues (duplicate title page & missing meta description tag), but no major issues. I know its probably fairly early days for a "new" site, but wondered if anyone could advise if there is anything wrong which would explain our lack of visibility.
Moz Pro | | sharon.bathurst0 -
Web Site Migration Testing and SEO-QA Automation?
Hey Mozzers, Are there any good Migration-SEO-QA Tools out there? Given a prioritized list of URLs and prioritized list of Keywords, is there a tool that can compare basic SEO factors, old URL vs. new URL, and identify all the specific gaps that need to be fixed? Here is a basic SEO-QA acceptance checklist, for porting any website. . . . Until the porting work is completed we cannot accept the new website. Givens: 1. A list of the Top 100 URLs from the old site, prioritized by conversion rates, landing page traffic, and inbound links. 2. A list of the planned 404 - mapped URLs, old to new site, from the porting team. 3. A list of the current Top 200 Keywords, prioritized. 4. A good amount of SEO work has already been done, by several professionals, for the current (old) site. **How to evaluate if the new site will be acceptable to Google? Check ON-PAGE SEO Factors... ** **. . . that is, the NEW site must be AS GOOD AS (or better than) the current (old) site,
Moz Pro | | George.Fanucci
in the eyes of Google, to preserve the On-Page SEO work already done. ** Criteria: URLs ok? :: Is the URL mapping ok, old to new, best web page? LINKS ok :: Are all internal LINKS and keyword Anchor Text ported? TEXT ok :: On-page content, TEXT and keywords ok? TITLE ok :: HTML Title and title keywords ok? DESCRIPTION ok :: HTML Meta Description ok? H1, H2 ok :: HTML H1, H2 and keywords ok? IMG kwds :: HTML IMG and ALT keywords ok? URL kwds :: URL - keywords in new URLs ok? Potential porting defects: Keywords in URL missing: Keywords in HTML Title missing: Keywords in Meta Description missing: Any internal LINKS or Link anchor text missing: Keywords in Page TEXT missing: H1, H2 missing keywords: HTML IMG alt-text, IMG file URLs, any missing keywords: Notes: Until the porting work is completed we cannot accept the new site, or set a target date for potential cutover. There are eight (8) data items per URL, and about one hundred (100) URLs to be considered for SEO-QA before going live. We were expecting to cutover before the end of February, at the latest. There is no point in doing full QA acceptance-tests until the porting work is completed. QA spot-checks have found far too many defects. About 60% of the landing-page traffic comes via the top 40 URLs. With over 100 URLs to look at, it can take more than a week or two just to do SEO-QA in detail, manually, item-by-item, page-by-page, side-by-side, old vs. new. Spot-checks indicate a business disaster would occur unless the porting defects are fixed before going live. _Any Migration-QA Tools?_Given a prioritized list of URLs and prioritized list of Keywords, is there a tool that can compare basic On-Page SEO factors, old URL vs. new URL, and identify most of the specific gaps that need to be fixed before going live with the new site? _ *** Edit: Any comments on the SEO criteria, tools, or methods will be appreciated!_0 -
Lag time between MOZ crawl and report notification?
I did a lot of work to one of my sites last week and eagerly awaited this week's MOZ report to confirm that I had achieved what I was trying to do, but alas I still see the same errors and warnings in the latest report. This was supposedly generated five days AFTER I made the changes, so why are they not apparent in the new report? I am mainly referring to missing metadata, long page titles, duplicate content and duplicate title errors (due to crawl and URL issues). Why would the new crawl not have picked up that these have been corrected? Does it rely on some other crawl having updated (e.g. Google or Bing)?
Moz Pro | | Gavin.Atkinson0 -
Moz Crawl Test: WordPress sites with and without /feed and /trackback entires?
I have multiple WP websites and on some of the websites, on my Moz Crawl test, I see an entry for every blog post but also entries for /feed and /trackback for that single blog post. For example, www...com/someArticle www....com/someArticle/feed www...com/someArticle/trackback 1. Can anyone explain why the Crawl test is picking up the /feed and /trackback items? Is it simply because they are 301 redirects to the original post (www...com/someArticle)? 2. What setting(s) in WordPress are making this information appear? Or is it just that the site(s) that have the /feed and /trackback are displaying "normal" behavior for a WP site with a lot of trackbacks and feed entires? 3. Should /fee and /trackback, as well as /author be blocked in robots.txt? Thanks in advance for your advice and input!
Moz Pro | | Titan5520 -
Issues with Moz producing 404 Errors from sitemap.xml files recently.
My last campaign crawl produced over 4k 404 errors resulting from Moz not being able to read some of the URLs in our sitemap.xml file. This is the first time we've seen this error and we've been running campaigns for almost 2 months now -- no changes were made to the sitemap.xml file. The file isn't UTF-8 encoded, but rather Content-Type:text/xml; charset=iso-8859-1 (which is what Moveable Type uses). Just wondering if anyone has had a similar issue?
Moz Pro | | BriceSMG0 -
Find a 4xx or 5xx link referenced in an SEO Crawl Report
So I just got the Crawl Diagnostics report for a client site and it came back with a number of 4xx errors and even 1 5xx error. So while I can find the URL that has the problem, I cannot find the pages that have the links pointing to these non-existent or problematic pages. Normally I would just search the database for the site, but in this case I don't have access to it as the site is on a proprietary platform with no access other than to the CMS. Is there anyway to get the linking URL from the report? Thanks!
Moz Pro | | farlandlee0 -
Settings to crawl entire site
Not sure what happened but I started a third campaign yesterday and only 1 pages was crawled, The other two campaigns has 472 and 10K respectively. What is the proper setting to choose in the beginning of campaign setup to have the entire site crawled. Not sure what I did different and I must be reading the instructions incorrectly. Thanks, Don
Moz Pro | | NicheGuy210 -
Schedule crawls for 2 subdomains every 24 hours
I saw at this link: http://pro.seomoz.org/tools/crawl-test "As a PRO member, you can schedule crawls for 2 subdomains every 24 hours, and you'll get up to 3,000 pages crawled per subdomain." However I am having trouble finding where to schedule this 24 hour crawl in my Pro Dashboard. I did not see the option for this setting in the crawl diagnostics tab or in the campaign settings section from the dashboard home page. Can you help? thanks! Michael
Moz Pro | | texmeix0