Page not Accesible for crawler in on-page report
-
Hi All,
We started using SEOMoz this week and ran into an issue regarding the crawler access in the on-page report module. The attached screen shot shows that the HTTP status is 200 but SEOMoz still says that the page is not accessible for crawlers. What could this be?
Page in question
http://www.tiasnimbas.edu/Executive_MBA/pgeId=307Regards,
Coen
-
Most welcome
You can install Mozbar if you are using FF or Chrome, it has a lot of useful features.
-
Ok super thank you
-
I've just checked your Google Cache for this page and it seems to be fine. I advise you don't worry that much about SEOmoz robot.
-
Hi, Yes i have check it through the webmaster tools :
`Result: HTTP/1.1 200 OK Cache-Control: public, max-age=1 Content-Type: text/html; charset=utf-8 Content-Encoding: gzip Expires: Tue, 06 Dec 2011 14:27:57 GMT Last-Modified: Tue, 06 Dec 2011 14:27:56 GMT Vary: Accept-Encoding Server: Microsoft-IIS/7.5 City: /Executive_MBA/pgeId=307 Set-Cookie: ASP.NET_SessionId=wb3k3i45cxmag3mh31swghbz; domain=.tiasnimbas.edu; path=/; HttpOnly X-AspNet-Version: 2.0.50727 X-Powered-By: ASP.NET Date: Tue, 06 Dec 2011 14:27:56 GMT Content-Length: 12991 The robot.txt allows all crawlers:` User-agent: * Sitemap: http://www.tiasnimbas.edu/sitemap.xml
-
Hi have you tried to check if this page accessible for Google robots in Webmaster Panel. Because sometimes there are some sort of software which are blocking SEOmoz robot.
Also check your robots.txt file if there are no any restrictions. For ex. it might be setup to accept only Google robot but not others.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to find orphan pages
Hi all, I've been checking these forums for an answer on how to find orphaned pages on my site and I can see a lot of people are saying that I should cross check the my XML sitemap against a Screaming Frog crawl of my site. However, the sitemap is created using Screaming Frog in the first place... (I'm sure this is the case for a lot of people too). Are there any other ways to get a full list of orphaned pages? I assume it would be a developer request but where can I ask them to look / extract? Thanks!
Technical SEO | | KJH-HAC1 -
Page Authority on Huffington Post
I was looking into my website backlinks and noticed that a link Huffington Post post has only one page authority while the other post has high page authority like 30 or 40. http://www.huffingtonpost.com/toby-nwazor/is-it-time-to-retire-the-_b_10610052.html Please suggest what can be the issue. Thanks!
Technical SEO | | 1MS0 -
Are image pages considered 'thin' content pages?
I am currently doing a site audit. The total number of pages on the website are around 400... 187 of them are image pages and coming up as 'zero' word count in Screaming Frog report. I needed to know if they will be considered 'thin' content by search engines? Should I include them as an issue? An answer would be most appreciated.
Technical SEO | | MTalhaImtiaz0 -
How Does Google's "index" find the location of pages in the "page directory" to return?
This is my understanding of how Google's search works, and I am unsure about one thing in specific: Google continuously crawls websites and stores each page it finds (let's call it "page directory") Google's "page directory" is a cache so it isn't the "live" version of the page Google has separate storage called "the index" which contains all the keywords searched. These keywords in "the index" point to the pages in the "page directory" that contain the same keywords. When someone searches a keyword, that keyword is accessed in the "index" and returns all relevant pages in the "page directory" These returned pages are given ranks based on the algorithm The one part I'm unsure of is how Google's "index" knows the location of relevant pages in the "page directory". The keyword entries in the "index" point to the "page directory" somehow. I'm thinking each page has a url in the "page directory", and the entries in the "index" contain these urls. Since Google's "page directory" is a cache, would the urls be the same as the live website (and would the keywords in the "index" point to these urls)? For example if webpage is found at wwww.website.com/page1, would the "page directory" store this page under that url in Google's cache? The reason I want to discuss this is to know the effects of changing a pages url by understanding how the search process works better.
Technical SEO | | reidsteven750 -
Can you 301 redirect a page to an already existing/old page ?
If you delete a page (say a sub department/category page on an ecommerce store) should you 301 redirect its url to the nearest equivalent page still on the site or just delete and forget about it ? Generally should you try and 301 redirect any old pages your deleting if you can find suitable page with similar content to redirect to. Wont G consider it weird if you say a page has moved permenantly to such and such an address if that page/address existed before ? I presume its fine since say in the scenario of consolidating departments on your store you want to redirect the department page your going to delete to the existing pages/department you are consolidating old departments products into ?
Technical SEO | | Dan-Lawrence0 -
Too many on page links
Hello I have about 800 warnings with this. Example of one url with this problem is: http://www.theprinterdepo.com/clearance?dir=asc&order=price I was checking and I think all links are important. But I suppose that if I put a nofollow on the links on the left which are only for navigation purposes I can get rid of these warnings. Any other idea?
Technical SEO | | levalencia10 -
Dealing with 404 pages
I built a blog on my root domain while I worked on another part of the site at .....co.uk/alpha I was really careful not to have any links go to alpha - but it seems google found and indexed it. The problem is that part of alpha was a copy of the blog - so now soon we have a lot of duplicate content. The /alpha part is now ready to be taken over to the root domain, the initial plan was to then delete /alpha. But now that its indexed I'm worried that Ill have all these 404 pages. I'm not sure what to do.. I know I can just do a 301 redirect for all those pages to go to the other ones in case a link comes on but I need to delete those pages as the server is already very slow. Or does a 301 redirect mean that I don't need those pages anymore? Will those pages still get indexed by google as separate pages? Please assist.
Technical SEO | | borderbound0 -
Pages not being found in serp
Hi I'm helping a collegue with his website. For what ever reason the pages in the Solutions Menu are not being found in the search result for keywords related to the pages. (Homepage mainly comes up in the search result). Does anyone have any advise to why this may be happening? *To give you a bit of a background understanding, previously all the menu content was copied (which I made him change), he also had hidden text on some pages (i made him remove, white text on white background) plus the url structure changed as well. Persoanlly I think he is over using , links, internal linking is not great & the general content is not great in the menu. Your Thoughts are welcomed, thank you.
Technical SEO | | Socialdude0