Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Do you know any tool(s) to check if Google can crawl a URL?

Web Design

1323

esiow2013 last edited by

Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.

In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.

Thanks in advance!
1 Reply Last reply
Reply Quote 0
danwebman last edited by

I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
1 Reply Last reply
Reply Quote 1
masdigitalmedia last edited by

You could consider this online tool: http://web-sniffer.net
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Changed url and now not listing

hi guys I had this page "https://www.stauntonrook.co.uk/marketing-services/offline/corporate-identity/" indexed in google at position 5 for a search term, after some research in the keyword planner I decided to change my url to https://www.stauntonrook.co.uk/marketing-services/offline/logo-design/ because I wanted the keyword in the url. I added a 301 redirect to the htaccess and removed the old page from google using webmaster tools fully expecting the new url to drop back in anytime hopefully in a better position than 5, now after a good week or so, still nothing 😞 also doesn't seem to be in any of the other search engines either I've updated the xml too any ideas?
Web Design | | Staunton_Rook

0
Googlebot Reports All URLs as Unreachable

Webmaster Tools is reporting that all of our site's URLs - located www.zuken.com - are "unreachable." The URLs display correctly in all browsers. We recently switched hosting providers, but they assure us there is no security setting that would be causing this issue. Any ideas? Is this a glitch with Webmaster Tools?
Web Design | | Zuken

0
Can anyone help me detect some SEO improvements onpage please...

Can anyone help me detect some SEO improvements onpage please... I have shortened the website URl so its not easily found when searched via search engines.. http://goo.gl/GlfMRl Please have a look and give me some tips. Thanks
Web Design | | Nettv

0
What is the longest you would go back to ressurrect links that should have been 301's?

I have never thought of anything beyond a site that was possibly developed a month or two ago, but an interesting possible client has come along and begs a question. They had their site "redesigned" in April 2014 and it appears whomever did the work did not realize what a 301 was for. Using ahrefs or MajesticSEO, they have gone from roughly 15,000 referring pages to 500 and the time line perfectly intersects the redesign. Sooooo, just wondering if any of you geniuses has ever gone back that far to try and pull off a 301.... I am actually just thinking of a link building / content marketing plan but thought it was an interesting question. Thanks for the help, Robert
Web Design | | RobertFisher

1
Does Google count the domain name in its 115-character "ideal" URL length?

I've been following various threads having to do with URL length and Google's happiness therewith and have yet to find an answer to the question posed in the title. Some answers and discussions have come close, but none I've found have addressed this with any specificity. Here are four hypothetical URLs of varying lengths and configurations: EXAMPLE ONE:
my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (115 characters) EXAMPLE TWO: sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (126 characters) EXAMPLE THREE: www.sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (130 characters) EXAMPLE FOUR: http://www.sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (137 characters) Assuming the examples contain appropriate keywords and are linked to appropriate anchor text (etc.,) how would Google look upon each? All I've been able to garner thus far is that URLs should be as short as possible while still containing and contextualizing keywords. I have 500+ URLs to review for the company I work for and could use some guidance; yes, I know I should test, but testing is problematical to the extreme; I look to the collective/accumulated wisdom of the MOZVerse for help. Thanks.
Web Design | | RScime25

1
Does Google have problem crawling ssl sites?

We have a site that was ranking well and recently dropped in traffic and ranking. The whole site is https and and not just the shopping pages. Thats the way the server is setup, they make whole site https. My manager thinks the drop in ranking is due to google not crawling https. I think contrary, but would like some feedback on this. Site is here
Web Design | | anthonytjm

0
I have a button that repeats it self many times on same page, what can i do so button name does not affect my SEO?

I have a shopping car button named "Add to car" but it repeats on many pages on my website, is this affecting my seo? If yes.. What should i do so it does not affect? Should button appear on hover? Thanks
Web Design | | SeMeAntoja

0
Any reaction to the announcement from Google that 'signed in' searches won't pass through search query info to analytics?

Seems like SEO is about to get that much harder: http://analytics.blogspot.com/2011/10/making-search-more-secure-accessing.html Any thoughts on this?
Web Design | | PaulM01

1