Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Do you know any tool(s) to check if Google can crawl a URL?

Web Design

1318

esiow2013 last edited by

Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.

In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.

Thanks in advance!
1 Reply Last reply
Reply Quote 0
danwebman last edited by

I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
1 Reply Last reply
Reply Quote 1
masdigitalmedia last edited by

You could consider this online tool: http://web-sniffer.net
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Homepage Slider How to Handle H1 and H2's

Working on a site with a slider on the homepage, I dislike them but owner wants to keep in place. Currently, the slider has 4 slides with different images but the same text, so the slider has 4 slides with 4 identical H2 tags and accompanying text. There is no H1 tag on the page at all. It seems to me that a better solution would be to change the first slide to be H1 (with the target keyword) and rework the text in the other slides as H2 tags to appeal to the user. This does mean that the H1 and H2 tags in the slider would be styled the same. Is this a sensible approach?
Web Design | | GrouchyKids

1
Interlinking using Dynamic URLs Versus Static URLs

Hi Guys, Could you kindly help us in choosing best approach out of mentioned below 2 cases. Case. 1 -We are using: We interlink our static pages(www.abc.com/jobs-in-chennai) through footer, navigation & by showing related searches. Self referential Canonical tags have been implemented. Case. 2 -We plan to use: We interlink our Dynamic pages(www.abc.com/jobs-in-chennai?source=footer) through footer, navigation & by showing related searches. Canonical tags have been implemented on dynamic urls pointing to corresponding static urls Query 1. Which one is better & expected to improve rankings. Query 2. Will shifting to Case 2 negatively affect our existing rankings or traffic. Regards
Web Design | | vivekrathore

0
How does Google look at strings added to a URL

For example: http://localhost:3000/en-US/app/a-knsmtrhqrqs/personal where knsmtrhqrqs is a string Can Google tell this is a string and what's their policy? Will it hurt rankings? Thank you.
Web Design | | RoxBrock

0
Can someone please help with technical question!

I have noticed that our website tool to get a quote does not work with active scripting disabled is this bad? How many people have this disabled?
Web Design | | BobAnderson

0
How to verify http://bizdetox.com for google webmaster tools

Hey guys i tried to to make a Preferred Domain choice in webmaster tools, but it is not allowing me to save my choice bec its asking me to verify that i own http://bizdetox.com How do i go about doing that and what are the steps I have already verified www.bizdetox.com
Web Design | | BizDetox

0
Do I need to redirect soft 404s that I got from Google Webmaster Tools?

Hi guys, I got almost 1000+ soft 404s from GWT. All of the soft 404s produce 200 HTTP status code but the URLs are something like the following: http://www.example.com/search/house-for-rent (query used: house for rent) http://www.example.com/search/-----------rent (query used:-------rent) There are no listings that match these queries and there is an advanced search that is visible in these pages. Here are my questions: 1. Do I need to redirect each page to its appropriate landing page? 2. Do I need to add user sitemap or a list of URLs where they can search for other properties? Any suggestions would help. 🙂
Web Design | | esiow2013

0
Google also indexed trailing slash version - PLEASE HELP

Hi Guys, We redesigned the website and somehow our canonical extension decided to add a trailing slash to all URLs. Previously our canonical URLs didn't have a trailing slash. During the redesign we haven't changed the URLs. They remained same but we have now two versions indexed. One with trailing slash one without. I've now fixed the issue and removed the the trailing slash from canonical URLs. Is this the correct way of fixing it? Will our rankings be effected in a negative way? Is there anything else I need to do. The website went live last Tuesday. Thanks
Web Design | | Jvalops

0
Anyone Know Of A WP Plugin for Before and After Pictures

Hey Mozzers- Does anyone know of a good WordPress Plugin for displaying before and after images. This could be for a docs office, etc. Any feedback would be great! Thanks!
Web Design | | kpeacy

0