Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Do you know any tool(s) to check if Google can crawl a URL?

Web Design

1293

esiow2013 last edited by

Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.

In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.

Thanks in advance!
1 Reply Last reply
Reply Quote 0
danwebman last edited by

I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
1 Reply Last reply
Reply Quote 1
masdigitalmedia last edited by

You could consider this online tool: http://web-sniffer.net
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

How can I increase my website speed?

Hi, I hope you are doing good. My website speed is too much slow. Mobile speed is 12 and Desktop speed is 39. Please check my website speed.
&
What can I do for my website to increase speed?
What is best way to increase website speed. Here is my website: https://www.myqurantutor.com/
Web Design | | Bigbrand

0
NO Meta description pulling through in SERP with react website - Requesting Indexing & Submitting to Google with no luck

Hi there, A year ago I launched a website using react, which has caused Google to not read my meta descriptions. I've submitted the sitemap and there was no change in the SERP. Then, I tried "Fetch and Render" and request indexing for the homepage, which did work, however I have over 300 pages and I can't do that for every one. I have requested a fetch, render and index for "this url and linked pages," and while Google's cache has updated, the SERP listing has not. I looked in the Index Coverage report for the new GSC and it says the urls and valid and indexable, and yet there's still no meta description. I realize that Google doesn't have to index all pages, and that Google may not also take your meta description, but I want to make sure I do my due diligence in making the website crawlable. My main questions are: If Google didn't reindex ANYTHING when I submitted the sitemap, what might be wrong with my sitemap? Is submitting each url manually bad, and if so, why? Am I simply jumping the gun since it's only been a week since I requested indexing for the main url and all the linked urls? Any other suggestions?
Web Design | | DigitalMarketingSEO

1
Shortened URL is breaking when URL is in Upper Case

Hi there, Currently I'm having some troubling mitigating an odd occurrence with some redirected shortened URLs being in upper case. Here is how they should be behaving - www.rhinosec.com/webapp -> https://rhinosecuritylabs.com/landing/sample-report-webapp-pentest/
www.rhinosec.com/network -> https://rhinosecuritylabs.com/landing/sample-report-network-pentest/
www.rhinosec.com/se -> https://rhinosecuritylabs.com/landing/social-engineering-example-report/ but when the /______ is capitalized - for example - WEBAPP, NETWORK, SE; WordPress either gives me a 404 or guesses the pages and lands on: NETWORK = https://rhinosecuritylabs.com/assessment-services/network-penetration-testing/
SE = https://rhinosecuritylabs.com/assessment-services/secure-code-review/
WEBAPP = 404 I was wondering if this discrepancy should be taken care of in the Htaccess file, Cloudflare, or WordPress redirect plug-in?
Web Design | | BCaudill

0
Wow, does a website's hosting company have that much affect on SEO?

As a small SEO agency, we also handle hosting for some of our clients. Our clients' sites are Wordpress. We set them up with a Bluehost account with a dedicated IP address, and spend a lot of time focusing on load times (implementing a CDN, optimizing images, installing W3 Total cache and using recommended settings, etc.). Last month, we had a client inform us that they are bringing their web marketing efforts in-house, so they switched to a new hosting provider and took their (existing) site to the new hosting company. They kept their old Google Analytics code installed, so I can still see how much traffic they're getting. Since switching to a new host, despite the load times taking longer, no CDN, and other errors that came up prior to us spending time "optimizing" the website, their organic traffic has increased by 26%. Same exact website, same inbound link profile. According to Webmaster Tools, their impressions and clicks have also seen dramatic increases. So now, obviously, I'm considering looking into other options for the hosting of our other clients' websites. From your experience, and especially when it comes to Wordpress websites, do you think that a hosting company can make that big of a difference in terms of SEO? I've heard of positive results from people who have used WP-Engine, and other Wordpress-dedicated hosting companies, but I just find it hard to believe that we spent so much time on load-time-specific ranking factors and come to find out, a different hosting company would have made a huge difference. Any thoughts/feedback?
Web Design | | georgetsn

1
Hi, I have a doubt. If we want to hide unwanted text in a web page its possible with "" tag. And my question "does a search engine crawl those text? help me.

I want to hide a lot of text behind my site page. I know its possible with that tag. But in what way a search engine looks at those text? Hidden or they are crawled and indexed.
Web Design | | FhyzicsBCPL

0
301 forwarding during site migration problem - several url versions of the same page....

Hello, I'm migrating from an old site to a new site, and 301 forwarding many of the pages... My key problem is this I'm seeing www.website.com/ indexed in SE and www.website.com/default.aspx in showing as URL when I'm on homepage - should I simply 301 forward both of these? Then for several internal pages there are 2/3 versions of each page indexed. Canonicalization issues. Again, I'm wondering whether I should 301 forward each URL even if there are several different indexed URLs for the same page? Your advice will be welcome! Thanks in advance - Luke
Web Design | | McTaggart

0
Next Google Index..?

Hi Guys, Does anybody have an idea when the next Google index is due roughly and if there is anyway I can tell approx when these are due to happen and how would I know? Thanks In advance, Craig Fenton IT
Web Design | | craigyboy

0
New URL Strucutre

Hello all! I'm new to SeoMoz and would like to introduce myself and say thanks for all of the great content. This is definitely the premium website I have been looking for. My question is regarding a new URL structure that will be implemented with our website redesign. We are looking to make our URL's more user and SEO friendly. We will be using 301 redirects to reroute our existing structure when the site goes live. Background
I work in a company that retails automotive tires. I would consider class (performance, off-road, highway, etc.) to be the most important category, but based on our analytics we receive much more traffic from brand terms. This is most likely because our customers are not educated on specific tire classes and generally do head term searches or search by brand / product name. Ultimately the will have to buy from a certain class based on driving habits. Link and URL Structure
(Parameters) The model page includes links to individual tire size specific pages. We are using the model page as our primary SEO page because the only difference from the 5 - 60 tire size specific pages per model will be the tire size itself. This means that all of the remaining content will be duplicated multiple times over. All links to the tire size specific page from the model page will have a rel="nofollow" attribute attached to it. All tire size specific pages will have a rel="canonical" tag pointing back to the model page Out of the following structures, which would you consider to be the best format: 1. http://www.company.com/category/brand/model
or
2. http://www.company.com/brand/category/model
or
3. http://www.company.com/tires/category/model We will append the specific tire-size to the URL on the item specific pages. Any other options or strategies will be greatly appreciated. Thanks in advance for your help!
Web Design | | kauffmantire

0