Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Do you know any tool(s) to check if Google can crawl a URL?

Web Design

1318

esiow2013 last edited by

Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.

In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.

Thanks in advance!
1 Reply Last reply
Reply Quote 0
danwebman last edited by

I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
1 Reply Last reply
Reply Quote 1
masdigitalmedia last edited by

You could consider this online tool: http://web-sniffer.net
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

How Does Google View Hidden Content?

I have a website which contains a lot of content behind a show hide, does Google crawl the "hidden" copy?
Web Design | | jasongmcmahon

0
How to know if a wordpress theme is coded correctly for Seo

Hi, So I am curious if there is a tool to see if a site is coed properly for Google? I am running Avada, a standalone theme, yet I am also using a cache plugin. But when I search my code, its all like on one huge line. So I am curious if there is a way to verify or check if a theme is coded correctly? Thank you
Web Design | | Berner

1
Existing URL structure and how to handle new pages before migration

Hi there! Currently, our site uses underscores "_" within the url structure. We are moving to Wordpress soon (the site is currently static html) but it will be a couple of months before the migration. Here is an example of the current structure: www.oldsitestructure.com/about_us/success_stories/custom_vinyl_banners When we do change, our url structure will have hyphen's "-" to separate terms, so the preferred new structure will be: www.oldsitestructure.com/about-us/success-stories/custom-vinyl-banners The entire site (with the exception of our Wordpress blog) currently uses the old structure. We have about 10 - 15 pages we will add before our migration, my question is: Should we use the preferred url structure starting NOW or stick with the old one? And set up 301 redirects are part of the migration process? Many thanks!
Jon
Web Design | | SEOSponge

0
Question #1: Does Google index https:// pages? I thought they didn't because....

generally the difference between https:// and http:// is that the s (stands for secure I think) is usually reserved for payment pages, and other similar types of pages that search engines aren't supposed to index. (like any page where private data is stored) My site that all of my questions are revolving around is built with Volusion (i'm used to wordpress) and I keep finding problems like this one. The site was hardcoded to have all MENU internal links (which was 90% of our internal links) lead to **https://**www.example.com/example-page/ instead of **http://**www.example.com/example-page/ To double check that this was causing a loss in Link Juice. I jumped over to OSE. Sure enough, the internal links were not being indexed, only the links that were manually created and set to NOT include the httpS:// were being indexed. So if OSE wasn't counting the links, and based on the general ideology behind secure http access, that would infer that no link juice is being passed... Right?? Thanks for your time. Screens are available if necessary, but the OSE has already been updated since then and the new internal links ARE STILL NOT being indexed. The problem is.. is this a volusion problem? Should I switch to Wordpress? here's the site URL (please excuse the design, it's pretty ugly considering how basic volusion is compared to wordpress) http://www.uncommonthread.com/
Web Design | | TylerAbernethy

0
Hey, So I know wordpress is built with a user capability, but what about databasing custom user data?

Hi everyone, so I am working on a project with a friend of mine, without getting into too much detail, here's the problem. We need users to sign in, then for instance, click a button, and then have that action record a value of 1 in a database on our server (preferably accessible through the wordpress admin interface, and in .csv format so that we can make it easy to work with) Any help with databasing, or if wordpress is already built with SOME database technology, etc, etc For instance, SEOmoz's "mozpoint" system.. Wordpress comes built with the ability for users to log in and have profiles with special access, etc, but does it have the ability to log points and values to a specific user as well? Where should I look? What should I google to figure out options? Who could I call / hire?
Web Design | | TylerAbernethy

0
Whats happening with Google UK?

Within the last week we have had a handful of our rankings drop dramatically down the SERPS. About 15% but this an estimate and has not been fully investigated yet. Whilst looking into possible scenarios that could be causing this i wanted to check what the SERPS looked like for the terms that we are still holding position on. Typing "extending dining tables" into Google UK today i was amazed at what i found... Ranking in position 1 and 2 is a massive UK furniture store.
But isnt that the same landing page being returned for both positions?? It appears to be a navigation problem within the site category tags causing duplicate content. However they have been rewarded with the top two positons subsequently pushing our website onto page two. I find it so frustrating that we listen to Googles best practices when it comes to pagination issues yet this is how our hard work is rewarded! Anyone else have any thoughts about this? SERPS.jpg
Web Design | | Silkstream

0
Google Analtyics Conversion Tracking for Wordpress Life Coaching Site

Hello, How do I do conversion tracking for Google Analytics for this site: debidonner(dot)com She has a 'Thank You' page after you return from Paypal Thanks!
Web Design | | BobGW

0
Competitor Rockets to #1 and I'm looking at keyword stuffing. Will Google catch up with it?

We have a competitor whose home page rocketed up to number one, page one, on our key search term after they did a website redesign. They even beat out the original retailer for that position, as they are resellers of the product (not affiliate sales, resale in the secondary market.) They are the first to knock the original seller out of the #1 position. In the past couple of years that I have been doing in-house SEO, they have never ranked on page one for the term. I ran their site through the SEOmoz page grader for the specific search term, loading their page that is ranking, and found that they grade a “B,” but have some alerts for keyword stuffing, (the search term is on the home page 30+ times,) and they have eleven tags on said page. Aside from the two things listed above, they have pretty good site architecture on this new site, and are pretty well branded, etc. Should I expect Google to catch the keyword stuffing and eleven tags, and possibly adjust their rank? Will their keyword stuffing come back to bite them?
Web Design | | Ticket_King

0