Do you know any tool(s) to check if Google can crawl a URL?
-
Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.
In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.
Thanks in advance!
-
I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
-
You could consider this online tool: http://web-sniffer.net
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google analytic's API information
I have multiple websites and instead of having to log in to each google analytics I want to create a dashboard inside my MIS that has the audience overview graph, is there any way to use API to do this? Is there a way to
Web Design | | BobAnderson0 -
Fetch as Google not showing Waypoints.js on scroll animation
So I noticed that my main content underneath 4 reasons to choose LED Habitats did not show up in Fetch as Google as well as a few other sections. The site being brand new, so I'm not sure how this will be indexed. What happens is, as the user scrolls the content is brought in using Waypoints and Animate.css which offers an engaging yet simple user experience. I'm just afraid that If the content doesn't show up in "Fetch as Google" in webmaster tools that this content will never be found / indexed by Google. There are thousands of sites that use this library, I'm just curious what I'm doing wrong.. or what I can do. Is there a way for me to keep the simple animations but keep Google Happy at the same time? I took a screen shot of "Fetch as Google" and you can see blatant missing sections which are the sections animated by the waypoints library. Thanks for listening! Robert ZqgLWHi
Web Design | | swarming0 -
I want to create a 301 redirect on a WordPress site, nothing's working...
Hello all, I'm hoping someone out there can give me a hand with this. I'm trying to modify my .htaccess file so that the site will go from maxcarnagemusic.com to www.maxcarnagemusic.com and also, so viewers will be redirected to www.maxcarnagemusic.com/home when they try to access the site. I've tried a few different things, including adding the 301 redirect plugin for Wordpress, but nothing seems to work. Can someone out there show/tell me how to create an htaccess file that will execute as much. I apologize in advance, my Apache experience is very, very limited. Thank you all in advance!
Web Design | | maxcarnage0 -
May I know what's the "CSS Class" here?
ul.nav a#homelink { text-indent: -9999px; background: url('http://www.elegantthemes.com/preview/MyProduct/wp-content/themes/MyProduct/images/home-icon.png') no-repeat center 6px; padding: 0px; } ul.nav a#homelink span { display: block; padding: 5px 9px 8px 5px; width: 16px; } I'm having problems in adding back the "house" icon instead of a "Home" text in my menu. I don't what "CSS class" should I use to be added in the Home menu button (Appearance>Menu). I'm currently using a "MyProducts" Wordpress theme by ElegantThemes. Thanks in advance!
Web Design | | esiow20131 -
Does Google count the domain name in its 115-character "ideal" URL length?
I've been following various threads having to do with URL length and Google's happiness therewith and have yet to find an answer to the question posed in the title. Some answers and discussions have come close, but none I've found have addressed this with any specificity. Here are four hypothetical URLs of varying lengths and configurations: EXAMPLE ONE:
Web Design | | RScime25
my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (115 characters) EXAMPLE TWO: sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (126 characters) EXAMPLE THREE: www.sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (130 characters) EXAMPLE FOUR: http://www.sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (137 characters) Assuming the examples contain appropriate keywords and are linked to appropriate anchor text (etc.,) how would Google look upon each? All I've been able to garner thus far is that URLs should be as short as possible while still containing and contextualizing keywords. I have 500+ URLs to review for the company I work for and could use some guidance; yes, I know I should test, but testing is problematical to the extreme; I look to the collective/accumulated wisdom of the MOZVerse for help. Thanks.1 -
Is it better to redirect a url or set up a landing page for a new site?
Hi, One of our clients has got a new website but is still getting quite a lot of traffic to her old site which has a page authority of 30 on the home page and has about 20 external backlinks. It's on a different hosting package so a different C block but I was wondering if anyone could advise if it would be better to simply redirect this page to the new site or set up a landing page on this domain simply saying "Site has moved, you can now find us here..." sort of idea. Any advice would be much appreciated Thanks
Web Design | | Will_Craig0 -
Need help in website URL Structure
I have been working on a brand new website currently it is live but I have disallow Googlebots temporarily as I dint want any negative impact. The business of the site is to generate leads , they install and sell Stairlifts and used Stairlifts. There are two main categories New Stairlifts and Reconditioned Stairlifts Currently the URL for new Stairlifts is : http://willowstairlifts.co.uk/stairlifts/ and for Reconditioned Stairlifts is: http://willowstairlifts.co.uk/reconditioned-stairlifts/ My concerns are that the word Stairlifts is mentioned twice in the urls so is it going to have a negative impact or panda penalty? I am thinking of changing them to http://willowstairlifts.co.uk/new/ and the product pages to display as http://willowstairlifts.co.uk/new/brooks/ Currently its http://willowstairlifts.co.uk/stairlifts/brooks/ Same with reconditioned Stairlifts I like to change it to : http://willowstairlifts.co.uk/reconditioned Also its product pages to http://willowstairlifts.co.uk/reconditioned/brooks/ As currently its http://willowstairlifts.co.uk/reconditioned-stairlifts/brooks/ Thanks
Web Design | | conversiontactics0 -
URL question for SEO...
I'm thinking of creating a new url off an existing url and was wondering if there would be any impact. For example I have the URL www.baseball.com and rather than secure a new url for a new product launch such as www.newbaseballproduct.com I want to do newproduct.baseball.com Will this hurt my SEO rankings for this new site? Basically wanting to figure out if this will hurt me or not? Should I get a new url or re-utilize an existing URL... really for a landing page/micro site, etc.,
Web Design | | gritacco0