Do you know any tool(s) to check if Google can crawl a URL?
-
Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.
In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.
Thanks in advance!
-
I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
-
You could consider this online tool: http://web-sniffer.net
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google analytic's API information
I have multiple websites and instead of having to log in to each google analytics I want to create a dashboard inside my MIS that has the audience overview graph, is there any way to use API to do this? Is there a way to
Web Design | | BobAnderson0 -
Curious why site isn't ranking, rather seems like being penalized for duplicate content but no issues via Google Webmaster...
So we have a site ThePowerBoard.com and it has some pretty impressive links pointing back to it. It is obviously optimized for the keyword "Powerboard", but in no way is it even in the top 10 pages of Google ranking. If you site:thepowerboard.com the site, and/or Google just the URL thepowerboard.com you will see that it populates in the search results. However if you quote search just the title of the home page, you will see oddly that the domain doesn't show up rather at the bottom of the results you will see where Google places "In order to show you the most relevant results, we have omitted some entries very similar to the 7 already displayed". If you click on the link below that, then the site shows up toward the bottom of those results. Is this the case of duplicate content? Also from the developer that built the site said the following: "The domain name is www.thepowerboard.com and it is on a shared server in a folder named thehoverboard.com. This has caused issues trying to ssh into the server which forces us to ssh into it via it’s ip address rather than by domain name. So I think it may also be causing your search bot indexing problem. Again, I am only speculating at this point. The folder name difference is the only thing different between this site and any other site that we have set up." (Would this be the culprit? Looking for some expert advice as it makes no sense to us why this domain isn't ranking?
Web Design | | izepper0 -
301 Redirect Issue for URL with # and !
Hi All, We had a WIX website and now moved to Wordpress. I m having issue while doing redirecting from old URL to new URL. Example: Old Url: http://www.firsttraffic.com.au/#!traffic-management/ccfn New Url: http://www.firsttraffic.com.au/our-services/traffic-management/ I tried different wordpress plugin but nothing works. I m thinking its due to the # . But How can I to redirection for URL like this . Thanks
Web Design | | emarketexperts0 -
Magento URL Structure
I'm about to migrate to Magento and wanted to ask about the optimal URL structure for the following page: Knowledge Centre |-Videos |-Customer Testimonials |-Customer X Would it be better to use: Domain/knowledge-centre/videos/customer-testimonials/customer-x or Domain/customer-x Thanks in advance for any replies.
Web Design | | ssoneil0 -
Changing Links that Show Up when I Google Brand (Site) Name
Hi SEOmoz Community, A quick question for you all. I've added an attachment for reference. When I google my brand name, say for example, Applied StemCell, I see six links as well below the description. Oddly though, these links seem to be chosen at random, or at least I'm not sure how Google decides on them. When I click on one of the links that is the company's name, Applied StemCell it brings me to a PDF document! Is there any way I can choose which ones to display there? Thanks! OF2oVVN.png
Web Design | | swzhai0 -
Google search issue with exact domain
We had a site from Feb-2011 to Nov-2011 at the domain amcoexterminating.com. The site was pure HTML/CSS and the daily unique visitors steadily increased over that time. So all was fine. We then moved the site to a CMS (Joomla) on Dec. 6th. From that day forward, the daily visitors went into the tank. Before the move, if you typed "amcoexterminating.com" or "amco exterminating" into Google search, the site would be the first result (as you'd expect since those are the words that make up the actua domain). But we tried this yesterday and the site did not come up at all. NOT GOOD. It would work in Yahoo or Bing, but not in Google. So obviously, the problem with Google search directly affected the daily visitors. We just checked Webmaster tools yesterday (yes, this should have been done sooner, lesson learned) and it said "Site has severe health issues - Important page blocked by robots.txt". It listed the "important" page URL and it was just a link to an image. Regardless, I wiped out the Joomla created robots.txt file and added a new one and made it just say... User-agent: *Allow: / About 14 hours later, after the new robots.txt file was recognized by Google, the "severe health" message went away. However if I search in Google for "amcoexterminating.com", it still doesn't show up and the client is concerned (as they should be). Do you think the search engines just need more time to refresh? If so, once it refreshes, should the site show up first again right away? Or is it possible the robots.txt file had nothing to do with the issue? If so, what other things could I check into that might cause Google search to not find a site even if you search for exact domain name? Please share any and all things I should look into as I need to get this site showing in Google search again (as it was before moving to the CMS). Thanks!
Web Design | | MarathonMS0 -
Can i do this? Will Google penalize me?
I have a page for a Criminal Defense Attorney and i set up a list of the type of criminal charges he is certified to deal with. I wanted to use title tags and put the Keyword "Miami Criminal Defense Attorney" & "Miami Traffic Defense Lawyer"... My question is will Google penalize me for plugging the same Key words over and over on the title tag for each ?? CHECK THE IMAGE to see what I'm talking about... thanks guys. x97dl
Web Design | | marig0 -
Tools For Building Architectures ?
Hi Everyone, first question on here: Does anyone have any reccomendations for tools that they use to build site architectures ? It's something I would like to improve on. Thanks,
Web Design | | marcelo-2753980