Do you know any tool(s) to check if Google can crawl a URL?
-
Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.
In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.
Thanks in advance!
-
I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
-
You could consider this online tool: http://web-sniffer.net
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
May open the site's homepage...?
In mobile search results, many of my inner pages show the title tag, but the link is removed. And instead of showing the meta description, there is the following: May open the site's homepage
Web Design | | Dino64
Try anyway | Learn more Both "Try Anyway" and "Learn more" are links. Try anyway used to link to the homepage, but after we removed a redirect plugin and got rid of all the html mobile pages (now we're just using the Wordpress responsive layout), the link is now going to the correct page. So, it in fact, does NOT "open the site's homepage" any longer. But, we're still seeing this on mobile searches for our inner pages. It's been almost 3 weeks now. Any ideas on how to get this back to normal?0 -
How to know if a wordpress theme is coded correctly for Seo
Hi, So I am curious if there is a tool to see if a site is coed properly for Google? I am running Avada, a standalone theme, yet I am also using a cache plugin. But when I search my code, its all like on one huge line. So I am curious if there is a way to verify or check if a theme is coded correctly? Thank you
Web Design | | Berner1 -
Image URL's and naming
We're re-platforming on Magento and wondering about our images. 1. Should I be concerned about 301 redirects for my images. 2. Is there a "best practice" path for images? or is just the name important? Right now, all our images are in /meta/images/sm or /lg or /xlg. Since we're re-platforming, we're wondering if we should change the urls. But, I'm assuming this would require all of them to have 301 redirects and with all the other redirects, I'm not sure this is really feasible. thanks for any suggestions on this.
Web Design | | centralvacuumstores0 -
Responsive design and Google analytics mobile tracking codes?
Hi all, We are currently rebuilding a site using responsive design, however i have just had a thought. On another site where we have a mobile site under a sub-directory we utilise mobile tracking codes as we found that this was far more accurate for recording visits. On a responsive design site evidently all pages, desktop and mobile, will be under the same URL, yet the content will adjust to the screen size of the device. Should we also change the tracking code to be mobile code on the lower resolutions or would the same code be sufficient?
Web Design | | Sarbs0 -
301 Redirect ! Joomla Pages, Already ranking. ( just wanted to change the url
hey guys hope everyone had a new year. I am ranking for a page on my site that i want to ( not specifically move ), but just change the url name: It is too long i think and i want to move it from one portion of my architecture to another menu. I have never physically done a 301 redirect myself, always had someone do it for me. I wanted some pointers. Since it is a fairly new site 4 months old! What are my options. Do i need to 301 redirect the page, if i am changing the Structure and AI of my site, or can i just change the url as is and it will still get ranked? How do i keep that url put delete the page and redirect it ? Sorry its very simple but i wanted to get the communities help to continue on ! Best Wishes HAmpig
Web Design | | BizDetox0 -
Can anyone rcommend a UK Hosting company
As some of you may have seen in an earlier post, i have had problems with the speed of my site. After good advice i got an expert on board who done tests and found that my server (hosting company) was taking a long time to answer, on some occasions it was taking seven seconds. I have tried to get the hosting company to listen and sort the problem out but they are not interested and keep trying to sell me other things to get the site faster, i am already on a dedicated server. So now I am looking for a UK hosting company who offer good service and would be grateful if anyone could recommend some on here so i can speak to them, as i want to get my sites moves a.s.a.p many thanks
Web Design | | ClaireH-1848860 -
Development site accidentally crawled - Will this cause problems?
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought. Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain. I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain. We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts. Thank You!
Web Design | | AndrewY0 -
Custom URL's with Bigcommerce Issue (Is it worth it?)
We're building out a store in Bigcommerce, who for all intensive purposes is perfect for SEO besides the fact that you can not change the URL's to be custom. My question is, does this kill the SEO value of bigcommerce, despite everything else being great? So for example the URL's for a category page would be something like this www.mysite.com/categories/keyword and the product URL's are pulled in by product name, so product URL's could be something like www.mysite.com/products/Product-Description-Long-223.html (notice the words will be capitalized and their is no way to remove the trailing .html) I could go with Interspire (the liscenced version of Bigcommerce) or Magento so I can custom edit this stuff. But then its a lot more work for my employee's on the buildout.
Web Design | | iAnalyst.com0