Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Do you know any tool(s) to check if Google can crawl a URL?

Web Design

1318

esiow2013 last edited by

Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.

In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.

Thanks in advance!
1 Reply Last reply
Reply Quote 0
danwebman last edited by

I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
1 Reply Last reply
Reply Quote 1
masdigitalmedia last edited by

You could consider this online tool: http://web-sniffer.net
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Competitor's new site ranking with out much keywords - How?

Hi all, One of our competitors have recently redesigned their website with new content. Now I can see much less keywords in the content. And page title also changed away from keywords. Still this is ranking at good position. How? Previously they used to have much landing pages with related keywords which some of them are missing now. Still I wonder why this website is ranking high? Thanks
Web Design | | vtmoz

0
Fetch data for users with ajax but show it without ajax for Google

Hi, We have a thematic footer which shows similar pages links relevant to the search criteria made on a page. We want to fetch those footer similar links through ajax when users search on site but the links will be shown without using ajax when Google fetches those pages. We want to do this to improve our page load time. The links content & count will be exactly same in both cases whether Google fetches the search pages or user fetches those pages. Will this be treated as negative by Google, Can this have any negative affect on our rankings or traffic. Regards,
Web Design | | vivekrathore

0
New To SEO Management, I just want to double check that my idea will work.

I am new to SEO management. I had a 3 month SEO copy writing internship and a 5 month SEO temp job. In both I mostly wrote copy, but I've been teaching myself SEO on the side, I became Google certified. I ended up getting a telemarketing job and somehow the conversation of SEO came up and I winded up managing their SEO for 12 dollars an hour. They say that every lead generated from the website that turns into a sale will be worth 10 dollars and if and when the sales exceed my paycheck I will starting making commission so long as it stays above my hourly. SEO is very fun and this is like my dream job. They are leaving the planning 100% up to me and I want to make sure that what I am doing will work. My plan is as follows: Part 1: Page Authority via backlinks and social media We are health care brokers and my boss, the owner has a lot of contact. He is talking with large unions like, "The Teamsters," and large company retirment groups like, "Blue flame," which is apparently in some way connected to DTE or GE. Long story short, I am trying to get him to convince them to give us a back link to our main page. He also has a ton of clients that own companies. This is good because they may be persuaded to give us backlinks too. In addition, the tech guy thinks he can implement something where we can get a google +1, facebooks likes/shares, twitter likes and shares and pintrest pin it's that would be a part of an email that we send to people within the list of 12,000 clients. From what I can see, from the client base and the people we are working with we should be able to raise the page authority substantially despite the fact that the site is only a few months old and is not yet out of the sand box. I have been slowly picking off each error with SEO MOZ's website crawling. Part 2: Making a Insurance Jargon Dictionary Guide For The Tri-purpose of gathering traffic, proving our professionalism and helping people understand semi-complex insurance jargon. I could build these 2-3 keywords would be addressed per page and they would be defined in a way to help people looking for terms understand them, while simultaneously netting a strong keyword density and a strong page. I think as far as I can tell there are no issues. Part 3: The dictionary pages will pull in new traffic and the home page will receive links and distribute link juice to the sub-pages. This subpages will guide traffic back to the main page with no-follow links to direct people from the unique termed landing pages to the home page for insurance processing. As far as I can tell my logic is solid and on paper this should work. Am I missing anything (like key details, flaws in my plan)?
Web Design | | Tediscool

0
Wordpress themes causing google penalty(need experts to settle a debate)

Hi, I have been having a disagreement with another online marketing company. We are both promoting the same product under a different brand name but we ended up using the same theme to build our WordPress sites off of but in no way is the content the same. They are telling me that using the same theme in the same industry will cause a Google penalty. I do not believe this and do not see this causing a problem. The sites are relatively new so there is no proof of traffic dropping or penalties as of yet. What is everyone's professional opinion on this? Can a WordPress theme cause duplicate content penalty? If so would that not mean that anyone using themes will have some sort of penalty?
Web Design | | impact89

1
How we can check whether website design is good for SEO or not?

Is there any tool available to check website design whether it is good for SEO or not?
Web Design | | ross254sidney

0
301 forwarding during site migration problem - several url versions of the same page....

Hello, I'm migrating from an old site to a new site, and 301 forwarding many of the pages... My key problem is this I'm seeing www.website.com/ indexed in SE and www.website.com/default.aspx in showing as URL when I'm on homepage - should I simply 301 forward both of these? Then for several internal pages there are 2/3 versions of each page indexed. Canonicalization issues. Again, I'm wondering whether I should 301 forward each URL even if there are several different indexed URLs for the same page? Your advice will be welcome! Thanks in advance - Luke
Web Design | | McTaggart

0
Best way to add Google Analytics to a Joomla!

I am having difficulties installing (and getting it to work) google analytics to Joomla! 2.5.8 site. I have it working on some sites but then not on others even though I have installed it the same. Is there a recommended or easy way to do this that works?
Web Design | | Atlanta-SMO

1
I know frames aren't good, but are they bad?

About 3/4 of my website includes frames from the Amazon aStore, but the pages also have at least 500 words of content on them each. I understand that spiders aren't too good with frames but will search engines punish my site for having them or just disregard them? Thanks in advance.
Web Design | | Max_powers

0