Welcome to the Q&A Forum

Thanks @Ryan

I was referring https://www.google.com/#q=site:domain.com using site:domain.com , I know it doesnt give me all urls but isnt what we care about, at least matching numbers I mean?

Thanks Ryan and everyone else, amazing answers So here is my understanding:

For Internal 301:

1. Use Xenu or Screaming to scan url and create the list. I hope we can get a clean report from screaming.

For External:

1. Use Ose to find all backlinks and save them on excel

2. Use AHREFs to find all backlinks and save them

3. Use Majestic to find all backlinks and save them

4. Combine all url and remove duplicate(I guess manually we gotta do that)

Questions

a) How do we find out which one is 301 redirected beside checking each of them?

b) for backlinks we should check what google and bing crawled?

Hi Ryan,

Thanks for your quick response.

Yes we heard about the screaming frog but never used it, will give it a try.

We do not have access to google/bing webmaster tool or analytics. We are more like a third party company working on this project.

Any other ideas?

This is what we are thinking:

• Get all URL of the xyz.com/products using XENU software.
• Search  those URL on google (site;xyz.com url ) to find out if they are crawled by google, do the same on bing (as currently google shows 4k URL and bing 11k )
• Use opensiteexplorer (301 redirect ) and using (internal external) to get the desired result.

Is this the right approach? If not, what is the best way to find the correct result?

All suggestions are welcome.

my domain name is www.deoveritas.com and about us page was published 5 years back

I tried site:http://www.deoveritas.com it does not show homepage or inner pages links, full search result is with blog section links

the website url is correct, I tired majestic seo and OSE it is showing data

www.deoveritas.com

My site is www.deoveritas.com it is in magento framework and it has a blog section in wordpress. When I enter Site:www.deoveroitas.com in google it shows all blog links in search result. The homepage and other innerpages are not getting displayed in search results at all. I even tried searching for www.deoveritas.com/about-us and it displays blogs in result.

Checked Google webmaster fetch as google and it was index and successful.

Can you please help me with this. Is my site de-indexed or banned by Google? the same issue is on Bing and Yahoo search engines too. Please help

Thank you.

We found msnbot is doing lots of request at same time to one URL, even considering we have caching,  it triggers many requests at same time so caching does not help at the moment:

For sure we can use mutex to make sure URL waits for cache to generate, but we are looking for solution for MSN boot.

123.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET //Fun-Stuff HTTP/1.1" 200 0 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

1.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET //Type-of-Resource/Fun-Stuff HTTP/1.1" 200 0 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

1.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET /Browse//Fun-Stuff HTTP/1.1" 200 6708 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

We found the following solution: http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx

Bing offers webmasters the ability to slow down the crawl rate to accommodate web server load issues.

User-Agent: *

Crawl-Delay: 10

Need to know if it’s safe to apply that.  OR any other advices.

PS: MSNBot gets so bad at times that it could trigger a DOS attack – alone! (http://www.semwisdom.com/blog/msnbot-stupid-plain-evil#axzz2EqmJM3er).

Google webmaster tool is showing DNS error and that is leading to many server error (502,500) almost 50+ in every crawl. Recently Google crawled one of our sub domains that we did not want google to crawl. We blocked it via Robots.txt and also removed all the URL's and since then we are having this issue. Any suggestions how to fix this DNS error? Thanks in advance.