Welcome to the Q&A Forum

Thanks @Ryan

I was referring https://www.google.com/#q=site:domain.com using site:domain.com , I know it doesnt give me all urls but isnt what we care about, at least matching numbers I mean?

Thanks Ryan and everyone else, amazing answers So here is my understanding:

For Internal 301:

1. Use Xenu or Screaming to scan url and create the list. I hope we can get a clean report from screaming.

For External:

1. Use Ose to find all backlinks and save them on excel

2. Use AHREFs to find all backlinks and save them

3. Use Majestic to find all backlinks and save them

4. Combine all url and remove duplicate(I guess manually we gotta do that)

Questions

a) How do we find out which one is 301 redirected beside checking each of them?

b) for backlinks we should check what google and bing crawled?

Hi Ryan,

Thanks for your quick response.

Yes we heard about the screaming frog but never used it, will give it a try.

We do not have access to google/bing webmaster tool or analytics. We are more like a third party company working on this project.

Any other ideas?

This is what we are thinking:

• Get all URL of the xyz.com/products using XENU software.
• Search  those URL on google (site;xyz.com url ) to find out if they are crawled by google, do the same on bing (as currently google shows 4k URL and bing 11k )
• Use opensiteexplorer (301 redirect ) and using (internal external) to get the desired result.

Is this the right approach? If not, what is the best way to find the correct result?

All suggestions are welcome.

my domain name is www.deoveritas.com and about us page was published 5 years back

I tried site:http://www.deoveritas.com it does not show homepage or inner pages links, full search result is with blog section links

the website url is correct, I tired majestic seo and OSE it is showing data

www.deoveritas.com

My site is www.deoveritas.com it is in magento framework and it has a blog section in wordpress. When I enter Site:www.deoveroitas.com in google it shows all blog links in search result. The homepage and other innerpages are not getting displayed in search results at all. I even tried searching for www.deoveritas.com/about-us and it displays blogs in result.

Checked Google webmaster fetch as google and it was index and successful.

Can you please help me with this. Is my site de-indexed or banned by Google? the same issue is on Bing and Yahoo search engines too. Please help

Thank you.

We found msnbot is doing lots of request at same time to one URL, even considering we have caching,  it triggers many requests at same time so caching does not help at the moment:

For sure we can use mutex to make sure URL waits for cache to generate, but we are looking for solution for MSN boot.

123.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET //Fun-Stuff HTTP/1.1" 200 0 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

1.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET //Type-of-Resource/Fun-Stuff HTTP/1.1" 200 0 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

1.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET /Browse//Fun-Stuff HTTP/1.1" 200 6708 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

We found the following solution: http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx

Bing offers webmasters the ability to slow down the crawl rate to accommodate web server load issues.

User-Agent: *

Crawl-Delay: 10

Need to know if it’s safe to apply that.  OR any other advices.

PS: MSNBot gets so bad at times that it could trigger a DOS attack – alone! (http://www.semwisdom.com/blog/msnbot-stupid-plain-evil#axzz2EqmJM3er).

Google webmaster tool is showing DNS error and that is leading to many server error (502,500) almost 50+ in every crawl. Recently Google crawled one of our sub domains that we did not want google to crawl. We blocked it via Robots.txt and also removed all the URL's and since then we are having this issue. Any suggestions how to fix this DNS error? Thanks in advance.

We launched our new site 2 days ago , since site was down for 12 hours for maintenance, we saw google webmaster tool shows this error . Since then google hasnt crawled, its been 36 hours. Do we need to do anyting? We have close to a million page google crawled before and I am wondering if this will effect anything.

Hi, i run a small company and provide SEO services to clients. is it worthy to buy paid directory submission?

Our client ecommerce site load from 3 different servers using load balancing.

abc.com : IP: 222.222.222

Abc.com: IP: 111.111.111

For testing purpose 111.111.111 also point to beta.abc.com

Now google crawling site beta.abc.com

If we block beta.abc.com using robots.txt it will block google bot also , since beta.abc.com is really abc.com

I know its confusing but I been trying to figure out. Ofcourse I can ask my dev to remove beta.abc.com make a seperate code and block it using .htaccess

thanks I will apply it

https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=site%3A50.97.XXX.XXX

About 7,050 results (0.24 seconds)

when we do list by domain we get :

About 10,400,000 results (0.29 seconds)

is it ok? would google smart enough to count IP address not as duplicate content?

Hello folks,

I wanted to implement hcard to my client website, so i wanted to know the procedure to go about.

Thanks in Advance!!

Oh yes we have CloudFare on our servers. I have changed it, now its working fine.

Thanks guys!!

Hello guys,

I need some help.

I can see the following javascript in the header which is not added by me. What code is it ? Something related to Google Analytics ?

<script type="text/javascript">
//
script>

Thanks in advance