Welcome to the Q&A Forum

Hi Guys,

Found this question.

Do you know of any other tools? I'm already using screaming frog but when spidering a 10.000+ page e-commerce site screaming frog hangs and the site gets very slow

Any suggestions?

Hi Dan,

Thanks for the answer!

Indexation is already back to 42.000 so slowly going back to normal

And thanks for the last tip, that's totally right. I just discovered that several pages had duplicate url's generated so by continually monitoring we'll fix it !

Hi Chris,

Thanks for your answer.

I'm either using a 301 or noindex, not both of course.

Still have to check the server logs, thanks for that!

Another weird thing. While the old url is still in the index, when i check the cache date it's a week old. That's what i don't get. Cache date is a week old but Google still has the old url in the index.

Hi Guys,

I've been working on a E-commerce site for a while now. Let me sum it up :

• February new site is launched
• Due to lack of resources we started 301's of old url's in March
• Added rel=canonical end of May because of huge index numbers (developers forgot!!)
• Added noindex and robots.txt on at least 1000 urls.
• Index numbers went down from 105.000 tot 55.000 for now, see screenshot (actual number in sitemap is 13.000)

Now when i do site:domain.com there are still old url's in the index while there is a 301 on the url since March!

I know this can take a while but I wonder how I can speed this up or am doing something wrong. Hope anyone can help because I simply don't know how the old url's can still be in the index.

4cArHPH.png

Hi,

A few weeks later now and index is now on 63.000 url's so that's a good thing.

Another weird thing is the following.

There's a (old) url still in the index. When i visit it redirects me to the new url, which is good. Cache date is 2 weeks ago but Google still shows the old url.

How is this possible? The 301 redirect is already in place since April 2013.

It is ridiculous that they charge this kind of money for just installing WP.

I think you can do your client a favor by telling him it usually doesn't take that amount of time and indeed like Martijn says that they need to specify their hours.

I'm in the same situation by the way. Doing some SEO consulting for a client and the webdevelopment company is charging 125 euro's an hour (which is not specifically bad) but counting a lot of hours for small tasks.

What we did.. I made a list of things that needed to be done and we visited them (physically). So I sat next to the developer while he was changing things and suddenly tasks that took 2 hours before were done in 15 minutes

If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.

But it has to be online somewhere so the software can reach it.

Did this answer your question?

Hi allen Jarosz!

Thanks for your reply

I've actually done all the things you said in the last few weeks. Site is totally indexed but the main problem is that are over 85.000 url's indexed but the site only exists of 13.000 urls.
So the main question is wether i can speed things up in one way or another to get those 70.000 url's deindexed.

Are any options besides noindex, robots.txt and removing some url's ? Because now it's just waiting.

It looks like we are going the right way when you check the image.

gaHmQ54.png

Hi Guys,

Refering back to my original post I would first like to thank you guys for all the advice.

We implemented canonical url's all over the site and noindexed some url's with robots.txt and the site already went from 100.000+ url's indexed to 87.000 urls indexed in GWT.

My question: Is there way to speed this up? 
I do know about the way to remove url's from index (with noindex of robots.txt condition) but this is a very intensive way to do so.

I was hoping you guys maybe have a solution for this..

Allright guys, thanks alot for the answers.

Gonna try some things out coming monday.

Canonical url's and pagination (rel=prev) will work i guess.

The hard part is, i'm working on this site with a development company that tells me they can url redirect all the 404's to the homepage while they must be redirected either to other products or category pages.

So only solution is that i have to do that by hand, one by one via a tool they build. But it's a hell of a job!

@ Andy , I checked it and it actually says :

Total indexed : 98.000
Ever crawled: 929.762

And when i check the questionmark at total indexed it says:
Total number of url's added to Google index.

Thanks again for your answers

It is ridiculous that they charge this kind of money for just installing WP.

I think you can do your client a favor by telling him it usually doesn't take that amount of time and indeed like Martijn says that they need to specify their hours.

I'm in the same situation by the way. Doing some SEO consulting for a client and the webdevelopment company is charging 125 euro's an hour (which is not specifically bad) but counting a lot of hours for small tasks.

What we did.. I made a list of things that needed to be done and we visited them (physically). So I sat next to the developer while he was changing things and suddenly tasks that took 2 hours before were done in 15 minutes

If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.

But it has to be online somewhere so the software can reach it.

Did this answer your question?