100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Breadcrumbs and internal links
Hello, I use to move up my site structure with links in content. I have now installed breadcrumbs, is it is useful to still keep the links in content or isn't there a need to duplicate those links ? and are the breadcrumbs links enough. Thank you,
Intermediate & Advanced SEO | | seoanalytics1 -
Removing Bad Links
Hi all, I am in the process of conducting a Link Audit and I am faced with quite a lot of seemingly poor quality examples, such as; http://gotogetaways.com/tag/cunard/ http://jobhiringlocalandabroad.blogspot.com/p/job-hiring-for-cruise-liner-orchestra.html http://lumukixu.xlx.pl/p-o-cruises-aurora.php To me these should be removed \ disavowed but I am getting a little resistence from stakeholders regarding the amount of links I am seeking to rid ourselves of - all are of a similar quality to my examples above... Just so that I know that I am not being 'over eager' with my audit, I welcome your opinions Thanks Andy
Intermediate & Advanced SEO | | TomKing0 -
Is a scholarship seen as buying links?
We work in the student loan debt industry, so providing a scholarship for our viewers makes total sense for our business. Would links to this page be seen by Google as buying links? Does this fall anywhere into gray-hat tactics? My gut tells me no because helping people with student debt is what we do. The scholarship really synergizes with our business model, but who knows..?
Intermediate & Advanced SEO | | DemiGR0 -
Multiple Internal links to same destinations
My company is redoing our homepage and there will be 4 links to our main play pages (5 games). 2 in the menu and 2 within the content. I was thinking I should no follow one of the links on the homepage + 1 in the menu so that we don't have link dilution from having multiple internal links to the same destination within 1 page. Does this make sense? Any downside of this or suggestions of a solution that may be more effective? Thanks!
Intermediate & Advanced SEO | | theLotter0 -
First Link on Page Still Only Link on Page?
Bruce Clay and others did some research and found that the first link on the page is the most important and what is accredited as the link. Any other links on the page mean nothing. Is this still true? And in that case, on an ecommerce site with category links in the top navigation (which is high on the code), is it not useful to link to categories in the content of the page? Because the category is already linked to on that page. Thank you, Tyler
Intermediate & Advanced SEO | | tylerfraser0 -
Disavowing Links for Subcategory of Site
Has anyone tried using Google's Disavow tool with only a specific subcategory of their site? We're an ecommerce company and our site took a small hit with this recent Penguin update. We're certain previous linkbuilding efforts are the cause. But we'd like to try the Disavow tool with 1 subcategory to start, see if our rankings for that category improve (we used to be top 3, now ~12 or 13), and if so then roll it out through the rest of the site. Looking for input from others on if they have any experience with this or if it'd be better to just go for the whole thing at once. Thanks.
Intermediate & Advanced SEO | | Kingof50 -
How many links home on a page?
We are planning on a mega menu which will have around 300 links and a mega slider which will have around 175 links if our developer has their way. In all I could be looking at over 500 links from the home page. The Mega Menu will flatten the site link structure out but I am worried this slider on the home page which is our 4th most visited page behind our 3 core category pages. What are your thoughts?
Intermediate & Advanced SEO | | robertrRSwalters0 -
Advertising links hurt SEO?
I'm working with a publisher who said that having DFA links on his site will hurt his SEO. He is taking my link and pointing it back to his site and then to mine. Does that sound right to you?
Intermediate & Advanced SEO | | GFTMarketer0