100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google webmaster reports non-existent links between syndicated sites
We have run into an issue with linking that we are completely puzzled by. We syndicate our content to various clients, taking care to ensure that we have followed all the best practices that Google recommends for syndicating content. But recently, we noticed Google Webmaster report links from ClientA to ClientB, and we cannot figure out why it thinks that way. We have never created, and we have never found the links that Google Webmaster claims are there. It is important for us to keep our clients isolated. Has anyone seen such behavior? Any ideas/pointers/hunches would be very much appreciated. Happy to provide more information. We even asked on the Google Webmaster Forum (https://productforums.google.com/forum/#!topic/webmasters/QkGF7-HZHTY;context-place=forum/webmasters), but thought this might be a better place to get expert advice. Thanks!
Intermediate & Advanced SEO | | prakash.sikchi0 -
Alternative Link Detox tools?
My company is conducting a link detox for a client, and it seems like every tool we utilize is giving us a different answer on how many links we actually have. the numbers range anywhere from 4,000 to 200,000. Does anyone have any suggestions as to what tools will give us an accurate count, and will also email the webmasters on your behalf requesting the links removal? We are trying to have this process be as automated as possible to save time on our end.
Intermediate & Advanced SEO | | lightwurx0 -
What To Do With Too Many Links?
We have four pages that have over 100 links (danger, danger from what I gather), but they're not spammy footer links. They are FAQ videos for our four main areas of practice. Does that make a difference? If not, should I just take half the questions on each page and make four additional pages? That strikes me as a worse UX, but I don't want to get penalized either. Thanks, Ruben
Intermediate & Advanced SEO | | KempRugeLawGroup0 -
Link Reclimation & Redirects
Hello, I'm in the middle of a link reclamation project wherein we're identifying broken links, links pointing to dupe content etc. I found a forgotten co-brand which is effectively dupe content across 8 sub-domains, some of which have a significant number of links (200+ linking domains | 2k+ in-bound links). Question for the group is what's the optimal redirect option? Option 1: set 301 and maintain 1:1 URL mapping will pass all equity to applicable PLPs and theoretically improve rank for related keyword(s). requires a bit more configuration time and will likely have small effect on rank given links are widely distributed across URLs. Option 2: set 301 to redirect all requests to the associated sub-domain e.g. foo.mybrand.cobrand.com/page1.html and foo.mybrand.cobrand.com/page2 both redirect to foo.mybrand.com/ will accumulate all equity at the sub-domain level which theoretically will be roughly distributed throughout underlying pages and will limit risk of penalty to that sub-domain. Option 3: set 301 to redirect all requests to our homepage. easiest to configure & maintain, will accumulate the maximum equity on a priority page which should positively affect domain authority. run risk of being penalized for accumulating links en mass, risk penalty for spammy links on our primary sub-domain www, won't pass keyword specific equity to applicable pages. To be clear, I've done an initial scrub of anchor text and there were no signs of spam. I'm leaning towards #3, but interested in others perspectives. Cheers,
Intermediate & Advanced SEO | | PCampolo
Stefan0 -
Link Building Question
Hey Moz'ers, I have created several blogs on different domains for the purpose of writing good content articles that contain 2-3 links per article that go back to my website. It has been up for about 3-4 weeks. I am not seeing my results/links showing up in OSE, is this because it still needs more time or is there something else I could be advised to look into? In theory these blogs will only contain 2-3 links from each domain to the site. I was also going to make sure the anchor text per link is different (keyword, brand name, random anchor like click here). Side note: How does this system sound as part of one small aspect to link building? red flags? Thanks for all the responses and advice.
Intermediate & Advanced SEO | | MonsterWeb280 -
Why traffic to my link has dropped suddenly?
Hi I would like to know why the traffic for the website link http://theindustrymeasure.com/2010/07/15/rediffmail-login has dropped suddenly on google.I used to get around 5000 page views on this page and then suddenly dropped to 15-20 . I still get good traffic from yahoo (around 500). Just before the drop I noticed that I started to get spammy trackbacks from Many questionable sources. I have not approved any of these trackbacks. The trackbacks are regular frequency of. 1 per day. is there any action which I can take to ensure that I get back my traffic. Traffic to other links are fine , only this page seems to have dropped off ever since the spam attack. As per seomoz tool I have a grade a for keyword rediffmail
Intermediate & Advanced SEO | | ShoutOut0 -
Are duplicate links on same page alright?
If I have a homepage with category links, is it alright for those category links to appear in the footer as well, or should you never have duplicate links on one page? Can you please give a reason why as well? Thanks!
Intermediate & Advanced SEO | | dkamen0 -
Dark Matter Links
From 2007 - 2004 I worked for Sprint in several positions with my last one being a Corporate Account Manager for fortune 1000 customers. In 2004 I left Sprint after the Nextel merger and created an eCommerce site called thesprintstore.net as a Sprint Nextel preferred partner. I used my inner working knowledge of Sprint to my wonderful advantage and began making 3x my original salary. My desire for more business turned to greed and I began leaking information that consumers loved i.e. phone release dates, price points, warehouse stock levels and tricks of the trade. This garnered me thousands of links from big sites (had no idea at the time) and eventually my site was issued a Cease and Desist order from Sprint's Corporate Headquarters. I recently realized one evening that I had a GEM of a domain with powerful backlinks that I could redirect to my current site TECHeGO.com [staff removed hyperlink]. (Some of the back links are from Engaget, Engaget Mobile, Rimmarkable and even one from Sprint.) The redirection has been in place for months now and I have confirmed that all that sweet Link Nectar is flowing through! I have found it interesting, however, that my back link and referral domain count have never increased leading me to believe that in doing a 301 Redirect existing links become what can only be described as 'Dark Matter Links' i.e. the links are there, simply invisible. Dark Matter Definition: dark matter is matter that is inferred to exist from gravitational effects on visible matter and background radiation, but is undetectable by emitted or scatteredelectromagnetic radiation. Dark Matter Links: dark matter links are visible links that have passed through a 301 redirect which are now inferred to exist but are no longer visible by crawlers? Is there a better definition that could be applied to the term 'Dark Matter Links'?
Intermediate & Advanced SEO | | TECHeGO1