100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Tumblr links
I have several Tumblr blogs. Created when Tumblr links were worth more, and now primarily for my amusement. But, I'd like to get whatever link juice I can out of them. I thought only the footer links were do follow, but when I check Moz it's showing all links as do follow. Any idea which is true?
Intermediate & Advanced SEO | | julie-getonthemap1 -
How Do You Do Link Building??
I am starting to use the Moz pro tools like optimizing on page SEO for keywords and looking for opportunities. I know link building is a huge part for getting rankings on keywords in google search. Where do I start and how do I do the link building process for specific keywords I can rank for?? Thank you in advance for your help.
Intermediate & Advanced SEO | | wickerparadise1 -
How to tell the date a link was created
Does anybody know of a website that can let you know when an external link was created to a site? Or any other way of finding this info out. Thanks
Intermediate & Advanced SEO | | RobSchofield0 -
Block Level Link Juice
I need a better understanding of how links in different parts of the page pass juice. Much has been written about how footer links pass less juice than other parts of the page. The question I have is that if a page has a hypothetical 1000 points of Link Juice and can pass on +/-800 points via links, and I have 1 and only 1 link in the footer to another page, does it pass the full 800 points? Or... since footers only pass a small fraction of link juice, it passes lets say 80 points, and the other 720 points stays locked up on the page. This question is a hypothetical - I'm just trying to understand relationships. I don't know if I've explained the question too well, but if someone could answer i it, or point me in the right direction, I would appreciate it.
Intermediate & Advanced SEO | | CsmBill0 -
Internal Javascript Links
Hi, We have a client who has internal links pointing to some relatively new pages that we asked them to implement. The problem is that instead of using standard HTML links, their developers have used javascript - e.g. javascript:GoTo... The new pages have links from the homepage (among others) and have been live for about 3-4 weeks now - yet are still to be indexed by Google, Bing & Yahoo. Is it possibe that Javascript links are making them difficult to be found? Thanks in advance for any tips.
Intermediate & Advanced SEO | | jasarrow0 -
Removed Site-wide links
Hi there, I have recently removed quite a lot of site-wide links leaving the only link on homepage's of some websites, since doing this I have seen a dramatic drop on my keywords, going from position 2-3 to nowhere. Has anyone else experienced anything like this, should I expect to see a return on these keywords? Thanks
Intermediate & Advanced SEO | | Paul780 -
Asking for a link in return for doing a webmaster a favor.
I had this idea that I would reach out to webmasters in my niche and offer to teach them something - for example, how to set up a killer Facebook landing page. In return I would ask them for a link from their site. I have a few quesitons: 1. Would this be considered "white hat"? 2. How would you word the email to the webmasters? I was thinking of something like this: "Hi [webmaster name]. I was checking out your website and your Facebook page. I was wondering if you would like me to show you how to create a great Facebook landing page that will gain you more fans. (As an example, you can see my Facebook page here: [insert link].) In return, all I ask for is a mention, in the form of a link, from your site. What do you think?" What do you guys think?
Intermediate & Advanced SEO | | MarieHaynes0 -
Does having multiple links to the same page influence the Link juice this page is able to pass
Say you have a page and it has 4 outgoing links to the same internal page. In the original Pagerank algo if these links were links to an page outside your own domain, this would mean that the linkjuice this page is able to pass would be devided by 4. The thing is i'm not sure if this is also the case when the outgoing link, is linking to a page on your own domain. I would say that outgoing links (whatever the destination) will use some of your link juice, so it would be better to have 1 outgoing link instead of 4 to the same destination, the the destination will profit more form that link. What are you're thoughts?
Intermediate & Advanced SEO | | TjeerdvZ0