100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Many Natural Links Do You Earn?
Hello Mozzers, This is a bit of an open ended question and I don't think any one person is going to be the same. I have recently seen the light in my link building practices and I am trying to get a feel for what to expect in terms of natural link acquisition in an effective content marketing strategy. My question is how many natural links do you generally find yourself earning after the first 12 months of content creation/placement with a new website/industry? I know this is going to be a question with a multitude of different answers. I look forward to your valuable insight as always!
Intermediate & Advanced SEO | | ChoChauRice1 -
Rel=canonical and internal links
Hi Mozzers, I was musing about rel=canonical this morning and it occurred to me that I didnt have a good answer to the following question: How does applying a rel=canonical on page A referencing page B as the canonical version affect the treatment of the links on page A? I am thinking of whether those links would get counted twice, or in the case of ver-near-duplicates which may have an extra sentence which includes an extra link, whther that extra link would count towards the internal link graph or not. I suspect that google would basically ignore all the content on page A and only look to page B taking into account only page Bs links. Any thoughts? Thanks!
Intermediate & Advanced SEO | | unirmk0 -
Google WMT/search console showing thousands of links in "Internal Links"
Hi, One of our blog-post has been interlinked with thousands of internal links as per search console; but lists only 2 links it got connected from. How come so many links it got connected internally? I don't see any. Thanks, Satish
Intermediate & Advanced SEO | | vtmoz0 -
Links in body text
From a purely SEO /link juice perspective, is there any benefit to linking from body text to a page that is in a pervasive primary navigation? The primary nav puts a link at the top of the HTML. With the tests done by members of this site, the "first link counts" rule negates the link juice value of a link in the body text if there is already a link in the nav. Now I've also seen the data on using hash tags to get a second or third link, but ignoring that, it would seem that links in the body text to pages in the nav have zero effect. This brings me to another question - block level navigation. If anchor text links pass more juice than links in the top navigation, why would you put your most coveted target pages in the top nav? You would be better off building links in the content, which would create a poor user experience. To me, the theory that anchor text links in the body pass more juice than links in the primary nav doesn't make any sense. Can someone please explain this to me?
Intermediate & Advanced SEO | | CsmBill0 -
Internal Links not being Identified on OSE
Greetings Mozzers, When ever I check my home page on OSE it says I have a total of 5 internal links. Obviously this is WAY off. I've used all relative links, if I were to removed all relative and make them absolute, would there be a better chance of OSE identifying them instead of losing that juice? I think this is huge to resolve as when I compare my site to competitors, almost all factors are in our favor except this huge gap of only 5 internal links. I'm using Drupal CMS. For example, Drupal normally outputs internal links as "/about" and "/about/team" in the menus. If we changed it to "https://monsterweb.net/about", and "https://monsterweb.net/about/team", would that make a difference? Thanks for all the advice and clarification on this matter.
Intermediate & Advanced SEO | | MonsterWeb280 -
Link Building Question
Hi, I have a 2 month old blog with me, i have submitted a few press releases for the start, later in these 2 months, i got about 40 guest posts, which i've written and submitted at myblogguest site. My niche is in health. Currently my serps are at 16th page which is not a good position. I want to do more link building, but at myblogguest, no one are interested in my niche and don't want to publish content related to my niche, so it becoming hard for me to find guest blogs related to my niche. But i want to get more links in order for my blog to rank well. Is it ok if i write guest posts in other niches as well like technology and put a link in author's resource box? Does it become useful? please help as i find no other sources for my link building task, i tried researching for guest blogs in google also, but i don't find any related to my niche. Seems like, i cannot go further with my link building. Please help me. Thanks
Intermediate & Advanced SEO | | Vegitt
Dheer0 -
Penguin or paid link penalty, or both?
Hello, I have a site, macpokeronline.com, that has seen dramatic decrease in visitors in the last few months, it has went down from 800 per day to 200 per day. It is a pretty complex situation. The site owner purchased paid links from reputable mac sites for years (they were more of followed advertisements, but were only there for SEO Purposes), now that i'm going through the link profligate ins OSE, I can see that a majority of their links come from these sites. There is also a branding issue, there are almost 15,000 links with the anchor text of "macpokeronline.com" These are obviously branded links, I don't know the best way to deal with them (though the majority are coming from the paid link sites) We have just sent the request in to remove the paid links from the sites, and i'm guessing since he is paying over $1000 a month for the links, they will be removed quickly. The site has been receiving significantly less traffic since penguin (apr 24-25) We received a message on July 19th which was the generic unnatural link warning, saying that once we remove links make a reconsideration request. Then on July 23rd, we received another message that says they are taking a "very targeted action on the unnatural links instead of your site as a whole" which I have never seen before. This damage was done before I was hired by this client, I just want to get his traffic back up so I can help him even further, I want to know more about the steps I should take. 1. I will definitely remove the paid ads What else should I do, thanks Zach
Intermediate & Advanced SEO | | BestOdds0 -
Is there an optimal ratio of external links to a page vs internal links originating at that page ?
I understand that multiple links fro a site dilute link juice. I also understand that external links to a specific page with relevant anchortext helps ranking. I wonder if there is an ideal ratioof tgese two items
Intermediate & Advanced SEO | | Apluswhs0