100K Webmaster Central Not Found Links?
-
http://screencast.com/t/KLPVGTzM I just logged into our Webmaster Central account to find that it shows 100k links that are not found? After searching through all of them they all appear to be from our search bar, with no results? Are we doing something wrong here?
-
Ya, I read through that article yesterday & see that they recommend the same setting as the Yoast plugin should be doing? Although I didn't ever get a response from me to see if there is something missing?
For now, I plan on adding this to the robots.txt file & see what results I get?
Do you know the time frame that it takes to get the updates in GWT? Will this update within a few weeks or would it take longer than that?
Thanks for all the help!
BJ
-
Hello BJ.
The robots.txt file must be on your server, in the document root.
Here is information about how to configure robots.txt
Note that is does have a warning at the end, about how you could possibly lose some link juice, but that is probably a much smaller problem than the problem you are trying to fix.
Nothing is perfect, and with the rate that google changes its mind, who knows what is the right thing to do this month.
Once you have edited robots.txt, you don't need to do anything.
- except I just had a thought - how to get google to remove those items from your webmaster tools. I think you should be able to tell them to purge those entries from GWT. Set it so you can see 500 to a page and then just cycle through and mark them fixed.
-
Sorry to open this back up after a month, in adding this to the robot.txt file is there something that needs to be done within the code of the site? Or can I simply update the robots.txt file within Google Webmaster Tools?
I was hoping to get a response from Yoast on his blog post, it seems there were a number of questions similar to mine, but he didn't ever address them.
Thanks,
BJ
-
We all know nothing lasts forever.
A code change can do all kinds of things.
Things that were important are sometimes less important, or not important at all.
Sometimes yesterdays advice no longer is true.
If you make a change, or even if you make no change, but the crawler or the indexer changes, then we can be surprised at the results.
While working on this other thread:
http://www.seomoz.org/q/is-no-follow-ing-a-folder-influences-also-its-subfolders#post-74287
I did a test and checked my logs. A nofollow meta tag and a nofollow link do not stop the crawlers from following. What it does (we think) is to not pass pagerank. That is all it does.
That is why the robots.txt file is the only way to tell the crawlers to stop following down a tree. (until there is another way)
-
Ok, I've posted a question on Yoast.com blog to see what other options we might have? Thanks for the help!
-
It is because Roger ignores those META tags.
Also, google often ignores them too.
The robots.txt file is a much better option for those crawlers.
There are some crawlers that ignore the robots file too, but you have no control over them unless you can put their IPs in the firewall or add code to ignore all of their requests.
-
Ok, I just did a little more research into this, to see how Yoast was handling this within the plugin & came across this article: http://yoast.com/example-robots-txt-wordpress/
In the article he stats that this is already included within the plugin on search pages:
I just confirmed this, by doing this search on my site & looking at the code: http://www.discountqueens.com/?s=candy
So this has always been in place. Why would I still have the 100K not found links still showing up?
-
We didn't have these errors showing up previously, so that's why I was really suspicious? Also we have Joost De Valk's SEO plugin installed on our site & I thought there was an option to turn off the searches from being indexed?
-
Just to support Alan Gray's response, I'll say it's very important to block crawlers from your site search, because it not only throws errors (bots try to guess what to put in a search box), but also because any search results that get into the index will cause content conflicts, dilute ranking values, and worst case scenario, potentially create the false impression that you have a lot of very thin content / near duplicate content pages.
-
the search bar results are good for searchers but not for search engines. You can stop all search engines and Roger (the seomoz crawler) from going into those pages by adding an entry to your robots.txt file. Roger only responds to his own section of the robots file, so anything you make global will not work for him.
User-agent: rogerbot Disallow: /search/*
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
GWT app and link emulation
Hi, I have a gwt site - https://www.whatiswhere.com. I have tab control which emulates the menu. I am planning to put <a>links instead of text into tab labels to create internal links. I am thinking to add java script to stop onclick event of</a> <a>otherwise i will get to the new session of GWT site. What I want is to just change the tab but at the same time have a link for the crawler. Would my approach work? Will it be equivalent to non-follow link? Will it improve the ranking comparing to 'no link at all' case?</a> <a>Thanks, Andrei.</a>
Intermediate & Advanced SEO | | Anazar_20010 -
Do I have to many internal links which is diluting link juice to less important pages
Hello Mozzers, I was looking at my homepage and subsequent category landing pages on my on my eCommerce site and wondered whether I have to many internal links which could in effect be diluting link juice to much of the pages I need it to flow. My homepage has 266 links of which 114 (43%) are duplicate links which seems a bit to much to me. One of my major competitors who is a national company has just launched a new site design and they are only showing popular categories on their home page although all categories are accessible from the menu navigation. They only have 123 links on their home page. I am wondering whether If I was to not show every category on my homepage as some of them we don't really have any sales from and only concerntrate on popular ones there like my competitors , then the link juice flowing downwards in the site would be concerntated as I would have less links for them to flow ?... Is that basically how it works ? Is there any negatives with regards to duplicate links on either home or category landing page. We are showing both the categories as visual boxes to select and they are also as selectable links on the left of a page ? Just wondered how duplicate links would be treated? Any thoughts greatly appreciated thanks Pete
Intermediate & Advanced SEO | | PeteC120 -
Lower quality new domain link vs higher quality repeat domain link
First time poster here with a dilemma that head scratching and spreadsheets can't solve! I'm trying to work out whether to focus on getting links from new domains or to nurture relationships with the bigger sites in our business and get more links. Of the two links below which does the community here think would be more valuable a signal to Google? Both would be links from within relevant text/post copy. Link 1. Site DA 30. No links currently from this domain. Link 2. Site DA 60. Many links over last 12 months already from this domain. I suspect link 1 but given the enormous disparity in ranking power am I correct?! Thanks for any considered opinions out there! Matthew
Intermediate & Advanced SEO | | mat20150 -
301 Externally Linked, But Non-Producing Pages, To Productive Pages Needing Links?
I'm working on a site that has some non-productive pages without much of an upside potential, but that are linked-to externally. The site also has some productive pages, light in external links, in a somewhat related topic. What do you think of 301ing the non-productive pages with links to the productive pages without links in order to give them more external link love? Would it make much of a difference? Thanks... Darcy
Intermediate & Advanced SEO | | 945010 -
Advice on Link Building?
I know webmasters shouldn't focus on link building but unfortunately there are some types of content that doesn't get shared as much as other. And for content to go viral, it ain't that easy and it's almost impossible in some smaller niches where you don't have the volume to go "viral". That said I know about the common link building techniques. I know I can submit guest posts but when you're competing with websites that have over 10,000 backlinks, there is no way I'm going to get close to this with guest posting and commenting on other blogs. One way I found for getting backlinks is to publish interviews. Most of the time, people/businesses you interview like to link to this type of content. Publishing value-added content about other businesses' products or services may get some backlinks in return but not that often. So other than that, can some of you share some "out-of-the-box" link building strategies? Thank you in advance!
Intermediate & Advanced SEO | | sbrault740 -
Link Building for E-Commerce
Hi, Our on page optimization, albeit for a few dupe content issues, is ok - We have good keyword rich URL's, Titles, H1's and unique product descriptions. So now I want to look at building links that will boost our DA and PA's. We have over 2000 products on the store and around 130 categories/subcategories -and I would appreciate any views on where to start - My initial view is to get backlinks from the relevant manufacturer websites to the "shop by brand" page on our site related to these manufacturers - What other strategies should I look at? Thanks, Ben
Intermediate & Advanced SEO | | bjs20100 -
Competing with Spammy Links
One of my client's leading competitors is well stacked in terms of rank/authority. PA: 61, DA: 53. However, in OSE I estimate that +/- of all links on the first page are from sites such as "http://www.shopp011.freedownloadhub.com/Link-Exchange/browse.php?id=17", "http://www.shopp002.freedownloadhub.com/Link-Exchange/browse.php?id=17", "http://www.shopp029.freedownloadhub.com/Link-Exchange/browse.php?id=17". Personally, I would consider this to be a little spammy. However, I admit that I could be wrong. What's the best approach when trying to take on a competitor like this? Wait it out and tell my client to keep blogging/selling as per the schedule until Google pics up on these links?
Intermediate & Advanced SEO | | ShippingContainer0 -
Google, Links and Javascript
So today I was taking a look at http://www.seomoz.org/top500 page and saw that the AddThis page is currently at the position 19. I think the main reason for that is because their plugin create, through javascript, linkbacks to their page where their share buttons reside. So any page with AddThis installed would easily have 4/5 linbacks to their site, creating that huge amount of linkbacks they have. Ok, that pretty much shows that Google doesn´t care if the link is created in the HTML (on the backend) or through Javascript (frontend). But heres the catch. If someones create a free plugin for wordpress/drupal or any other huge cms platform out there with a feature that linkbacks to the page of the creator of the plugin (thats pretty common, I know) but instead of inserting the link in the plugin source code they put it somewhere else, wich then is loaded with a javascript code (exactly how AddThis works). This would allow the owner of the plugin to change the link showed at anytime he wants. The main reason for that would be, dont know, an URL address update for his blog or businness or something. However that could easily be used to link to whatever tha hell the owner of the plugin wants to. What your thoughts about this, I think this could be easily classified as White or Black hat depending on what the owners do. However, would google think the same way about it?
Intermediate & Advanced SEO | | bemcapaz0