Get a list of robots.txt blocked URL and tell Google to crawl and index it.
-
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list.
My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches,
One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file.
I need urgent recommendation as I do not want to see drop in my traffic any more.
-
"changing the lastmod of those pages to today".
How can I make these changes?
Right now the news is that Resubmitted the Sitemap and no warnings this time.
-
I imagine that since you've got a robots txt error you'll probably ended closing a whole directory to bots which you wanted to be indexed. You can easily spot the directory and resubmit a sitemap to google changing the lastmod of those pages to today and the priority to 1 but only of those pages.
If you still receive warnings it may be due to errors in your sitemap. You're probably including some directory you don't want. You can try it in GWT putting in the box at the bottom the url you want to maintain in the index and then trying to see if some urls are being blocked by your robots.
If you want you can post here your robots and the URIs you want to be indexed without knowing the domain so that won't be public. Hope this may help you
-
Ok Resubmitted it.but even with updated file it gives a lot of errors.I think it takes some time.20,016 warnings
I have not added no index attribute in my header region.It was all messy stuff with robots.txt file.It means that with site showing up in SERP the rank will probably be the same or it was deranked?
-
Go into GWMT and resubmit sitemap.xml files (with the URLs you want indexed) for recrawling and Google will digest the sitemaps again, instead of waiting for Googlebot to come around on their own, you are requesting it to come around, also include those new sitemap files in your robots.txt file.
-
In Google Webmaster Tools, go to Health -> Fetch As Google. Then add the previously blocked URL and click Fetch. Once you've done that, refresh the page and click "Submit to index". That should get Google indexing those pages again.
Getting external links to your pages also helps get pages crawled & indexed, so it may be worth submitting your pages to social bookmarking sites, or get other types of backlinks to your previously blocked pages if possible.
-
Since you fixed your robots.txt file you should be good to. It will probably take a few days for Google to recrawl your site and update the index with the URLs they are now allow to crawl.
Blocked URLs can still show up in SERPs if you haven't defined the no-index attribute in your section.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to take down some pages, how do I inform google?
Hey Guys, I'm hoping someone can help - I'm in the midst of a site re-design - whilst one of our biggest reasons for the redesign was to create more space to write valuable content and unique content I have been reading other posts on moz about content auditing. I have come across a few articles in my own blog that are from 250 - 300 words to which the articles seem similar and the traffic is low. I'm wondering while I'm left to consolidate these articles and create a fresh article for each entry that is more in-depth, when I consolidate or delete these pages - do I need to inform google these pages have now been deleted? If so using Wordpress what is the best way to do this? Cheers and would appreciate some advise Thanks
Reporting & Analytics | | edward-may0 -
Why is this tag not firing in Google Analytics?
I setup Google Tag Manager on this site- http://germanhausbarn.com I am trying to setup event tracking for the donate, newsletter, and Contact Us button at the bottom of the page. The most recent version is published, and I ran debug and it shows that they fire, but nothing is coming up in analytics. Any thoughts?
Reporting & Analytics | | EcommerceSite0 -
Google Analytics, new property or views?
Hi All We have an existing Google analytics account which was customised relatively heavily a few years back, with custom variables, e-commerce tracking and the a large number of filters. We have determined that the custom variables and e-commerce tracking, at least for now, are no longer required and actually complicating our lives. We want to simplify the account and add a fresh set of goals and events. Our initial train of thought was to create an entire new property with the usual views, raw, test, main etc. We are proposing to retain both properties (6 months) so we can still do some comparative stuff. We're now wondering if we should retain the existing property and just create new views, the downside being that whilst we can compare data with previous periods it's meaningless to some degree as the filters etc are different. The plus of more views is that we're not firing the tag twice retaining two properties. Views and opinions welcome. Thanks
Reporting & Analytics | | mde9110 -
Google Analytics Organic Search Keywords Suddenly Displaying FulL Urls
In my Google Analytics, the top keywords for Organic Search are suddenyl displaying full URLs. For example, now the third and fourth keywords are http://www.domain.com/highly-specific-URL. These have all started recently around the same day, July 12th. I've checked back, and we've made no internal changes to the site around that time that could affect this. Any thoughts on this? Thanks! P.S. It might be related to rich snippets, but I cannot tell at this point.
Reporting & Analytics | | 10SL0 -
Homepage disappeared from Google's index
The Title says it all..I just discovered that the Homepage is not in Google's index anymore. Homepage rankings have plummeted, our top keywords are nowhere to be found but most of keywords from deeper pages have dropped just one or two places. We just change the website design and some content but I strongly believe this is definitely something else due to the fact that it all happened so fast! There is one thing that I have to mention that might have been caused all this..our email client (Outlook) is using a lot of resources from our server and for the last couple of days the website was down quite a lot.
Reporting & Analytics | | echo1
There are some crawling errors in GWT but the homepage has been crawled because is not there, no other messages. Where should I look for?0 -
Is Google analytics not working properly anymore?
Hi mozzers, We launched a website 2 months ago and when monitoring Organic data on GA we had an average of 23% bounce rate like a month ago but the last 2 weeks I noticed that bounce rate=0%. This is very weird especially that we haven't modified the existing pages and that multiple keywords show an average site duration of 0 seconds. Any thoughts about this discrepancy? Do you think that there is an issue with the GATC implemented? Thanks
Reporting & Analytics | | Ideas-Money-Art0 -
Google Maps not passing referral data
Google Maps is not passing referral data (URLs, not KWs). Google+ Local is referring, but nothing from maps. Maps referrals appear to be coming across as direct. Any ideas? We haven't found anything online, one of the guys at the office documented what we did find, using Chrome's debugger - http://manofactionmetrics.com/2012/11/02/google-maps-not-passing-any-referral-data/
Reporting & Analytics | | Danieljacobree0 -
A lot of traffic to one page from Google referral
We recently received a lot of traffic to one page from
Reporting & Analytics | | underthesun808
google.com referral. When I look in analytics it reports that the traffic is
coming from /url that’s not real helpful. Is there a way to get more specific
information as to what the referring url was?0