Google is Still Blocking Pages Unblocked 1 Month ago in Robots
-
I manage a large site over 200K indexed pages. We recently added a new vertical to the site that was 20K pages. We initially blocked the pages using Robots.txt while we were developing/testing. We unblocked the pages 1 month ago. The pages are still not indexed at this point. 1 page will show up in the index with an omitted results link. Upon clicking the link you can see the remaining un-indexed pages. Looking for some suggestions. Thanks.
-
Thanks. I fetched both the main page and made a slight tweak to the robots and resubmitted last night. It looks like it is making a bit of progress. There is nothing else blocking the pages. We did add a new sitemap when we first launched the pages with no warnings. I did notice yesterday that we had approx 268 broken links that went to 404 pages in WMT in this specific sub-folder. The discovery of the broken link-404 pages by Google seems to be around the same time Google stopped crawling this section of the site. We took care of the broken links this morning. Thanks for the help!
-
Hi,
Fetch the main page(s) with "Fetch as Google" under the Crawl section in Webmaster tools - then submit to the index.
You are sure that there are no other elements blocking the indexing of the page (like meta tag or X-robots tags in the header?
Also fetch the new robots.txt file - to be sure that Google notices that it has changed.
Did you add a sitemap for this new section - does it show any notifications/warnings in WMT?
rgds,
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My brand name has 2 words but Google only indexing as 1 word. Is there a fix?
Hi all...I'm at a loss. I've never had this happen. Google only shows pages of my site when I search the brand name as one word. When I Google the site as one word BrandBrand- it only shows my blog page and about us page plus Twitter and Facebook on page 1. The homepage does not show up at all. When I Google the site as two words Brand Brand - My Facebook page is on page 1 but nothing else. The homepage isn't showing up at all. When I search both words on Bing and Yahoo both are indexing it as two words and shows on page 1. Any ideas?
Technical SEO | | TexasBlogger0 -
Google is still indexing the old domain a year after 301 redirects are put in place
Hi there, You might have experienced this before but for me this is the first. A client of mine moved from domain A (www.domainA.com) to domain B (www.domainB.com). 301 redirects are all in place for over a year. But the old domain is still showing in Google when you search for "site:domainA.com" The HTTP Header check shows this result for the URL https://www.domainA.com/company/cookie-policy.aspx HTTP/1.1 301 Moved Permanently =>
Technical SEO | | iQi
Cache-Control => private
Content-Length => 174
Content-Type => text/html; charset=utf-8
Location => https://www.domain_B_.com/legal/cookie-policy
Server => Microsoft-IIS/10.0
X-AspNetMvc-Version => 5.2
X-AspNet-Version => 4.0.30319
X-Powered-By => ASP.NET
Date => Fri, 15 Mar 2019 12:01:33 GMT
Connection => close Does the redirect look wrong? The change of address request was made on Google Console when the website was moved over a year ago. Edit: Checked the domainA.com on bing and it seems that its not indexed, and replaced with domainB.com, which is the right. Just Google is indexing the old domain! Please let me know your thoughts on why this is happening. Best,0 -
Google Listing Brand Name as Page Title rather than actual set Title
Any search result including our website is displaying our search result title like this: "Brand Name: Partial Page Title" instead of what it should be "Page Title" (as an example). Where is the "Brand Name:" coming from? We've verified we don't have any code that could cause this, and we're blocking in robots.txt directory listings being used for our search result title/meta. This isn't happening for our competitors. Any ideas on why this is, and if it's something we can control?
Technical SEO | | Closetstogo0 -
On our site by mistake some wrong links were entered and google crawled them. We have fixed those links. But they still show up in Not Found Errors. Should we just mark them as fixed? Or what is the best way to deal with them?
Some parameter was not sent. So the link was read as : null/city, null/country instead cityname/city
Technical SEO | | Lybrate06060 -
Should i put a full article on my home page to get google to visit more
Hi, our site is www.in2town.co.uk and i am thinking of putting an article on my home page in the middle under where it says lifestyle news, and getting rid of the middle column and instead have the latest news there to try and get google to visit more. I would like to know if you think this would look messy and not user friendly and if you think if i did do it would it get google to visit the site more often. all day we are always adding articles but on the home page it only shows a few lines of the article so i am concerned that these few lines are not getting google interested in visiting our site more often. we were on page one with our site but now since the upgrade we are on page eight so we are trying to combat this The article would change each time we put a new article on the site. so the article could be on there for ten mins before a new one is there any thoughts on this would be great
Technical SEO | | ClaireH-1848860 -
Two META Robots tags on a page - which will win?
Hi, Does anybody know which meta-robots tag will "win" if there is more than one on a page? The situation:
Technical SEO | | jmueller
our CMS is not very flexible and so we have segments of META-Tags on the page that originate from templates.
Now any author can add any meta-tag from within his article-editor.
The logic delivering the pages does not care if there might be more than one meta-robots tag present (one from template, one from within the article). Now we could end up with something like this: Which one will be regarded by google & co?
First?
Last?
None? Thanks a lot,
Jan0 -
Rectified the onsite after 3 months. How to tell Google to refresh my site ?
Hi.. We sell Joomla addons here, http://www.joomclub.org/joomla-extensions/ This site was started in september 2011 and its converting very well from other sources, but I need it to rank high and get organic traffic as well. We have over 295 pages and almost alla re index, I guess, but doesnt come anywhere in the SERPS, so I thought of investigating it and found that, all the product pages h1 tag was messed up and gone. Just fixed it right now. So, how do I tell Google to refresh and recrawl my pages and index ?
Technical SEO | | qubesys0 -
Home page URL disappears in Google after switching to WordPress
It was a 10 page static HTML page website. 3 year old, PR2. Monday night, copied a WordPress from somewhere to this website's public_html folder and activate it. The home page was "index.html" before switching to WordPress. Now this html file (index.html) has been deleted, so WordPress' Home page can work. All other 9 static html pages are still there in Google index. Just notice it today that the home page URL disappears in Google completely. Why? All other 9 static html pages' URL are still in Google. robots.txt is Allow: / What may have gone wrong to remove the home domain URL from Google index? Thank you for your help!
Technical SEO | | johnzhel0