Are you redirecting everything on www to non-www? If so, you don't really need a robots.txt to be served for the www subdomain. Google will ignore the original robots.txt file if it is given a 301 anyway.
Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Best posts made by Xiano
-
RE: I have two robots.txt pages for www and non-www version. Will that be a problem?
-
RE: Is there a way to get a list of all pages of your website that are indexed in Google?
There are a number of different ways, but I'd always start with Google Search Console Coverage reports.
If you haven't already, sign up and configure Search Console.
Then, go to your property > Coverage.
Then select the Valid "tab". You'll then be able to click on the two types of valid ("Submitted and indexed" and "Indexed, not submitted in sitemap"). Within each of these categories, you'll be able to download a CSV of the pages lists.
In the same area you'll be able to see lists of pages that Google knows about but hasn't indexed for one reason or another.
-
RE: Does Google ignore content styled with 'display:none'?
It will be crawled, but Google will generally apply less weighting to any content. Google is generally pretty good at understanding things that are display:none for design reasons and usually won't penalise unless it thinks you are trying to manipulate the system.
That said, if it isn't the main heading, it shouldn't really be H1; you should only have one H1 per page. Although HTML5 allows for multiple H1 within sections of a page, that doesn't really apply here.