Files blocked in robot.txt and seo
-
I use joomla and I have blocked the following in my robots.txt is there anything that is bad for seo ?
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
Disallow: /mailto:myemail@myemail.com/
Disallow: /javascript:void(0)
Disallow: /.pdf
-
What you have there is just blocking rootdomain.com/javascript:void(0). Googlebot can execute and index JavaScript; you should not block it without a good reason. I'd let it read the JavaScript and see the submenus.
-
Thank you and is blocking javascript bad ? ( I was thinking about submenus )
-
If you don't want pages in those Disallowed directories to be indexed, then you're doing fine. These pages won't be able to be crawled, so, they won't be likely to appear in search results for any search engines.
The last three entries look fishy to me. I'd need to know what types of URLs you're trying to block to fix them. For the last one, if you're looking to block all pdfs on your site, the syntax would be Disallow: /*.pdf.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Javascript and SEO
I've done a bit of reading and I'm having difficulty grasping it. Can someone explain it to me in simple language? What I've gotten so far: Javascript can block search engine bots from fully rendering your website. If bots are unable to render your website, it may not be able to see important content and discount these content from their index. To know if bots could render your site, check the following: Google Search Console Fetch and Render Turn off Javascript on your browser and see if there are any site elements shown or did some disappear Use an online tool Technical SEO Fetch and Render Screaming Frog's Rendered Page GTMetrix results: if it has a Defer parsing of Javascript as a recommendation, that means there are elements being blocked from rendering (???) Using our own site as an example, I ran our site through all the tests listed above. Results: Google Search Console: Rendered only the header image and text. Anything below wasn't rendered. The resources googlebot couldn't reach include Google Ad Services, Facebook, Twitter, Our Call Tracker and Sumo. All "Low" or blank severity. Turn off Javascript: Shows only the logo and navigation menu. Anything below didn't render/appear. Technical SEO Fetch and Render: Our page rendered fully on Googlebot and Googlebot Mobile. Screaming Frog: The Rendered Page tab is blank. It says 'No Data'. GTMetrix Results: Defer parsing of JavaScript was recommended. From all these results and across all the tools I used, how do I know what needs fixing? Some tests didn't render our site fully while some did. With varying results, I'm not sure where to from here.
Intermediate & Advanced SEO | | nhhernandez1 -
Onsite SEO vs Offsite SEO
Hey I know the importance of both onsite & offsite, primarily with regard to outreach/content/social. One thing I am trying to determine at the moment, is how much do I invest in offsite. My current focus is to improve our onpage content on product pages, which is taking some time as we have a small team. But I also know our backlinks need to improve. I'm just struggling on where to spend my time. Finish the onsite stuff by section first, or try to do a bit of both onsite/offsite at the same time?
Intermediate & Advanced SEO | | BeckyKey1 -
Cookieless subdomains Vs SEO
We have one .com that has all our unique content and then 25 other ccltd sites that are translated versions of the .com for each country we operate in. They are not linked together but we have href lang'd it all together. We now want to serve up all static content of our global website (26 local country sites, .com, .co.uk, .se, etc) from one cookie-less subdomain. Benefit is speed improvement. The question is whether from an SEO perspective, can all static content come from static.domain.com or should we do one for each ccltd where it would come form static.domain.xx (where xx is localised to the domain in question)
Intermediate & Advanced SEO | | aires-fb770 -
Social Impacts on SEO? How to Do this?
I'm new in SEO and heard by one of my friend that social signals are important for SEO of a website. If people have shared a website's url on their twitter, then it will automatically get rank in google. Is that true and how google sees this social sharing? and how can I do this for my website?
Intermediate & Advanced SEO | | hammadrafique0 -
Robots.txt & Duplicate Content
In reviewing my crawl results I have 5666 pages of duplicate content. I believe this is because many of the indexed pages are just different ways to get to the same content. There is one primary culprit. It's a series of URL's related to CatalogSearch - for example; http://www.careerbags.com/catalogsearch/result/index/?q=Mobile I have 10074 of those links indexed according to my MOZ crawl. Of those 5349 are tagged as duplicate content. Another 4725 are not. Here are some additional sample links: http://www.careerbags.com/catalogsearch/result/index/?dir=desc&order=relevance&p=2&q=Amy
Intermediate & Advanced SEO | | Careerbags
http://www.careerbags.com/catalogsearch/result/index/?color=28&q=bellemonde
http://www.careerbags.com/catalogsearch/result/index/?cat=9&color=241&dir=asc&order=relevance&q=baggallini All of these links are just different ways of searching through our product catalog. My question is should we disallow - catalogsearch via the robots file? Are these links doing more harm than good?0 -
Shopify and multiple stores for seo
Hi, We have a potential client who has multiple stores targeting different countries using Shopify. He has set the domains for specific countries using geo targeting in webmaster tools. Anybody had any experience on how fool proof this is? (all the sites have the exact same products/content on them) Thanks
Intermediate & Advanced SEO | | OnlineAssetPartners0 -
SEO firm site audit
needs recommendation for a site audit. Post panda/ post penguin experience preferred. thanks
Intermediate & Advanced SEO | | skyao0 -
Local SEO tips
Hi! I'm working on http://www.hgsplumbingandheating.co.uk/ at the moment, and it's going pretty well for local terms such as 'Norwich plumber' etc. Has anyone got any tips though on how it can be improved, especially with regard to getting the homepage ranking above the Places listings? Thanks!
Intermediate & Advanced SEO | | neooptic0