Best way to create robots.txt for my website
-
How I can create robots.txt file for my website guitarcontrol.com ?
It is having login and Guitar lessons.
-
Hi,
First you need to understand your website need, you have to decide which part of your website should not be indexed or crawled by SE bots, like your website provides user login and user areas, if you are providing private dashboard for your user then it should be blocked by robots.txt (or you can use meta tag to prevent robots from crawling and indexing your particular page like ) or you can learn more about robots.txt here https://moz.com/learn/seo/robotstxt
Hope it helps
-
I see that you're on WordPress.
This CMS create "virtual" robots.txt. You can see this here:
https://codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_OptimizationBut on your website there is error in robots.txt and you should see in web server log files (access and error) why this is happening. Also you may need looking .htaccess because something preventing this text file to be accessed.
There is alternative way for using robots.txt in WordPress. All you need is to create new and blank robots.txt in same folder and put this there:
User-agent: *
Disallow:Then save file and that's all. Now bad news - WP can't control indexing and crawling anymore.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My website is constantly decreasing
For few weeks ago my website is constantly decreasing in search position. I lost keywords and is gooooing down.
Technical SEO | | Dan_Tala
Although it is well rated on several on page and off page seo verification software that I have tried.
I checked Google search console and Analytics and found no major problems. However… from one day to another it keeps going down.
I also checked what the main competitors are doing and they are not doing well, at all.
The main competitor actually has a creepy website. Totally devoid of onpage or offpage SEO but with an enormous number of backlinks. And of a very bad quality, which should disqualify it, still…
Few weeks ago I changed something.
In the pages I had H1, 4xH2, no H3 and an H4 without content.
An unnatural H tag structure.
Now I have H1, H2, H3, 3xH4, with the coherent information.
Theoretically, Google should have been “happy” or I’m missing something. I use a SAAS platform.
I just found out that they made changes to the keywords (tags).
I am selling toner cartridges for printers.
So…
The tags are printer models and generate a url in which they have the products.
Ex. https://www.sertit.ro/cartus-imprimanta-cilindru-color-hp-laserjet-pro-m-177fw goes to the products for that printer model.
The question is… should I make tag canonical?
Is it possible for products to loose so much in Google search?0 -
Robots.txt Syntax for Dynamic URLs
I want to Disallow certain dynamic pages in robots.txt and am unsure of the proper syntax. The pages I want to disallow all include the string ?Page= Which is the proper syntax?
Technical SEO | | btreloar
Disallow: ?Page=
Disallow: ?Page=*
Disallow: ?Page=
Or something else?0 -
Google is indexing blocked content in robots.txt
Hi,Google is indexing some URLs that i don't want to be indexed and also is indexing the same URLs with https. This URLs are blocked in the file robots.txt.I've tried to block this URLs through Google WebmasterTools but Google doesn't let me do it because this URL are httpsThe file robots.txt is correct so, what can i do to avoid this content to be indexed?
Technical SEO | | elisainteractive0 -
Is there any value in having a blank robots.txt file?
I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place. Is there any merit in having a blank robots.txt file? What is the minimum you would include in a basic robots.txt file?
Technical SEO | | NicDale0 -
What I doing wrong when trying to search for links from external websites to my website
This is just the little frustrating question nothing important but I’m sure somebody will know the answer. In the white board Friday this week Rand suggested at one point that when you’re searching for results links to your website if you put a - followed by site followed by your url like –site:yourwebsite.com you get the results of pages with links on other websites but excluding your own webpages but it just doesn’t work I get no results just an error message, any idea why? If I remove the - I get tons of results but there on my own webpages……….
Technical SEO | | whitbycottages0 -
Multi Language websites!
Hello, What is the best way for a website to have more languages? For example www.site.com is in english BUT the content will be translated in other 4 languages .es .it .fr .de . Should i buy 4 diferent domains www.site1.es www.site2.it etc... or create folders like site.com/es/ site.com/it etc OR create subdomains es.site.com Few months ago i was 100% sure..to go and buy diferent domains then to create diferent content on that languages..but now..i am not so sure. Thank you very much for your help!
Technical SEO | | willyg0 -
Anchor Text the best way to use it
Hi i am wanting to gain links to my site and make the links relevant and have seen many sites that do not have the keyword on their site and have been told as well as having the keywords in the image that they will also have anchor text from other sites. Can anyone please tell me the important of anchor text and how i should use it. Is it wise to build anchort text links within your own site and what other sites should you be looking for. I have used a few article sites to gain anchor text links but most of these are no follow which does not help me. any advice would be great
Technical SEO | | ClaireH-1848860 -
Best way to create page title to products catalog
Hi guys, i'm having problems with duplicated page title, and i want to know what is the best way to avoid this problem. The example is like this: Title page (A): Product name A - category - section Title page (B): Product name B - category - section, What you think guys i can resolve this problem, Thank you so much for your help.
Technical SEO | | NorbertoMM0