How to get a list of robots.txt file
-
This is my site.
Its in wordpress.I just want to know is there any way I can get the list of blocked URL by Robots.txt
In Google Webmaster its not showing up.Just giving the number of blocked URL's.
Any plugin or Software to extract the list of blocked URL's.
-
If you use Bing Webmaster tools you can see a complete list all URLs blocked by robots.txt. You can export the file and then filter.
Just go to Reports & Data > Crawl Information within your Bing webmaster account. I am not aware of this feature being in Google webmaster tools. Hope this helps.
-
simon_realbuzz buddy If I use this /classifieds/ it means I am blocking all URL starting with it.I want to get a list of all blocked URL's of site.
Example
http://muslim-academy.com/classifieds/
How many URL's associated with this classified are blocked by my robots.txt.
-
I'm sorry I don't follow. If you go to that URL you will see the list of blocked URLs as I've pasted below.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /forum/viewtopic.php?p=
Disallow: /forum/viewtopic.php?=&p=
Disallow: /forum/viewtopic.php?t=
Disallow: /forum/viewtopic.php?start=
Disallow: /forum/&view=previousDisallow: /forum/&view=next
Disallow: /forum/&sid=
Disallow: /forum/&p=
Disallow: /forum/&sd=a
Disallow: /forum/&start=0
Disallow: /forum/memberlist.php
Disallow: /forum/posting.php
Disallow: /classifieds/
Disallow: /forum/index.php
Disallow: /forum/ucp
Disallow: /http://muslim-academy.com/الا�%A..
Disallow: /http://muslim-academy.com/особенн%D
Disallow: /http://muslim-academy.com/ислам-ка%
Disallow: /http://muslim-academy.com/classifieds/ads/Disallow: /http://muslim-academy.com/значени%D..
Disallow: /.ifieds/
Disallow: /.ifieds/ads/
Disallow: /forum/alternatelogin/al_tw_connect.php?authentication=1
Disallow: /forum/search.php -
simon_realbuzz I need a list of blocked URL's not the robots.txt file path.
-
You can view your robots file simply by appending /robots.txt to your site URL. Just put the following http://muslim-academy.com/robots.txt and you'll be able to view your robots file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multiple Pages get rank for one query
Hi to all experts, In google search console , I've found that multiple pages get rank for my main query. How can I solve that? aoi59
Reporting & Analytics | | tarahshiiid0 -
Is there an automated way to determine which pages of your website are getting 0 traffic?
I'm doing a content audit on my company website and want to identify pages with zero traffic. I can use GA for low traffic, but not zero traffic. I can do this manually, but it would take a long time. Are there any tools to help me determine these pages?
Reporting & Analytics | | Ksink0 -
Linked my adwords account to GA and vice versa and still paid search is getting recorded into organic traffic??
Hi Mozzers, I have linked properly my adwords account to GA and vice versa and somehow I can see 3/4 of this paid traffic recorded to organic search. The most confusing part is that I can see 1/4 of the paid traffic under the "paid" metric. At this point I don't know really what should I do? Thank you guys in advance!
Reporting & Analytics | | Ideas-Money-Art0 -
Page Views for a List of URL's (Excel & Analytics)
I've been playing around with a few excel plugins trying to get the visit count for over 350 different URL's on the same website. I cant seem to find a function that can do this. Can anyone advise a work around or a way i can achieve this? Greg
Reporting & Analytics | | AndreVanKets0 -
Getting Traffic for an Unranked Phrase
Over the last month, 40% of a client's search traffic as resulted from a phrase that they are not even ranking for in the top 100; nor are they popping up on PPC ads for it. How is this happening? I feel like I am missing something very obvious.
Reporting & Analytics | | ScriptiLabs0 -
Getting traffic for another site
Hi Everyone, Our website url/brand is very close to another website url/brand. We are non-competing entities. It appears as though this other company has begun a marketing program which has resulted in our traffic skyrocketing. However, it seems to have also resulted in our Pages/Visit and Visit Duration to decrease and our Bounce Rate to increase. Can anyone suggest how to deal with this type of scenario? Thanks,
Reporting & Analytics | | AC_Pro
Robert0 -
Why do I get lots of traffic from a bizarre keyword?
Bit of an odd one but I've been getting a large and steady stream of traffic over the last few months from a very random keyword that according to addwords figures shows "on data". Its our second biggest referring term only beaten by our brand name. We get more traffic from this term than keywords we have invested a lot of time in that show thousands of traffic volume in addwords. When looking at behavioral data its gets odder, a bounce rate of 98.11% time on site 2 seconds and page visits 1.02. So this traffic isn't real traffic and it's not real people. So my questions are, what is it? why do we get this random traffic, has anyone els noticed things like this and is it a problem? I presume it must be something to do with some sort of spam but apart from that i'm stumped. It's just one of those things that has been bugging me so I would appreciate any help. Kind Regards Paul
Reporting & Analytics | | pauldoffman0
