How to get a list of robots.txt file
-
This is my site.
Its in wordpress.I just want to know is there any way I can get the list of blocked URL by Robots.txt
In Google Webmaster its not showing up.Just giving the number of blocked URL's.
Any plugin or Software to extract the list of blocked URL's.
-
If you use Bing Webmaster tools you can see a complete list all URLs blocked by robots.txt. You can export the file and then filter.
Just go to Reports & Data > Crawl Information within your Bing webmaster account. I am not aware of this feature being in Google webmaster tools. Hope this helps.
-
simon_realbuzz buddy If I use this /classifieds/ it means I am blocking all URL starting with it.I want to get a list of all blocked URL's of site.
Example
http://muslim-academy.com/classifieds/
How many URL's associated with this classified are blocked by my robots.txt.
-
I'm sorry I don't follow. If you go to that URL you will see the list of blocked URLs as I've pasted below.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /forum/viewtopic.php?p=
Disallow: /forum/viewtopic.php?=&p=
Disallow: /forum/viewtopic.php?t=
Disallow: /forum/viewtopic.php?start=
Disallow: /forum/&view=previousDisallow: /forum/&view=next
Disallow: /forum/&sid=
Disallow: /forum/&p=
Disallow: /forum/&sd=a
Disallow: /forum/&start=0
Disallow: /forum/memberlist.php
Disallow: /forum/posting.php
Disallow: /classifieds/
Disallow: /forum/index.php
Disallow: /forum/ucp
Disallow: /http://muslim-academy.com/الا�%A..
Disallow: /http://muslim-academy.com/особенн%D
Disallow: /http://muslim-academy.com/ислам-ка%
Disallow: /http://muslim-academy.com/classifieds/ads/Disallow: /http://muslim-academy.com/значени%D..
Disallow: /.ifieds/
Disallow: /.ifieds/ads/
Disallow: /forum/alternatelogin/al_tw_connect.php?authentication=1
Disallow: /forum/search.php -
simon_realbuzz I need a list of blocked URL's not the robots.txt file path.
-
You can view your robots file simply by appending /robots.txt to your site URL. Just put the following http://muslim-academy.com/robots.txt and you'll be able to view your robots file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to get multiple pages to appear under main url in search - photo attached
How do you get a site to have an organized site map under the main url when it is searched as in the example photo? SIte-map.png
Reporting & Analytics | | marketingmediamanagement0 -
Referral Exclusion List - Data Questsion
This may seem like a silly question. Since we are having an issue with self-referrals, we checked all the pages and everything is tagged properly, I used the referral exclusion list to exclude our domains. Question is, since we had a large number of our revenue coming in from the self-referring traffic, what happens to that revenue data once I add our domains to the referral exclusion list?
Reporting & Analytics | | K2_Sports0 -
Stop getting info from Google analytics on purchases in our site
Hi guys, We have eCommerce.
Reporting & Analytics | | WayneRooney
We connected the site to the Google analytic eCommerce.
Everything was work fine until 3 weeks ago. Suddenly we stooped getting purchases information in the analytic although i see purchases in the website. We didn't change anything in the website and i really don't know how to solve this problem.
If someone here can point me where i can get some info on how to fix it it can be great. Thanks a lot!0 -
Is it possible to get demographic and interest information from DoubleClick cookies?
We use Google Analytics and we are currently extracting information from the Google Analytics cookies about our visitors. Is it possible to access DoubleClick cookies in a similiar fashion and get some demographic/interest information for each visitor to our website (if they have a DoubleClick cookie set)? If so, any information on how to retrieve it would be very appreciated.
Reporting & Analytics | | WebpageFX0 -
Uptick in not tracked conversions / anyone have a list of things that google analytics will not track
There seems to have been an uptick in users on our site not being tracked in Google Analytics cause I see a lot more un-tracked revenue in the last 6 months then I used to. I know analytics is still working as it has been tracking a normal amount of visits but I assumed there might be a reason less would be actually showing up in analytics (mabye a change is what is being reported as organic). I know a lot of stuff goes into "not provided" such as logged in search and stuff like that but is there a list of all of the ones that go into not provided and all that just do not get tracked (javascript not enabled, iOS?). If it could be something else as well let me know. Thanks for the help!
Reporting & Analytics | | Gordian0 -
Page Views for a List of URL's (Excel & Analytics)
I've been playing around with a few excel plugins trying to get the visit count for over 350 different URL's on the same website. I cant seem to find a function that can do this. Can anyone advise a work around or a way i can achieve this? Greg
Reporting & Analytics | | AndreVanKets0 -
Search Engine blocked by robots.txt
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
Reporting & Analytics | | MirandaP0 -
List all URL's indexed by google
Hi all i need a list of all urls google has indexed from my site i want this in excel format or csv how do i go about getting this thanks in advance
Reporting & Analytics | | Will_Craig0