Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site property is verified for new version of search console, but same property is unverified in the old version
Hi all! This is a weird one that I've never encountered before. So basically, an admin granted me search console access as an "owner" to the site in search console, and everything worked fine. I inspected some URL's with the new tool and had access to everything. Then, when I realized I had to remove certain pages from the index it directed me to the old search console, as there's no tool for that yet in the new version. However, the old version doesn't even list the site under the "property" dropdown as either verified or unverified, and if I try to add it it makes me undergo the verification process, which fails (I also have analytics and GTM access, so verification shouldn't fail). Has anyone experienced something similar or have any ideas for a fix? Thanks so much for any help!
Reporting & Analytics | | TukTown1 -
Search Console Linking
Hey everyone! Quick question; there have been problems in the past with some of our accounts' Search Console integration becoming unlinked from Google Analytics without us knowing about it. I did some digging into Google Analytics to see if there was a way to create an alert for search queries dropping off, but there doesn't seem to be any options for Search Console alerts like there are for other integrations like Google Ads. Has anyone discovered a way to have an alert or some type of notification set up for when a Search Console integration is unlinked in Google Analytics or if search queries drop off significantly? I'd appreciate the help!
Reporting & Analytics | | ReunionMarketing0 -
Google search console
Checking our links to our site on GSC and we've dropped from 515 to 324 over night - is this normal?
Reporting & Analytics | | RayflexGroup0 -
Can not divide in different properties a domain in Search Console (Webmaster Tools)
Dear Moz Community, I hope you can give me a hand with the following questions. Im in charge of SEO of an ecommerce site in LATAM. It´s service is available in several countries, therefore each country has it subdirectory Eg. /ar /pe /co /bo /cl /br,etc... (in the future we will move to differente ccTLDs). I have been recomended to split or create different Search Console or Webmaster Tools properties (one for each subdirectory) but when Im creating a new property with a subdirectory, lets say www.domain.com/ar, Webmaster tools starts creating a property for www.domain.com/ar/ (NOTICE THE LAST SLASH) and it returns since that page doesn´t exist, what do you recomend me to do? Best wishes, Pablo Lòpez C
Reporting & Analytics | | pablo_carrara0 -
Looking for a GA developer to block internal traffic and integrate with Snipcart JS API - paid
Hi There, We're looking for someone to help us setup: Blocking internal traffic using a custom dimension/query URL as described here: http://www.simoahava.com/analytics/block-internal-traffic-gtm/ Integrate Google Analytics ecommerce tracking with the Snipcart API as described here: https://snipcart.com/blog/integrating-snipcart-with-google-analytics-ecommerce-tracking Discuss Kiss Metrics and possibly integrate with Snipcart We're a small, 2 person startup so are ideally looking for a freelancer or small firm to help - Any recommendations or pointers in the right direction would be much appreciated. Cheers Ben
Reporting & Analytics | | cmscss0 -
(Not provided) organic search results in Analytics
Hi, So from what I've read, (not provided) organic search results in Analytics is a result of the user being logged into Google. The problem I have is about 20% of the organic search results are (Not provided), so what is mainly left are branded search terms. I think I'm clutching at straws, but is there anything I can do to see these organic traffic sources?
Reporting & Analytics | | JuiceBoxOM0 -
'Search Queries Report' in Webmaster Tools Question
Hi, How much do you use the search queries report in webmaster tools to research current rankings/movements? It does look like a great tool but the data doesn't seem to be spot on. For example a keyword over a week might have flux in position so lets say 6.0 then 9.2 for 3 days then back to 6.0. But I check the serp's for this keyword everyday and didn't see any movement?!?! Is this a good tool for you?
Reporting & Analytics | | activitysuper0 -
How can I track search engine optimization data in Google analytics?
My website is linked to a Google Analytics web property. But, I am not able to track search engine optimization data in Google Analytics. So, How can I get it done?
Reporting & Analytics | | CommercePundit0