Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Console Linking
Hey everyone! Quick question; there have been problems in the past with some of our accounts' Search Console integration becoming unlinked from Google Analytics without us knowing about it. I did some digging into Google Analytics to see if there was a way to create an alert for search queries dropping off, but there doesn't seem to be any options for Search Console alerts like there are for other integrations like Google Ads. Has anyone discovered a way to have an alert or some type of notification set up for when a Search Console integration is unlinked in Google Analytics or if search queries drop off significantly? I'd appreciate the help!
Reporting & Analytics | | ReunionMarketing0 -
Google search console
Checking our links to our site on GSC and we've dropped from 515 to 324 over night - is this normal?
Reporting & Analytics | | RayflexGroup0 -
Search traffic hit after switching magazine to subdomain
Hi there, I'm fairly new at all of this and would appreciate any help with understanding why our website has taken a hit in traffic. We curate an online magazine, which was previously accessible through: forensicoutreach.com. It was receiving about 2,000+ unique visitors per day up until a week ago, when we changed a few things. However, the magazine doesn't reflect what our business does, so we created a product-focused web presence on forensicoutreach.com, and moved the magazine (which everyone loved) to library.forensicoutreach.com (DA 37, PA 1). We thought separating the properties was a good idea, but now I'm not so sure. Our traffic on library.forensicoutreach.com is 1,500 (so 500 less than usual!) and our main property has about 56 unique visits a day. It's pretty substantial. A few questions: 1. If we move the library to forensicoutreach.com/library, will that make any difference? 2. Where did we go wrong here and how can we fix it? Any help would be appreciated. Thanks.
Reporting & Analytics | | shivaniseos1 -
Looking for a GA developer to block internal traffic and integrate with Snipcart JS API - paid
Hi There, We're looking for someone to help us setup: Blocking internal traffic using a custom dimension/query URL as described here: http://www.simoahava.com/analytics/block-internal-traffic-gtm/ Integrate Google Analytics ecommerce tracking with the Snipcart API as described here: https://snipcart.com/blog/integrating-snipcart-with-google-analytics-ecommerce-tracking Discuss Kiss Metrics and possibly integrate with Snipcart We're a small, 2 person startup so are ideally looking for a freelancer or small firm to help - Any recommendations or pointers in the right direction would be much appreciated. Cheers Ben
Reporting & Analytics | | cmscss0 -
Blog posts not appearing in Search
http://www.themorrisagency.co.uk/blog works well to index my niche terms. Looked at webmaster tools today and since today it has put my indexed pages from 900+ to 11 ??? I have recently added a new site format via a web designer and was all working fine. Now none of my posts are appearing.I have resubmitted site map for 993 pages Is there anything I can do to stop/ prevent this from happening again? and what do you think was the cause? Thanks everyone! Daniel
Reporting & Analytics | | Agentmorris0 -
Difference between site: search and Total Indexed in Google Webmaster Tools.
This morning I did a search on Google for my site using the site: operator. I noticed that the number of results returned was significantly different than the "Total indexed" in Google Webmaster Tools. What is the difference and is it normal to have two very different numbers here?
Reporting & Analytics | | Gordian0 -
My Google Webmaster Search Queries are 0??!!
For the past 2 days Google Webmaster has recorded 0 search queries to my site, but traffic seems stable. Any ideas about why this is happening and what I should do about it? Or is it a Google glitch I should not be concerned about (hoping for this option)
Reporting & Analytics | | theLotter0 -
SERPS different based on location of search even with non-personalization
Hello Mozzers, Our agency's website, www.kenta.ro - ranked for a long time at #1 for "ann arbor seo" and similar keywords. For the past several (4-5) months we've been sitting around #5. My guess was that this was Google playing around with the results but I'm not sure why we have been at this position for such a long time. I have a vpn that I use for checking rankings overseas and if I connect to a server in Chicago, LA, Ontario, etc we show up as #1 - only when you search for "ann arbr seo" in our area do we get a lower ranking. All rank checking programs including seomoz show us at the #1 position because of this. What this means for us is that all of the traffic we target with this keyword sees the poor result, while the rest of the world sees the great result (should they search for it). How can we ensure that our target market finds us at #1 like the rest of the world does? Thank you in advance. weabi.png
Reporting & Analytics | | kentaro-2569290