Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strange - Search Console page indexing "../Detected" as 404
Anyone seen this lately? All of a sudden Google Search Console is insisting in Page indexing that there is a 404 for a page that has never existed on our client's site: https://........com.au/Detected We've noticed this across a number of sites, precisely in this way with a capitalised "/Detected" To me it looks like something spammy is being submitted to the SERPs (somehow) and Google is trying to index that and then getting a 404. Naturally MOZ isn't picking it up, cause the page simply never existed - it's just happening in Search Console 2afc7e35-71e4-4e25-80a3-690bf10776a7.png It comes and it goes in the 404 alerts in Console and is really annoying. I reckon it started happening late 2022.
Reporting & Analytics | | DanielDL0 -
Why would on-site search queries show up as referral traffic?
The site analytics have been set up for over a year and suddenly last month there was a huge spike in referral traffic (1100+ sessions). Upon further investigation, the majority of it was coming directly from internally, either as mysite.com or search.mysite.com and the landing pages from the referrals are all /search.html?query=* This was never an issue before so I'm trying to understand what could have changed. I'm following up with the client to find out if their dev team may have changed anything related to their search engine but I'm wondering if there may be another explanation. A few notes: previously mysite.com / search.mysite.com were not in the Referral Exclusion list. I've added them now but this was never an issue before. Thanks in advance!
Reporting & Analytics | | SEMnMs0 -
In Google Search Console: Total Clicks differ from the sum clicks on search queries
Hi.This may be a google technical question, but I've searched a lot and I couldn't find any certain information about that.The problem appears when you look at two stats in Search Console ( WebMasters Tools, some months ago), particulary in the Search Analysis.1- Total clicks2- Sum of clicks at every search query.I attached an image to make me clear.8IZsxs6.png
Reporting & Analytics | | NachoRetta2 -
Differences in site search revenue in GA
I just put in a piece of software to replace a really bad built in site search engine on my 3dcart website. Now I am trying to measure the change, but I am having some issues. When I check the ecom data in the conversions section of GA with the built in segment Performed Site Search, I get promising results. Approximately 5% revenue increase over LY. But if we jump to behavior, site search, usage, and then check the visits with site search, I get a decrease by 4%. And the actual revenue is off, by like double (150k compared to 80k) Anyone have any idea why I am getting these results? The site search function is set up. Tracking is enabled, query parameter is keyword and search url is /search.asp?keyword=
Reporting & Analytics | | ShockoeCommerce0 -
Changes to cookies, August 24 or 25? (organic search not being cookied?)
Hi folks. We've seen a precipitous change in the referral data that we're able to gather from cookies. Specifically: On August 24 or 25, traffic to our site that was cookied as coming from organic search dropped by ~25%, and traffic coming in with no referrer data at all (i.e., it appears to be "direct") rose by roughly the same amount. As far as we can see, we haven't changed anything in our systems that would have caused this (we're not just mis-reading the cookie info), so I'm looking for external reasons. Has anyone else seen this? Or have any ideas why it would happen?
Reporting & Analytics | | RobM4160 -
Improving Search Click through Rate
We are having a problem on our website with click through rates. We are getting between 100-150k impressions through search but we are only getting between 500-1000 clicks to the site. What strategies have you used in the past to help improve your click through rates? Thanks!
Reporting & Analytics | | pdangermond2 -
Differences in organic search visits and non-paid keyword visits
Hi folks, I was just wondering at the disparity between the "Organic Search Visits" total in the Traffic Data tab and the total visits from "Non Paid Keywords Sending Search Visits". Once you add up all the traffic generated from the individual keywords (including the not provided numbers) shouldn't the total match the number of organic search results? Or is there something I'm missing? At the moment the total visits from non-paid keywords is about 2,000 short of the total organic search visits.
Reporting & Analytics | | BrettCollins0 -
Since the 6th June there has been a serious drop in search impressions and clickthroughs in my GA + webmaster accounts.
Since the 6th June there has been a serious drop in organic search impressions and click throughs in my GA + webmaster accounts for my website. This leads me to believe my site has been punished for something? Where would be the best place to start diagnosing the issue causing the drop in search traffic? 2T9hz.jpg
Reporting & Analytics | | Scurrypoint0