Moz crawler is not able to crawl my website
-
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe -
@jasontorney Hi, I had the same problem after moving several of my websites to a Virtual Private Server that had enhanced security features.
One of these features was specifically to stop the Moz bot from crawling websites, and the hosting engineers advised they had done this because it was particularly aggressive in nature.
With my VPS control panel I have found the switch that allows me to disable bot blocking, and I occasionally do this if I'm grading a page with Moz, but advice from hosting support was to otherwise leave it active to protect the websites from attached (which means I don't get feedback from Moz crawls).
If you check with your hosting company you may find that they have a similar bot blocker configured for security purposes.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz is showing a canonical error that dont belong.
Hi guys, and thanks for this excellent source of information. i have an issue with the moz system because is telling to me that i dont have canonical instructions but i have canonical instructions on all my pages, so... im confused because maybe im not understanding what the system want to show to me. if you can help me i will be very gratefull. here you can see a page that have the canonical instruction. https://drive.google.com/file/d/14U_-Sgu_NQaB7kMBH3AguHQMHyHX9L8X/view?usp=sharing and here you can see what is reporting to me the MOZ system. https://drive.google.com/file/d/1pqgSC-V9WOyBPvQEr06pbqpLf_w7-q8J/view?usp=sharing this is happening on 19 pages, and all the 19 pages have the canonical instruction.
On-Page Optimization | | b-lab
thanks in advance guys.0 -
Migrating micro site into existing website
My company is planning to migrate an existing (ecommerce) micro site - which sits on its own domain - into their main ecommerce site. This means that the content will be moved from www.microdomain.co.uk to www.maindomain.com/category. Some products already exist on the main domain. The micro site is fairly small with just over 400 pages - I am planning to map each URL to the new URL (exact corresponding page) and create 301 redirects for each. Where any additional content does not exist yet on the existing main domain, we will create it and 301 redirect to it. The micro site currently ranks fairly well for some keywords - being such a specialised micro site, (some of) the keywords also form part of the domain name, however, they won't on the main page although they may form part of the URL (category). As an example (using a made up URL), our micro site www.bread-sticks.co.uk ranks on page 1 for the keyword bread sticks - we don't just sell bread sticks on www.bread-sticks.co.uk but also rolls and bread though, bread sticks is one category of very closely related categories. Say our main domain is www.supermarket.co.uk (selling a wide range of food / drink products. The micro site will be moving to www.supermarket.co.uk/baked-products/ - which is a category. Within that category, there are sub categories, i.e. bread sticks, rolls and bread which will sit under www.supermarket.co.uk/bread-sticks/ etc. What would be the best way for ensuring that our main domain would take over the rankings from our micro site, given that it will be sitting on our main domain as a category (one of many)? Can we expect www.supermarket.co.uk/baked-products/ or www.supermarket.co.uk/bread-sticks/ to replace www.bread-sticks.co.uk in the rankings simply by 301 redirecting? Thanks for your help!
Technical SEO | | ViviCa10 -
SEO for a a static content website
Hi everyone, We would like to ask suggestions on how to improve our SEO for our static content help website. With the release of each new version, our company releases a new "help" page, which is created by an authoring system. This is the latest page: http://kilgray.com/memoq/2015/help-en/ I have a couple of questions: 1- The page has an index with many links that open up new subpages with content for users. It is impossible to add title tags to this subpages, as everything is held together by the mother page. So it is really hard to for users to find these subpage information when they are doing a google search. 2- We have previous "help" pages which usually rank better in google search. They also have the same structure (1 page with big index and many subpages) and no metadata. We obviously want the last version to rank better, however, we are afraid exclude them from bots search because the new version is not easy to find. These are some of the previous pages: http://kilgray.com/memoq/2014R2/help-en/ http://kilgray.com/memoq/62/help-en/ I would really appreciate suggestions! Thanks
Technical SEO | | Kilgray0 -
Is my website is over optimized for ON page SEO?
The keyword for the page is “locksmith Logan” based in: Brisbane, Queensland, Australia Is webpage over used main keyword 'Logan locksmith' and what other areas are for improvement.
Technical SEO | | bondhoward0 -
Non Moving #1 Website
Hi Guys, Bit of an odd one we have recently undertaken our SEO in house (6-7 Months) and since doing this have had some great results, we have identified our competitors utilising Ahrefs, Moz and other tools that we have selected. However we have come up against something that we really cannot work out. One of our competitors ranks for everything related to there term on their website and industry at position #1 They are not actively doing SEO, Poor backlink structure from the research that we have carried out No fresh content being generate I was wondering if someone would perhaps mind spending 5 mins having a quick look to see what you think as i can't think what we might have missed. Tim.
Technical SEO | | fordy0 -
Big Mess - Multiple Websites
I have a customer, a Psychologist, who put up +/-20 websites many years ago. He has 1 main site (with his name as the domain) with hundreds of pages of quality content. The other sites are all exact match domains - anxiety counseling, couples counseling, etc. Some are single page sites, others have a good amount of quality content. Many of the EMD sites were getting ranked on the first page, as was the main site. The money site was ranking on the first page for the best keywords All of the EMD pages linked back to the main site, many with site wide footer links. The main site did not link back. All of the sites are on the same IP address. These sites have been in place for years. I don't believe that he has a duplicate content problem. About 8 weeks ago, the rankings for the main site crashed, moving 10 or more SERP pages deep. The EMD sites are still ranking. He has not gotten any nasty-grams from Google in Webmaster Tools. The Psychologist relies exclusively on organic for his business, and it has taken a significant hit. 1. Has anyone else seen this happen? 2. Is it safe to assume that Google finally nailed him for using a linking scheme? 3. How can we unwind this? The other sites are still generating business, and if those go away, he is really screwed. 4. Will taking down all of the links from the other sites be enough? Would moving the money site to another hosting company on a different IP make a difference? Ideas? I think the white hat answer would be to take down the EMD sites, and 301 redirect to the main site. The problem is that the loss of business from this process could be catastrophic.
Technical SEO | | CsmBill0 -
Crawling issues in google
Hi everyone, I think i have crawling issues with one of my sites. It has vanished form Google rankings it used to rank for all services i offered now it doesn't anymore ever since September 29th. I have resubmitted to Google 2 times and they came back with the same answer: " We reviewed your site and found no manual actions by the web spam team that might affect your site's ranking in Google. There's no need to file a reconsideration request for your site, because any ranking issues you may be experiencing are not related to a manual action taken by the webspam team. Of course, there may be other issues with your site that affect your site's ranking. Google's computers determine the order of our search results using a series of formulas known as algorithms. We make hundreds of changes to our search algorithms each year, and we employ more than 200 different signals when ranking pages. As our algorithms change and as the web (including your site) changes, some fluctuation in ranking can happen as we make updates to present the best results to our users. If you've experienced a change in ranking which you suspect may be more than a simple algorithm change, there are other things you may want to investigate as possible causes, such as a major change to your site's content, content management system, or server architecture. For example, a site may not rank well if your server stops serving pages to Googlebot, or if you've changed the URLs for a large portion of your site's pages. This article has a list of other potential reasons your site may not be doing well in search. " How i detected that it may be a crawling issue is that 2 weeks ago i changed metas - metas are very slow in getting updated and for some of my pages never did update Do you know any good tools to check for bad code that could slow down the crawling. I really don't know where to look other than issues for crawling. I validated the website with w3c validator and ran xenu and cleaned these up but my website is still down. Any ideas are appreciated.
Technical SEO | | CMTM0
