How do I exclude my blog subfolder from being crawled with my main domain (www.) folder?
-
I am trying to setup two separate campaigns for my blog and for my main site.
While it is easy enough to do through the wizard the results I am getting for my main site still include pages that are in my blog sub folder.
Please Advise!
-
Hi,
I would like to know if this idea, of excluding a subfolder from a campaign, is available now? If so, how do I set it up?
We have a forum in a subfolder and all the data from it kind of blurs my main interest using Seomoz for this website.
-
Hey,
now with the new Moz Analytics platform, has this been implemented yet or still on the plan? I would like to exclude a forum placed in a subfolder as this forum is not our main focus to work with SEO on. It has so many URLs and "eats" up the moz crawl budget for us.
Thanks!
-
Hey Henry,
Thanks for writing in. Unfortunately, there isn't a way to exclude the blog from the crawl for the main site at this time. If you exclude our crawler from those pages, then we wouldn't be able to crawl for the separate blog campaign either. We do plan to allow you to ignore or remove certain pages from your crawl when we switch to the new Moz Analytics platform, but it may be some time before that is available for all of our users. I'm sorry for the inconvenience that causes in the meantime.
Please let me know if you have any other questions.
Chiaryn
Help Team Ninja -
i was just thinking, you might be able to put a robots file in the subfolder telling seomoz bot not to crawl that folder, not sure if that's possible. But make sure you only target seomozs bot only (don't want to tell google not to crawl it).
but that would stop the subfolder being crawled completely, which I don't think you want
-
Yes I am referring to the seomoz bot and I understand that google sees the two sub folders as one domain.
I would like them to be separate just for my own work flow/ability to mange each individually with no spill over.
But if it can't be done it can't be done.
Thank you!
-
I'm guessing you mean the seomoz, bot crawling your site? If so why do you want to crawl them separately, because google won't, they are on the same domain.
to answer your question, I don't think it can be done
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
WEbsite cannot be crawled
I have received the following message from MOZ on a few of our websites now Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. I have spoken with our webmaster and they have advised the below: The Robots.txt file is definitely there on all pages and Google is able to crawl for these files. Moz however is having some difficulty with finding the files when there is a particular redirect in place. For example, the page currently redirects from threecounties.co.uk/ to https://www.threecounties.co.uk/ and when this happens, the Moz crawler cannot find the robots.txt on the first URL and this generates the reports you have been receiving. From what I understand, this is a flaw with the Moz software and not something that we could fix form our end. _Going forward, something we could do is remove these rewrite rules to www., but these are useful redirects and removing them would likely have SEO implications. _ Has anyone else had this issue and is there anything we can do to rectify, or should we leave as is?
Moz Pro | | threecounties0 -
Before Migration/after(www/non-www/http/https) - Good concentration needed :p
Hi all, Im confusing between those www's and http's. If i go to searchbar (chrome) and ENTER: www.mywebsite.nl, It changes to https://www.mywebsite.nl
Moz Pro | | Dreamgame2016
( with www, and https:// not used) / Its OK next: typing in searchbar and enter: mywebsite.nl, It changes to https://mywebsite.nl (without www and https:// ) / OK Next: www.mywebsite.nl, it stay the same, just https:// added: https://mywebsite.nl (used with https://) / OK Now its comes: If I do it again without http**(s)://mywebsite.nl, **It changes to https://www.mywebsite.nl/?SID=bccbuhvi1cf53r188bpvskn597 / NOT OK 😛 In google search console (webmastertool) I gave property for the https://mywebsite.nl and https://www.mywebsite.nl Each of the website, Im seeying data clicks/ volume keywords etc, so both of them functionating By search console: https://www.mywebsite.nl (With www) I see crawlfaults/errors: 1633 (the url has not linked existing page) I see again: "?SID=..." after urls, example: mywebsite.nl/blabla/?SID=m07ev6lliefbf0tfhe4kf0ih54 By search console - other website: https://mywebsite.nl **(none-www) **you see two crawlfaults/errors! Bad influance for my SEO, because of no existed pages, bad urls and dubble content. Bye bye keywords! Lets analyze/crawl with Moz tool ofcourse ^^: Pages with High Priority Issues: | 2646 | Duplicate Page Content |
| 14 | 4XX Client Error |
| 3 | Crawl Attempt Error |
| 1 | Title Missing or Empty | Medium priority: | 9618 | Temporary Redirect |
| 2688 | Duplicate Page Title |
| 13 | Title Element is Too Long |
| 1 | Missing Meta Description Tag | After seeying this results what is the best option (no losing link-juice)? redirect 301? www to none-www (https://) ? Shortly I am going to change my domain provider and the website template in magento. After that I am going to focus on the SEO implementation. First, I have to solve this problem. Who can give me an advice for this situation? Regarding, Newbee0 -
Why my Domain Authority (DA) is Decreased from 21 to 19?
Hello, I would like to know how the changes in domain authority is counted by MOZ? Domain Authority for my this domain http://www.grandcrayon.ca/ was 21 and it is decreased to 19 in last week. I have checked the moz report and it shows DA as 19. We already have grandcrayon.ca and we recently redirected our .com domain (http://www.grandcrayon.com also to .ca so all the back links of .com are now showing in webmaster for .ca. So i would like to know the reason behind decrement in DA and is there any connection with redirection of .com domain? How can i increase DA for my website?
Moz Pro | | CommercePundit0 -
Crawl Diagnostics - Canonical Question
On one of my sites I have 61 notices for Rel Canonical. Is it bad to have these or is this just something that's informative?
Moz Pro | | kadesmith0 -
Can we add sites to the crawl queue for OSE?
Is it possible to request that Open Site Explorer crawls a new URL on its next run? This tool is the first place I go to when working on a new site, and when there is "No Data Available" this is a little frustrating. I fully appreciate that this lack of data is usually a signal that the website is either very new or of low quality, however that if often the reason that I am brought in and would very much like to benchmark and provide initial analysis using this tool. It would make sense that OSE crawls the sites that Moz members are working on wouldnt it? Scott.
Moz Pro | | eseyo0 -
Crawler has stopped crawling my website
Hi all, I recently changed my site www.ipbskinning.com to ipbskinning.com. For some reason my weekly scan states that no pages have been crawled. I tried making a new campaign with the sub-domain ipbskinning.com but still the starter crawl returns nothing? Am I doing something wrong? :S Thanks all!
Moz Pro | | pezza34340 -
Settings to crawl entire site
Not sure what happened but I started a third campaign yesterday and only 1 pages was crawled, The other two campaigns has 472 and 10K respectively. What is the proper setting to choose in the beginning of campaign setup to have the entire site crawled. Not sure what I did different and I must be reading the instructions incorrectly. Thanks, Don
Moz Pro | | NicheGuy210 -
How does SEOMoz crawl sites? Does it follow the sitemap?
I only ask because in my report it returned a few "no follow" links. Was wondering how SEOMoz operates.
Moz Pro | | Pikimal0