Crawling a subfolder with a dev site
-
I am trying to set up a campaign where I am crawling a subfolder of our main site where I have dev version of the new site. However, even though the new site resolves and I have included the full resolving URL but the crawl results come back saying that only one page has been crawled. The site has had a protected block on it for a period of time but this has now been removed.
Any ideas?
Thanks
Nick
-
Here's a post with the common reasons why you get that error. https://seomoz.zendesk.com/entries/409821-why-isn-t-my-site-being-crawled-you-re-not-crawling-all-my-pages
If that doesn't solve the problem, contact the help team at help@seomoz.org.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?
Hi Everyone! My website is not being crawled regularly by Google - there are weeks when it's regular but for the past month or so it does not get crawled for seven to eight days. There are some specific pages, that I want to get ranked but they of late are not being crawled AT ALL unless I use the 'Fetch As Google' tool! That's not normal, right? I have checked and re-checked the on-page metrics for these pages (and the website as a whole, backlinking is a regular and ongoing process as well! Sitemap is in place too! Resubmitted it once too! This issue is detrimental to website traffic and rankings! Would really appreciate insights from you guys! Thanks a lot!
Technical SEO | | farhanm1 -
Site Wide Links
I have a link on pr 3 home page website placed in the side bar. It is on a WordPress website that spans a couple hundred pages and the side bar is on every page. The majority of the pages are not ranked or have any pr. Can this affect me negatively?
Technical SEO | | raph39880 -
Keyword targeting by page, site, or both?
Hi, We recently discovered that a product we sell has a misnomer, and that a ton of people take to Google and use variations of that misnomer while trying to find us. Unfortunately we don't rank in Google for this keyword, and its costing us thousands in lost sales. I've been slowly building the misnomer into the content of our site in hopes that the spiders will pick up on it. It has started to work in the last couple weeks, but we're nowhere near the top (and we are #1 and #2 for most of our other prime keywords.) The site which sells the product is specialized, and only sells this specific product (in different models, but they're all the same product essentially.) With that in mind, I'm trying to figure out the best way to attack a new keyword. I know that normally you would dedicate a specific page (in an eCommerce store probably that product's own page) to employ your SEO tactics. However, because this site specializes in this product and offers different models and information about it I'm confused about the best approach. Does Google take into consideration the entire site a s whole, or are the pages within my site competing against each other for rank?
Technical SEO | | ninjaprecision0 -
One large site or a few microsites?
Hi, I have a client who runs a professional expo company and wants to redo his website. Right now he has one website that has the following sections: expo company info, wedding show info, electrical wire show info, fishing show info. My question is, when I rebuild the site would it be better to do one site or four microsites?
Technical SEO | | JohnWeb120 -
What to include on a sitemap for a huge site?
I have a very large site and I'm not sure what all to include on the sitemap page. We have categories such as items1, items2 and in the items1 category are 100 vendors with their individual vendor pages. Should I link all 100 vendor pages on the sitemap or just the main items1 category?
Technical SEO | | CFSSEO0 -
I am Posting an article on my site and another site has asked to use the same article - Is this a duplicate content issue with google if i am the creator of the content and will it penalize our sites - or one more than the other??
I operate an ecommerce site for outdoor gear and was invited to guest post on a popular blog (not my site) for a trip i had been on. I wrote the aritcle for them and i also will post this same article on my website. Is this a dup content problem with google? and or the other site? Any Help. Also if i wanted to post this same article to 1 or 2 other blogs as long as they link back to me as the author of the article
Technical SEO | | isle_surf0 -
Subfolder to Root Domain, Should i or not?
Dear All, I am kinda of stuck and need your helps, I have major client's site which is located in subdirectory. Instance: http://www.majorclient.com.cn/cn/ and as you can see domain name is already localized and they are not planning to make any english version of the site, So there will be no /eng/ folder but clients whole website is /cn/ subdirectory which is totally unnecessary and i would like to 301 whole site to **http://www.majorclient.com.cn ** But the problem is that this site has been /cn/ folder forever since the beginning and lot of trusted links are pointing to www.majorclient.com.cn/cn/ So should i move it or just configure 301 www.majorclient.com.cn to www.majorclient.com.cn/cn/ leave it there and don't bother. Help?
Technical SEO | | DigitalJungle0 -
Site Relaunch
Hello, I recently launched my new site (Nov. 25, 2011) but still have the old site live because I still need old customer data from the old admin for customer service issues and I cannot delete the old front-end without deleting the old back-end!. I am seeing a lot of referrals coming from the old site IP address with many backlinks to the new site but dont know if this is actually hurting the new site due to duplicate content, ect .. Any input would be greatly aaaaaapreciated 😉 Thanks in advance, Byron-
Technical SEO | | k9byron0