My site is not being regularly crawled?
-
My site used to be crawled regularly, but not anymore. My pages aren't showing up in the index months after they've been up. I've added them to the sitemap and everything. I now have to submit them through webmaster tools to get them to index. And then they don't really rank?
Before you go spouting off the standard SEO resolutions...
- Yes, I checked for crawl errors on Google Webmaster and no, there aren't any issues
- No, the pages are not noindex. These pages are index,follow
- No, the pages are not canonical
- No, the robots.txt does not block any of these pages
- No, there is nothing funky going on in my .htaccess. The pages load fine
- No, I don't have any URL parameters set
What else would be interfereing?
Here is one of the URLs that wasn't crawled for over a month: http://www.howlatthemoon.com/locations/location-st-louis
-
Google may have reduced their crawl budget for your site if they found enough pages to be low quality, thin, duplicate, etc... Here are some examples that you can probably apply a noindex tag to in order to reduce crawl budget waste on them.
Google these:
site:howlatthemoon.com/ inurl:tag
1,360 indexed pages that don't need to be in the SERPssite:howlatthemoon.com/ inurl:tag page
Even if you want the tag pages indexed, you don't need their paginated pages indexed toosite:howlatthemoon.com/ inurl:page inurl:category
A few dozen category pagination pages in the SERPs, many more on the site that have been booted from the index
Example:
http://www.howlatthemoon.com/dueling_piano_bar/category/nightlife/page/2/I would install the Yoast Wordpress SEO plugin, which should fix these in the following ways:
Use rel canonical tags
Use rel Next/Prev tagsNoindex tag pages
etc... -
That page is indexed. Can you give me an example of a page that hasn't been indexed yet? Your onsite SEO seems fine except you should get rid of your meta keywords tag and your Meta Description tag is a bit long.
-
oh sorry! Here is one of the pages that wasn't crawled since it was posted in early November.
-
You forgot one...what is the URL of your site
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Main Site and eCommerce Site URLs for SEO
My client currently has a main website on a url and an eCommerce site on a subdomain. The eCommerce site is currently not mobile friendly, has images that are too small and are problematic - and I believe it negates some of the SEO work we do for them. I had to turn off Google Shopping ads because the quality score was so low. That being said, they are rebuilding a shopping cart on a new platform that will be mobile friendly BUT the images are going to be tiny until they slowly replace images over several months. Would you keep the shopping cart on a subdomain, or make it part of the main website URL? Can it negatively impact the progress we have made on the main site SEO.
Technical SEO | | jerrico10 -
Similar pages on a site
Hi I think it was at BrightonSEO where PI DataMetrics were talking about similar pages on a website can cause rankings to drop for your main page. This has got me thinking. if we have a category about jumpers so: example.com/jumpers but then our blog has a category about jumpers, where we write all about jumpers etc which creates a category page example.com/blog/category/jumpers, so these blog category pages have no index put on them to stop them ranking in Google? Thanks in Advance for any tips. Andy
Technical SEO | | Andy-Halliday1 -
Changing a site from http to https
Will my rankings be affected if I change domain from http to https and force redirect?
Technical SEO | | Clickatell20 -
How does Google Crawl Multi-Regional Sites?
I've been reading up on this on Webmaster Tools but just wanted to see if anyone could explain it a bit better. I have a website which is going live soon which is going to be set up to redirect to a localised URL based on the IP address i.e. NZ IP ranges will go to .co.nz, Aus IP addresses would go to .com.au and then USA or other non-specified IP addresses will go to the .com address. There is a single CMS installation for the website. Does this impact the way in which Google is able to search the site? Will all domains be crawled or just one? Any help would be great - thanks!
Technical SEO | | lemonz0 -
What to do next with my site gamblingsites.co
So I have this site gamblingsites.co, which I launched about a year ago (I think.) This used to be internetgamblingsites.net (a domain I bought, but never managed to get in the index, and it appeared to violate the T/Cs after asking in GWMT) and before that the site used to be casinowarehouse.eu. After moving to gamblingsites.co, the pages were indexed almost instantly. I kept a 301 in place until today as I had some links pointing to internetgamblingsites.net. Now, until a few weeks ago, everything was fine. The site was ranking top 10 for gambling sites (8-10) and I had some traffic everyday. This site wasn't my top priority, so besides adding new unique content, I didn't do much with it. In each case no shady link building or what-so-ever. On February first of this year, however, it lost all of its rankings, and I have no idea why. Much worse site appear in the top 50, where a sub page of my site appears somewhere on the 9th SERP for keyword 'gambling sites.' Last week I started contacting some people and asked them to update my links. I also used my own sites (all on unique hosting accounts) to build some branded links, i.e. 'GamblingSites.co' and similar terms to down tune the exact match. I also decreased the instances of the exact match on the homepage, to avoid over optimization. Finally, I removed the 301 from internetgamblingsites.net, since the better links have been changed (or are about to get changed soon.) Now, couple of days later... no changes, but it's probably to early to judge. My question to you: "What would you do next, to try to save the site and at least get some traffic to it?" Thank you for your help, Giorgio PS: Feel free to ask for more information.
Technical SEO | | VisualSense0 -
Local business with multiple sites
I'm auditing a local business' sites (a spa) and I wanted to run my recommendations by everyone. There are 3 sites: www.sitename1.com -- main store location, used for Google Places listing #1 www.sitename2.com -- 2nd store location, used for Google Places listing #2 www.sitename3.com -- used for product sales for both locations Sitename1.com has the most ranking power. I'm going to recommend that they move sitename2.com and sitename3.com to sitename1.com as subfolders, 301 redirecting each page to the corresponding page on sitename1.com/subfolder. Google Places listing #2 would be changed from www.sitename2.com to www.sitename.com/location2. Any risks or problems with this strategy anyone can see?
Technical SEO | | 540SEO0 -
Are lots of links from an external site to non-existant pages on my site harmful?
Google Webmaster Tools is reporting a heck of a lot of 404s which are due to an external site linking incorrectly to my site. The site itself has scraped content from elsewhere and has created 100's of malformed URLs. Since it unlikely I will have any joy having these linked removed by the creator of the site, I'd like to know how much damage this could be doing, and if so, is there is anything I can do to minimise the impact? Thanks!
Technical SEO | | Nobody15569050351140 -
How to turn WP site into Ecom site?
I have a couple of old wordpress sites that are old affiliate blogs. I currently sell products that are on amazon on these sites and sell quite a bit of volume. I have found a source and can afford the inventory to replace Amazon with my own product. So the dilema is how to turn these wordpress sites into ecommerce sites. The thing I am worried most about is that each site gets about 100-200 visitors a day for great buying keywords. I obviously don't want to lose my rankings. What are the options of turning a wordpress site into a store. I am not interested in plugins or some of the other solutions that make the store look very cheap and I would assume horribly convert. If you have inner pages ranking for keywords how does that work? Do the post pages become product pages? So to sum up I guess I am asking, what are the options that are of the higher quality that will also help me keep my rankings? Thanks
Technical SEO | | PEnterprises0