Crawler issues
-
Can anyone please suggest why our site is not being crawled by Google at the moment?
Thanks,
-
No problem, happy to help
-
Thats a good point Adam. Thanks for your assistance.
-
Thanks Doug. That is really helpful.
-
Thanks for your help Chris
-
Yep. We did that. Thanks for the suggestion.
-
whoop there it is!
-
Double check your redirects again. When I tried to check your backlinks, I was showing redirects between tasteinc.com, www.tasteinc.com, and www.tasteinc.com/ Every time I tried to find the profile using one of those links, it would tell me that it redirects to another one of those.
I didn't have a problem going to your site, but the way that software was trying to crawl things, I would double check that again. That's the only issue that I saw. Has the site been up long?
-
Tried to crawl your site with Screaming Frog and noticed that your http response header contains a x-robots-tag which is being set to noindex, nofollow.
You can check your http headers on this site: http://web-sniffer.net/
-
I would agree with Mike you can see how here - https://support.google.com/sites/answer/100283?hl=en-GB
How long have you had the site up for? Remember it can take time for it to get crawled. You can ask google to crawl a page but this can still take time.
Good luck.
-
Hi Craig,
Did you submit your sitemap in Google Webmaster Tools?
Mike
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Drupal Domain Access SEO Issues
I'm working with a new developer to redesign several Drupal sites and have 3-4 sites with similar designs and modules. The developer is keen on using Drupal Domain Access to make maintenance and sharing user information easier. Each site currently has a unique domain and content (although the sites are in related niches). Are there issues from an SEO perspective with the Drupal Domain Access Module? With only one instance of Drupal on the backend will Google somehow not view these as independent sites? Thanks for any info!
Web Design | | talltrees0 -
Duplicate Title Issues using # anchor tags
Our homepage navigation uses anchor tags (?TabNumb=1#, ?TabNumb=1# etc) rather than directly linking to different pages to decrease load time (and simplify the build process I owuld imagine). These anchor links are showing up as duplicate titles in Moz. I am pretty sure if I were to use noindex or rel tags, that could have a negative affect on my search results. Any way to tackle this outside of a complete redesign of the structure? http://www.dedoose.com/about-us/?TabNum=2# as an example
Web Design | | sbnjl0 -
Is there an issue if we show our old mobile site to Google & new site to users
Hi, We have our existing mobile site that contains interlinking in footer & content and new mobile site that does not have interlinking. We will show existing mobile site to google crawler & new mobile site to users. Will this be taken as black hat by Google. The mobile site & desktop site will have same url across devices & browsers. Regards
Web Design | | vivekrathore0 -
Redirects Not Working / Issue with Duplicate Page Titles
Hi all We are being penalised on Webmaster Tools and Crawl Diagnostics for duplicate page titles and I'm not sure how to fix it.We recently switched from HTTP to HTTPS, but when we first switched over, we accidentally set a permanent redirect from HTTPS to HTTP for a week or so(!).We now have a permanent redirect going the other way, HTTP to HTTPS, and we also have canonical tags in place to redirect to HTTPS.Unfortunately, it seems that because of this short time with the permanent redirect the wrong way round, Google is confused as sees our http and https sites as duplicate content.Is there any way to get Google to recognise this new (correct) permanent redirect and completely forget the old (incorrect) one?Any ideas welcome!
Web Design | | HireSpace0 -
Wordpress Custom Permalinks Plugin Issue
Hello, I installed this plugin and I'm having trouble making it work. I'm new to WordPress so was hoping someone can help me out. I'm looking to set up the following pages for example
Web Design | | ec1976
http://testsite.azurewebsites.net/Services/Services1
http://testsite.azurewebsites.net/Services/Services2 I do the following steps Add New Page Enter the following title "Services1" Enter the following text in the Permalink textbox "Services/Services1" replacing the default text "?page_id=140" Then I publish the page. This automatically updates the "Custom Fields" section below. The name is set to "custom_permalink" and the value is set to "Services/Services1". Then I "view page" but get a 404 error. The url is "http://testsite.azurewebsites.net/Services/Services1" Some additional info 1. WordPress version 9.8.3 Settings - Permalinks is set to "post name" Custom Permalinks version 0.7.18 Any help would be greatly appreciated. Thanks0 -
SEO Issues From Image Hotlinking?
I have a client who is hotlinking their images from one of their domains. I'm assuming the images were originally stored on the first domain (let's call it SiteA.com) and when they were putting together SiteB.com, they decided to just link to the images directly on SiteA.com instead of moving the images to Site B. Essentially hotlinking. Site A is not using the images in any way and in essence is just a gateway for their other sites and in this case a storage for their images. It doesn't use those images at all, so it really doesn't get any benefits of the images being referenced since I read that Google sometimes counts that hotlinking as a "vote" for the original image. But again, since ite A doesn't use the images that are being hotlinked at all, there's no benefit for Site A. My concern is that it's affecting their SEO for Site B because it makes it look like Site B is simply scraping data by hotlinking those images from Site A. Their programmer suggested creating a virtual directory so that it "looked" like it was coming from Site B. My guess is that Google can see this, so then not only will it look like Site B is scaping/hotlinking images, but also trying to hide it which may send up red flags to Google. My suggesstion to them was to just upload the images correctly into their own images directory on Site B. They own the images, so there's not any copyright issue, but that if they want proper SEO credit for that content, it all needs to be housed on the correct server and not hotlinked. Am I correct in this or will the virtual directory serve just as well?
Web Design | | GeorgiaSEOServices1 -
Panda and Penquin Fall - Could HTML Design an Issue?
Hi, We were hit hard by Panda 3.4 on March 23rd 2012. Then Penguin came along and slapped us down a little farther on April 24th. White hat SEO for 13 years on the site. I have been trying to discover the reason we got hit so hard, to date 90% down. We ae wiped. I have a couple of keywords still #2 and #3 and we see up and down changes in Google webmaster tools, i.e. a keyword is supposedly up 50 points then another down 50. All other 150 keywords that we used to rank on the first page for are not even showing up. I have a person that is about to do a full link analysis but since we never went after links I just never had the feeling that is where our problem is at, but definitely going to explore it. The reason for my post is that last night I spoke with an SEO person that has some pretty good credentials (9 years experience and works currently at large online marketing company with seo with clients like Honda) and he was nice enough to just take a quick look at the site. He said he saw nothing really wrong and did not think that we were hit for any of the normal issues people are listing, i.e. duplicate content, backlinks. His first impression was that we were knocked down because the site is "hard to index". He said the site still uses tables and a lot of our Doc Statements were for HTML 4.01 from 1999. As we all know, there are 'many' experts in this industry. So I wanted a little feedback from the community. Our main site was built in Dreamweaver using tables. We do have a Wordpress blog that is very small and just now posting to add fresh content. (posts seem to rank pretty good, this is why I thought, you know he may be right) Would an older site be penalized like this for using tables? What would you do at this stage if you had a site that is not recovering? I have now reached panic mode and have to do something, just not sure of the next step. I will be happy to post the URL if anyone wants to help with advice. Thanks,
Web Design | | Force7
Force70 -
Has Anyone Had Issues With ASP.NET 4.0 URL Routing?
I'm seeing some odd results in my SEOMOZ results with a new site I just released that is using the ASP.NET 4.0 URL routing. I am seeing thousands(!) of duplicate results, for instance, because the crawl has uncovered something like this: http://www.mysite.com/
Web Design | | TroyCarlson
http://www.mysite.com/default.aspx (so far, so good, though I wish it wouldn't show both)
http://www.mysite.com/default.aspx/about/ (what the heck -?)
http://www.mysite.com/default.aspx/about/about/ (WTF!?)
http://www.mysite.com/default.aspx/about/about/products/ (and on and on ad infinitum) I'm also seeing problems pop up in my sitemap because extensionless urls have an odd "eurl.axd/abunchofnumbersgohere" appended to the end of every address which is breaking links. sigh Buyer beware. I've found articles that discuss the "eurl.axd" issue here and there (this one seems very good), but nothing about the weird crawl issue I outlined above. Any advice?0