Crawler issues
-
Can anyone please suggest why our site is not being crawled by Google at the moment?
Thanks,
-
No problem, happy to help
-
Thats a good point Adam. Thanks for your assistance.
-
Thanks Doug. That is really helpful.
-
Thanks for your help Chris
-
Yep. We did that. Thanks for the suggestion.
-
whoop there it is!
-
Double check your redirects again. When I tried to check your backlinks, I was showing redirects between tasteinc.com, www.tasteinc.com, and www.tasteinc.com/ Every time I tried to find the profile using one of those links, it would tell me that it redirects to another one of those.
I didn't have a problem going to your site, but the way that software was trying to crawl things, I would double check that again. That's the only issue that I saw. Has the site been up long?
-
Tried to crawl your site with Screaming Frog and noticed that your http response header contains a x-robots-tag which is being set to noindex, nofollow.
You can check your http headers on this site: http://web-sniffer.net/
-
I would agree with Mike you can see how here - https://support.google.com/sites/answer/100283?hl=en-GB
How long have you had the site up for? Remember it can take time for it to get crawled. You can ask google to crawl a page but this can still take time.
Good luck.
-
Hi Craig,
Did you submit your sitemap in Google Webmaster Tools?
Mike
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there an issue if we show our old mobile site to Google & new site to users
Hi, We have our existing mobile site that contains interlinking in footer & content and new mobile site that does not have interlinking. We will show existing mobile site to google crawler & new mobile site to users. Will this be taken as black hat by Google. The mobile site & desktop site will have same url across devices & browsers. Regards
Web Design | | vivekrathore0 -
Im having duplicate content issues in wordpress
all of my pages are working fine. but i added my sitemap to my footer in my website and when i click on my blog from my footer it takes me to the homepage. so now im having duplicate content for two diff urls. ive tried adding a rel=canonical and a 301 redirect to the blog page but it doesnt resolve the problem. also, when i go to my footer and click blog. after it brings me to the homepage ill try to click on my pages from the original bar at the top of my screen and it will bring me to the right pages. but it will have the same blog url in the search bar even when im on other pages. other than that all of my pages in my footer and in my homepage toolbar work fine. its just that one particular problem with the blog page in the footer and how it stays with the same blog url on every page after i click the blog in the footer. can someone please help. im using yoast and idk if i should disable it or what.
Web Design | | ClearVisionDesign0 -
International SEO issues for multiple sites
We currently have 3 websites: oursite.co.uk oursite.fr oursite.ch We also own Oursite.com, and that URL currently redirects to Oursite.fr. We are considering a complete site redesign and a possible merge of the 3 sites. Assumptions: ** the 3 sites currently receive organic search traffic to varying degrees
Web Design | | darkgreenguy
** Oursite.ch is almost identical to Oursite.fr in terms of the site content
** Our target market is NOT the USA for English-language searches. It is the UK. With a re-design, we see our options as follows: Merge the 3 sites and make Oursite.com the "main site" and then have subfolders as follows: /uk /fr /ch Keep the 3 sites as they are. We see Option 1 as the best in terms of saving time when updating the site, and saving money paid to the site developers (1 site vs 3 sites). We see Option 2 as the best in terms of ability of the site to rank, as well as confidence of searchers when seeing our site in the search results (in other words, a person searching in France would be more likely to buy and/or submit a form on our site if they saw Oursite.fr vs Oursite.com/fr). I guess we're looking for some suggestions/guidance here. Are we missing any big issues? Does anyone have experience with an issue such as this? Thank you in advance...
-Shawn0 -
Panda and Penquin Fall - Could HTML Design an Issue?
Hi, We were hit hard by Panda 3.4 on March 23rd 2012. Then Penguin came along and slapped us down a little farther on April 24th. White hat SEO for 13 years on the site. I have been trying to discover the reason we got hit so hard, to date 90% down. We ae wiped. I have a couple of keywords still #2 and #3 and we see up and down changes in Google webmaster tools, i.e. a keyword is supposedly up 50 points then another down 50. All other 150 keywords that we used to rank on the first page for are not even showing up. I have a person that is about to do a full link analysis but since we never went after links I just never had the feeling that is where our problem is at, but definitely going to explore it. The reason for my post is that last night I spoke with an SEO person that has some pretty good credentials (9 years experience and works currently at large online marketing company with seo with clients like Honda) and he was nice enough to just take a quick look at the site. He said he saw nothing really wrong and did not think that we were hit for any of the normal issues people are listing, i.e. duplicate content, backlinks. His first impression was that we were knocked down because the site is "hard to index". He said the site still uses tables and a lot of our Doc Statements were for HTML 4.01 from 1999. As we all know, there are 'many' experts in this industry. So I wanted a little feedback from the community. Our main site was built in Dreamweaver using tables. We do have a Wordpress blog that is very small and just now posting to add fresh content. (posts seem to rank pretty good, this is why I thought, you know he may be right) Would an older site be penalized like this for using tables? What would you do at this stage if you had a site that is not recovering? I have now reached panic mode and have to do something, just not sure of the next step. I will be happy to post the URL if anyone wants to help with advice. Thanks,
Web Design | | Force7
Force70 -
Wordpress drop down menu issue...
While I am tweaking my current site with getting rid of my footer links, my site will not allow a drop down menu for my service area parent page. It will do a drop down menu for any other tab. Could not find anything in a google search besides just link to parent for drop down... Any ideas?
Web Design | | greenjoe0 -
Best method to stop crawler access to extra Nav Menu
Our shop site has a 3 tier drop down mega-menu so it's easy to find your way to anything from anywhere. It contains about 150 links and probably 300 words of text. We also have a more context-driven single layer of sub-category navigation as well as breadcrumbs on our category pages. You can get to every product and category page without using the drop down mega-menu. Although the mega-menu is a helpful tool for customers, it means that every single page in our shop has an extra 150 links on it that go to stuff that isn't necessarily related or relevant to the page content. This means that when viewed from the context of a crawler, rather than a nice tree like crawling structure, we've got more of an unstructured mesh where everything is linked to everything else. I'd like to hide the mega-menu links from being picked up by a crawler, but what's the best way to do this? I can add a nofollow to all mega-menu links, but are the links still registered as page content even if they're not followed? It's a lot of text if nothing else. Another possibility we're considering is to set the mega-menu to only populate with links when it's main button is hovered over. So it's not part of the initial page load content at all. Or we could use a crude yet effective system we have used for some other menus we have of base encoding the content inline so it's not readable by a spider. What would you do and why? Thanks, James
Web Design | | DWJames0 -
Google search issue with exact domain
We had a site from Feb-2011 to Nov-2011 at the domain amcoexterminating.com. The site was pure HTML/CSS and the daily unique visitors steadily increased over that time. So all was fine. We then moved the site to a CMS (Joomla) on Dec. 6th. From that day forward, the daily visitors went into the tank. Before the move, if you typed "amcoexterminating.com" or "amco exterminating" into Google search, the site would be the first result (as you'd expect since those are the words that make up the actua domain). But we tried this yesterday and the site did not come up at all. NOT GOOD. It would work in Yahoo or Bing, but not in Google. So obviously, the problem with Google search directly affected the daily visitors. We just checked Webmaster tools yesterday (yes, this should have been done sooner, lesson learned) and it said "Site has severe health issues - Important page blocked by robots.txt". It listed the "important" page URL and it was just a link to an image. Regardless, I wiped out the Joomla created robots.txt file and added a new one and made it just say... User-agent: *Allow: / About 14 hours later, after the new robots.txt file was recognized by Google, the "severe health" message went away. However if I search in Google for "amcoexterminating.com", it still doesn't show up and the client is concerned (as they should be). Do you think the search engines just need more time to refresh? If so, once it refreshes, should the site show up first again right away? Or is it possible the robots.txt file had nothing to do with the issue? If so, what other things could I check into that might cause Google search to not find a site even if you search for exact domain name? Please share any and all things I should look into as I need to get this site showing in Google search again (as it was before moving to the CMS). Thanks!
Web Design | | MarathonMS0 -
Are my duplicate meta titles and descriptions an issue ?
HelloMy website http://www.gardenbeet.com has been rebuilt using prestacart and there are 158 duplicate title and meta descriptions being reported by google.My developer advised the following Almost all the duplicates are due to the same page being accessible at the root and following the category heading. e.g; /75-vegetable-patio-planter-turquoise.html
Web Design | | GardenBeet
/patio-planters/75-vegetable-patio-planter-turquoise.html This is hard-wired into PrestaShop. Was the Canonical module (now disabled) responsible for the confusion by not including the category name? The Googlebot shouldn't be scanning the root versions now. I don't believe this to be a serious issue but I'd recommend a second opinion from someone more SEO savvy just to be sure.Opinions??0