Crawler issues
-
Can anyone please suggest why our site is not being crawled by Google at the moment?
Thanks,
-
No problem, happy to help
-
Thats a good point Adam. Thanks for your assistance.
-
Thanks Doug. That is really helpful.
-
Thanks for your help Chris
-
Yep. We did that. Thanks for the suggestion.
-
whoop there it is!
-
Double check your redirects again. When I tried to check your backlinks, I was showing redirects between tasteinc.com, www.tasteinc.com, and www.tasteinc.com/ Every time I tried to find the profile using one of those links, it would tell me that it redirects to another one of those.
I didn't have a problem going to your site, but the way that software was trying to crawl things, I would double check that again. That's the only issue that I saw. Has the site been up long?
-
Tried to crawl your site with Screaming Frog and noticed that your http response header contains a x-robots-tag which is being set to noindex, nofollow.
You can check your http headers on this site: http://web-sniffer.net/
-
I would agree with Mike you can see how here - https://support.google.com/sites/answer/100283?hl=en-GB
How long have you had the site up for? Remember it can take time for it to get crawled. You can ask google to crawl a page but this can still take time.
Good luck.
-
Hi Craig,
Did you submit your sitemap in Google Webmaster Tools?
Mike
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website Server Issue?
I'm getting error messages that a website cannot be crawled and it might be due to the following issues: Couldn't access the webpage because the server either timed out or refused/closed the connection before our crawler could receive a response. How to fix: Please contact your web hosting technical support team and ask them to fix the issue Could Possibly Be:
Web Design | | PrimeMediaConsulting
1. DDoS protection system.
OR
2. Overloaded or misconfigured server They asked me to talk to my hosting company about this issue and he's at a loss (I don't think he knows everything he needs to know potentially). Have you seen these issues before? Where is the best spot to start troubleshooting this issue?0 -
Duplicate Title Issues using # anchor tags
Our homepage navigation uses anchor tags (?TabNumb=1#, ?TabNumb=1# etc) rather than directly linking to different pages to decrease load time (and simplify the build process I owuld imagine). These anchor links are showing up as duplicate titles in Moz. I am pretty sure if I were to use noindex or rel tags, that could have a negative affect on my search results. Any way to tackle this outside of a complete redesign of the structure? http://www.dedoose.com/about-us/?TabNum=2# as an example
Web Design | | sbnjl0 -
Question Concerning HTML5/CSS Templates & Google Mobility Issues
Hi all, Looking for some kind of solution for a responsive update for a site and I am wondering if there are any templates (not Wordpress) that are both great SEO wise and would also pass muster with the impending Google update for responsiveness? I was looking at things like Canvas and Porto ( http://themeforest.net/popular_item/by_category?category=site-templates ) but can't find any discussion on whether or not these things have been addressed with any of these templates. If any of you have suggestions or other places to look for something that could possibly fit the bill (even if temporarily) I would be very appreciative. Thank you so much in advance!
Web Design | | Pixelwik1 -
Confirm responsive design is accessible to crawlers?
Trying to ensure crawlers can access all important info on a new responsive website, but it's javascript-heavy. Following Google's advice/best practices, but can't get the Fetch and Render tool to confirm it's working. Anyone dealt with this before? Also posted in Product Forums with more detail: https://productforums.google.com/forum/#!msg/webmasters/PjxtBYClupw/56id80FUQdMJ
Web Design | | BoxerPropertyHouston0 -
Wordpress Custom Permalinks Plugin Issue
Hello, I installed this plugin and I'm having trouble making it work. I'm new to WordPress so was hoping someone can help me out. I'm looking to set up the following pages for example
Web Design | | ec1976
http://testsite.azurewebsites.net/Services/Services1
http://testsite.azurewebsites.net/Services/Services2 I do the following steps Add New Page Enter the following title "Services1" Enter the following text in the Permalink textbox "Services/Services1" replacing the default text "?page_id=140" Then I publish the page. This automatically updates the "Custom Fields" section below. The name is set to "custom_permalink" and the value is set to "Services/Services1". Then I "view page" but get a 404 error. The url is "http://testsite.azurewebsites.net/Services/Services1" Some additional info 1. WordPress version 9.8.3 Settings - Permalinks is set to "post name" Custom Permalinks version 0.7.18 Any help would be greatly appreciated. Thanks0 -
ECWID How to fix Duplicate page content and external link issue
I am working on a site that has a HUGE number of duplicate pages due to ECWID ecommerce platform. The site is built with Joomla! How can I rectify this situation? The pages also show up as "external " links on crawls... Is it the ECWID platform? I have never worked on a site that uses this. Here is an example of a page with the issue (there are 6280 issues) URL: http://www.metroboltmi.com/shop-spare-parts?Itemid=218&option=com_rokecwid&view=ecwid&ecwid_category_id=3560081
Web Design | | Atlanta-SMO0 -
Best method to stop crawler access to extra Nav Menu
Our shop site has a 3 tier drop down mega-menu so it's easy to find your way to anything from anywhere. It contains about 150 links and probably 300 words of text. We also have a more context-driven single layer of sub-category navigation as well as breadcrumbs on our category pages. You can get to every product and category page without using the drop down mega-menu. Although the mega-menu is a helpful tool for customers, it means that every single page in our shop has an extra 150 links on it that go to stuff that isn't necessarily related or relevant to the page content. This means that when viewed from the context of a crawler, rather than a nice tree like crawling structure, we've got more of an unstructured mesh where everything is linked to everything else. I'd like to hide the mega-menu links from being picked up by a crawler, but what's the best way to do this? I can add a nofollow to all mega-menu links, but are the links still registered as page content even if they're not followed? It's a lot of text if nothing else. Another possibility we're considering is to set the mega-menu to only populate with links when it's main button is hovered over. So it's not part of the initial page load content at all. Or we could use a crude yet effective system we have used for some other menus we have of base encoding the content inline so it's not readable by a spider. What would you do and why? Thanks, James
Web Design | | DWJames0 -
Facing design issue in IE6 and IE7
Hi, I implemented custom search code on my website which is on a test url http://61.246.241.232/. It is working perfectly fine on all the browsers except on IE6 and IE7. Can anyone please let me know the solution to this problem. Thanks & Regards
Web Design | | IM_Learner0