My website pages are not crawled, what to do?
-
Hi all.
I have made some changes on the website so i like to crawled them by the search engines Google especially.
I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites.
Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml.
Still my pages are not crawled your opinion please.
Thanks
-
this is somenting I would love to know more!-)
I think news and updates, maybe a blog in your site can make google "feel" it has to come back more times to refresh your site again ( "hum dude, they are doing something there)... Links to your site and to new articles in social media could help a lot.
If you find anything let us know!-)
-
SEO Martin,
all these 93 results are index successfully also shows in our Google webmaster account.
But i am worried that why crawler is not visit our website pages on regular basis as we submit sitemap.xml in webmaster account after the changes have made.
what is the reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
Hi Sha,
I just add this meta tag (as below) to help to crawl the page but not sure how it is.
I am just worried that if we make changes in our website pages and submit the site map in webmaster (that's mean that we are put request to search engine crawler to crawl the page ) but why they do not.
what is the actual reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
HI Martin,
Exactly, all url are indexed in search engines but they are not crawled after making a changes that is the main concern for me now.
For example:
Take a look on Magento Development page that showing last crawled 17 Mar 2013. I have made some changes and that should be crawled on a regular basis same as previous crawling.
Url :http://www.lucidsoftech.com/magento-development.html
your thoughts?
Thanks
-
Lucid,
I´ve just made a search on google and I found 93 results for your site, http://www.lucidsoftech.com/ , is that url right?
How many results do you actually own?
-
HI Sha,
yes site is indexed and crawled on regular basis before making these changes. don't know why not this time and not crawled around 2 weeks. yes i checked for thart all have including tag.
Normally when u submitted your pages on few good social bookmarking, workout in webmasters, after generate few good back links all these thing are very helpful for page crawling.
Still waiting for the page crawling.Thanks Sha
-
HI Martin,
Before making these changes they crawled regular basis and I just changed a meta title keywords and description but don't know the reason why after making these changes on few pages my web pages are not crawled in past 2 weeks and and still they are not crawled.
Martin i have my robots.txt in public.html and i allow the page that should be crawled by the bots.
waiting fro the page crawling.Thanks
-
Hi lucidsoftech,
The site is definitely indexed, but without knowing the specific URLs you are waiting on, it is a little difficult.
I ran a quick crawl with Screaming Frog and there are actually quite a few pages on the site which have the noindex meta tag in place - have you checked that the pages you are wanting indexed are not excluded with a tag?
Just a thought,
Sha
-
make sure you have a robots.txt on public folder. You must allow bots and disallow some folders.
Example of robots.txt of wordpress site
User-agent: *
Allow: /Disallow: /*?
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /tmpAllow: /wp-content/uploads/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to move some pages of my website to a folder and nav menu in those pages should only show inner page links, will it hurt SEO?
Hi, My website has a few SaaS products, to make my website simple i want to move my website some pages to its specific folder structure , so eg website.com/product1/features
Technical SEO | | webbeemoz
website.com/product1/pricing
website.com/product1/information and same for product2 and so on, the website.com/product1/.. menu will only show the links of product1 and only one link to homepage (possibly in footer). Please share your opinion will it be a good idea, from UI perspective it will be simple , but i am not sure about SEO perspective, please help thanks0 -
First Crawl Report
Just joined SEOMoz today and am slightly overwhelmed, but excited about learning loads from it. I've just received my Crawl Report and there is a
Technical SEO | | iainmoran
404 : UserPreemptionError:
http://www.iainmoran.com/comments/feed/ This is a WordPress site and I've no idea what the best course of action to take. I've done some searching on Google and a couple of sites suggest removing that url from within the robots.txt file. I'm using the Yoast Plugin which apparently creates a robots.txt file, but I can't see any way to edit it. Is there another solution for resolving the 404 error? Many thanks, Iain.0 -
Why is the Page Authority of my product pages so low?
My domain authority is 35 (homepage Page Authority = 45) and my website has been up for years: www.rainchainsdirect.com Most random pages on my site (like this one) have a Page Authority of around 20. However, as a whole, the individual pages of my products rank exceptionally low. Like these: http://www.rainchainsdirect.com/products/copper-channel-link-rain-chain (Page Authority = 1) http://www.rainchainsdirect.com/collections/todays-deals/products/contempo-chain (Page Authority = 1) I was thinking that for whatever reason they have such low authority, that it may explain why these pages rank lower in google for specific searches using my exact product name (in other words, other sites that are piggybacking of my unique products are ranking higher for my product in a specific name search than the original product itself on my site) In any event, I'm trying to get some perspective on why these pages remain with the same non-existent Page Authority. Can anyone help to shed some light on why and what can be done about it? Thanks!
Technical SEO | | csblev0 -
Blog Ranking NOT home page main website?!
Hi, Our Blog (http://blog.thailand-investigation.com) is ranking for some of our major keywords but not our home page (http://www.thailand-investigation.com)!? Our blog is WordPress and our main website is HTML. It seems like the search engines consider that they are 2 separate websites!? When I check the incoming links to our website, I get also the blog links!!!??? Is it normal? Do I have to build a relation of some kind or write some code saying that it is our Blog... I don't know! I'm not a SEO specialist or even a webmaster. I'm a small business owner and take care on my website. I created by myself but never learned! So, please help! Thanks
Technical SEO | | MichelMauquoi0 -
How do I resolve Twin domains? redirect website.com to www.website.com?
I am new to this website. Tried to run a campain and got a warning that website.com resolves to www.website.com which hinders SERP by competing for Keyword indexing!. (website is my domain name) Would appreciate help with this. Thanks. S.H. PS: here is the exact wording of error : We have detected that the domain www.yfvaccine.com and the domain yfvaccine.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here.
Technical SEO | | sherohass0 -
How can I prevent duplicate content between www.page.com/ and www.page.com
SEOMoz's recent crawl showed me that I had an error for duplicate content and duplicate page titles. This is a problem because it found the same page twice because of a '/' on the end of one url. e.g. www.page.com/ vs. www.page.com My question is do I need to be concerned about this. And is there anything I should put in my htaccess file to prevent this happening. Thanks!
Technical SEO | | onlineexpression
Karl0 -
Website hacked
Hi I've been asked to help a colleague with his website. It seems to be hacked. He recently received an e-mail from Google saying his adwords account was suspended 'due to high probability his site may be hosting or distributing malicious software' I just checked his source and there seems to loads of weird on code on his pages, this would not have been but on by any members of the website owners. Please image attached when we try to access his website via google search I just contacted the hosting provider - does anyone have experience with this and how to prevent such hacking in the future. The site is build using HTML with no CMS. IjW19.jpg
Technical SEO | | Socialdude0 -
New Domain Page 7 Google but Page 1 Bing & Yahoo
Hi just wondered what other people's experience is with a new domain. Basically have a client with a domain registered end of May this year, so less than 3 months old! The site ranks for his keyword choice (not very competitive), which is in the domain name. For me I'm not at all surprised with Google's low ranking after such a short period but quite surprsied to see it ranking page 1 on Bing and Yahoo. No seo work has been done yet and there are no inbound links. Anyone else have experience of this? Should I be surprised or is that normal in the other two search engines? Thanks in advance Trevor
Technical SEO | | TrevorJones0