My website pages are not crawled, what to do?
-
Hi all.
I have made some changes on the website so i like to crawled them by the search engines Google especially.
I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites.
Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml.
Still my pages are not crawled your opinion please.
Thanks
-
this is somenting I would love to know more!-)
I think news and updates, maybe a blog in your site can make google "feel" it has to come back more times to refresh your site again ( "hum dude, they are doing something there)... Links to your site and to new articles in social media could help a lot.
If you find anything let us know!-)
-
SEO Martin,
all these 93 results are index successfully also shows in our Google webmaster account.
But i am worried that why crawler is not visit our website pages on regular basis as we submit sitemap.xml in webmaster account after the changes have made.
what is the reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
Hi Sha,
I just add this meta tag (as below) to help to crawl the page but not sure how it is.
I am just worried that if we make changes in our website pages and submit the site map in webmaster (that's mean that we are put request to search engine crawler to crawl the page ) but why they do not.
what is the actual reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
HI Martin,
Exactly, all url are indexed in search engines but they are not crawled after making a changes that is the main concern for me now.
For example:
Take a look on Magento Development page that showing last crawled 17 Mar 2013. I have made some changes and that should be crawled on a regular basis same as previous crawling.
Url :http://www.lucidsoftech.com/magento-development.html
your thoughts?
Thanks
-
Lucid,
I´ve just made a search on google and I found 93 results for your site, http://www.lucidsoftech.com/ , is that url right?
How many results do you actually own?
-
HI Sha,
yes site is indexed and crawled on regular basis before making these changes. don't know why not this time and not crawled around 2 weeks. yes i checked for thart all have including tag.
Normally when u submitted your pages on few good social bookmarking, workout in webmasters, after generate few good back links all these thing are very helpful for page crawling.
Still waiting for the page crawling.Thanks Sha
-
HI Martin,
Before making these changes they crawled regular basis and I just changed a meta title keywords and description but don't know the reason why after making these changes on few pages my web pages are not crawled in past 2 weeks and and still they are not crawled.
Martin i have my robots.txt in public.html and i allow the page that should be crawled by the bots.
waiting fro the page crawling.Thanks
-
Hi lucidsoftech,
The site is definitely indexed, but without knowing the specific URLs you are waiting on, it is a little difficult.
I ran a quick crawl with Screaming Frog and there are actually quite a few pages on the site which have the noindex meta tag in place - have you checked that the pages you are wanting indexed are not excluded with a tag?
Just a thought,
Sha
-
make sure you have a robots.txt on public folder. You must allow bots and disallow some folders.
Example of robots.txt of wordpress site
User-agent: *
Allow: /Disallow: /*?
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /tmpAllow: /wp-content/uploads/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a no-indexed parent page impact its child pages?
If I have a page* in WordPress that is set as private and is no-indexed with Yoast, will that negatively affect the visibility of other pages that are set as children of that first page? *The context is that I want to organize some of the pages on a business's WordPress site into silos/directories. For example, if the business was a home remodeling company, it'd be convenient to keep all the pages about bathrooms, kitchens, additions, basements, etc. bundled together under a "services" parent page (/services/kitchens/, /services/bathrooms/, etc.). The thing is that the child pages will all be directly accessible from the menus, so there doesn't need to be anything on the parent /services/ page itself. Another such parent page/directory/category might be used to keep different photo gallery pages together (/galleries/kitchen-photos/, /galleries/bathroom-photos/, etc.). So again, would it be safe for pages like /services/kitchens/ and /galleries/addition-photos/ if the /services/ and /galleries/ pages (but not /galleries/* or anything like that) are no-indexed? Thanks!
Technical SEO | | BrianAlpert781 -
Similar pages on a site
Hi I think it was at BrightonSEO where PI DataMetrics were talking about similar pages on a website can cause rankings to drop for your main page. This has got me thinking. if we have a category about jumpers so: example.com/jumpers but then our blog has a category about jumpers, where we write all about jumpers etc which creates a category page example.com/blog/category/jumpers, so these blog category pages have no index put on them to stop them ranking in Google? Thanks in Advance for any tips. Andy
Technical SEO | | Andy-Halliday1 -
Why are only a few of our pages being indexed
Recently rebuilt a site for an auctioneers, however it has a problem in that none of the lots and auctions are being indexed by Google on the new site, only the pages like About, FAQ, home, contact. Checking WMT shows that Google has crawled all the pages, and I've done a "Fetch as Google" on them and it loads up fine, so there's no crawling issues that is standing out. I've set the "URL Parameters" to no effect too. Also built a sitemap with all the lots in, pushed to Google which then crawled them all (massive spike in Crawl rate for a couple days), and still just indexing a handful of pages. Any clues to look into would be greatly appreciated. https://www.wilkinsons-auctioneers.co.uk/auctions/
Technical SEO | | Blue-shark0 -
Duplicate Page Title
Hi I just got back from first crawl report and there were plenty of errors. I know this has been asked before but I am newbie here so bear with me. I captured the video. Any ideas on how to address the issue? ktXKDxRttK
Technical SEO | | mcardenal0 -
After I 301 redirect duplicate pages to my rel=canonical page, do I need to add any tags or code to the non canonical pages?
I have many duplicate pages. Some pages have 2-3 duplicates. Most of which have Uppercase and Lowercase paths (generated by Microsoft IIS). Does this implementation of 301 and rel=canonical suffice? Or is there more I could do to optimize the passing of duplicate page link juice to the canonical. THANK YOU!
Technical SEO | | PFTools0 -
Googlebot take 5 times longer to crawl each page
Hello All From about mid September my GWMT has show that the average time to crawl a page on my site has shot up from an average of 130ms to an average of 700ms and peaks at 4000ms. I have checked my server error logs and found nothing there, I have checked with the hosting comapny and there are no issues with the server or other sites on the same server. Two weeks after this my ranking fell by about 950 places for most of my keywords etc.I am really just trying to eliminate this as a possible cause, of these ranking drops. Or was it the Pand/ EMD algo that has done it. Many Thanks Si
Technical SEO | | spes1230 -
I am trying to correct error report of duplicate page content. However I am unable to find in over 100 blogs the page which contains similar content to the page SEOmoz reported as having similar content is my only option to just dlete the blog page?
I am trying to correct duplicate content. However SEOmoz only reports and shows the page of duplicate content. I have 5 years worth of blogs and cannot find the duplicate page. Is my only option to just delete the page to improve my rankings. Brooke
Technical SEO | | wianno1680