Having issues crawling a website
-
We looked to use the Screaming Frog Tool to crawl this website and get a list of all meta-titles from the site, however, it only resulted with the one result - the homepage.
We then sought to obtain a list of the URLs of the site by creating a sitemap using https://www.xml-sitemaps.com/. Once again however, we just go the one result - the homepage.
There is something that seems to be restricting these tools from crawling all pages. If you anyone can shed some light as to what this could be, we'd be most appreciative.
-
That robots.txt should be fine.. its not blocking anything.
The reason the crawl is stopping on the homepage is this code:
<meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">nofollow</a>">Which tells bots to not follow any links on the page. Remove that and you should be good.
-
Hi,
I think it is your robots.txt file that is causing the issue. At the moment you have the following:
**User-agent: ***
Disallow:
I would recommend updating it to the following:
**User-agent: ***
Allow: /
Moz also has a good post about what else you can include in your robots.txt file for best practices etc. :
https://moz.com/learn/seo/robotstxt
Hope that helps
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website Indexing Issues - Search Bots will only crawl Homepage of Website, Help!
Hello Moz World, I am stuck on a problem, and wanted to get some insight. When I attempt to use Screaming Spider or SEO Powersuite, the software is only crawling the homepage of my website. I have 17 pages associated with the main domain i.e. example.com/home, example.com/sevices, etc. I've done a bit of investigating, and I have found that my client's website does not have Robot.txt file or a site map. However, under Google Search Console, all of my client's website pages have been indexed. My questions, Why is my software not crawling all of the pages associated with the website? If I integrate a Robot.txt file & sitemap will that resolve the issue? Thanks ahead of time for all of the great responses. B/R Will H.
Intermediate & Advanced SEO | | MarketingChimp100 -
Strange rankings on new website
HI All My website is 10 years old, and has decent rankings. The domain is www.advanced-driving.co.uk I have recently had a major overhaul of the site, before it was very outdated, with lots of duplicated content. My main keywords are "advanced driving course" and "advanced driving courses" both of which I am on page 1. However, since I have been live with new site - (5 days) I am not ranking for some easy win keywords. I have submitted new content thought webmaster tools, and whilst some content is ranking, others are not. The content not ranking is fresh and unique ( have used copyscape on all new pages). For example my homepage is on page 1 for "advanced driving courses london" - around rank 6. So I hand made some content titled advanced driving courses london to provide more of an exact match, outlining our courses in London and the routes we take - http://www.advanced-driving.co.uk/defensive-advanced-driving-courses-london/ However, this page which is unique does not rank at all....I have done this with another website and it worked well, but google is not understanding this at all. Also I am now on page 1 for "advanced driving course" but not for "advanced driving courses" - well I am but the page for the plural keyword is a page not really related - surely Googles semantic search should realise course and courses are the same! I suspect that Google is still getting used to my new website? No errors or anything in Webmaster tools... Can anyone confirm this - or outline if I have done something awful..!! Thanks Rob
Intermediate & Advanced SEO | | robert780 -
60 countries, 1 website. How to develop this?
I've been asked to rank the website of an international association under the same keywords but in multiple languages (they rank very highly for English speaking countries), however they only have the one website with 1 .com domain. Question: Is the better approach to translate the site in multiple languages and then allow people to select the language they desire? OR Buy ccTLD and run the site multiple times in different languages as separate entities?
Intermediate & Advanced SEO | | MassivePrime0 -
Same website, seperate subfolders or separete websites? 12 stores in two cities
I have a situation where there are 12 stores in separate suburbs across two cities. Currently the chain store has one eCommerce website. So I could keep the one website with all the attendant link building benefits of one domain. I would keep a separate webpage for each store with address details to assist with some Local SEO. But (1) each store has slightly different inventory and (2) I would like to garner the (Local) SEO benefits of being in a searchers suburb. So I'm wondering if I should go down the subfolder route with each store having its own eCommerce store and blog eg example.com/suburb? This is sort of what Apple does (albeit with countries) and is used as a best practice for international SEO (according to a moz seminar I watched awhile back). Or I could go down the separate eCommerce website domain track? However I feel that is too much effort for not much extra return. Any thoughts? Thanks, Bruce.
Intermediate & Advanced SEO | | BruceMcG0 -
Merging two websites to one...
Hi all. Could do with a second opinion on this please... At present a client of ours owns two shops (both doing the same but in towns about 20 miles apart - they sell flooring, but using different names) and has a website for each. The plan is to rebrand both of these stores the same and merge both websites into one. The problem comes that both of the individual websites rank very well in their respective Google Local search results and I fear that killing one of the sites will mean that one store will vanish from the local listings. One domain is a DA 45 and the other a DA 11 so the plan is to use the stronger of the two domains. The question I would like to ponder with people wiser than myself is how can we ensure that the new single domain ranks for both locations in the local? Would the easiest solution be to have pages such as domain.com/store1 and domain.com/store2 with full listings for that store inc name, address, phone number, customer reviews etc? At present the DA 45 domain ranks very well in it's Google local so we need to find a way to change the homepage of that to have both the stores phone numbers but without affecting the local listing. I was considering adding the second phone number as a text based image so that it's visible for people but not for bots Finally, would 301 redirecting the now unused store to domain.com/store2 help with ensuring that we do not lose any local listing for that keyword? If not, are there any suggestions people could offer up Many thanks for any help and sorry for the very long question Carl
Intermediate & Advanced SEO | | GrumpyCarl0 -
Website architecture and sitewide link
Hello, I was reading this article about website architecture http://www.seomoz.org/blog/site-architecture-for-seo and I have a question about site wide link... I don't think site wide link are good if you want your homepage to have the most " juice " can someone confirm that to me. What I mean by site wide link is let's say I have a page about golf in califorina and I create a link to another page of my website which is about golf in florida that a good practicee to do that or not ? I just used this calculator http://www.webworkshop.net/pagerank_calculator.php and and if I do site wide links the juice doesn't go to my most important page ( the homepage ) which is the one I currently want to have the most juice. However, I noticed that website like amazon do site wide link let's say you are looking at a book and they have lots of links for other books or categories and they still rank very well for the word "textbooks" so I am kind of lost and don't understand why they still rank even though to me their juice goes away by doing site wide links to other books or categories ( even though it is related ) Can someone tell me more about site wide links are they good , bad etc... and is this calculator I use right or wrong ? Thank you
Intermediate & Advanced SEO | | seoanalytics0 -
Production and Priority Issue for SEO and Website Usability
I am a NOVICE .........My website is about 4 months old. My developer/programmer only has 4-6 hours of work a week so it is going to take 4 months to finish two weeks of work. So I have to prioritize the things that are best for SEO (Our architecture is PHP,Apache and Zend) .** If you are interested I would be curious to how you would prioritize some or all of these. Or at least as many as you can until you get bored.** 1. Optimizing Cart/Conversion - 7 hrs - (Extremely low conversion rates)
Intermediate & Advanced SEO | | Boodreaux
2. Optimizing Speed for usability -10+ hrs (Very slow on initial load time) 10-14 sec
3. Filling in all Titles and Metadata - 2 hrs
4. Contact persistence with cookie...enter data only once. - 2 hrs
5. Social panels for sharing content - 3 hrs
6. Custom notifications for those who opt in. for updates - 5 hrs
7. Shorten 12 key URL's and optimize with key words - 3 hrs (I rank this very high)
8. Install Wordpress Blog - 5-10 hrs
9. RSS Feed - 5 hrs ( Run a feed real time on side of page)
10. Create Content Management System for me - 20 hrs (So I can make changes)
11. Keywords for H-1 Tags - 1 hr
12. At tag for images - 1 hr
13. Use of bold /italics - 2 hrs
14. Canonical tag in head - 3 hrs Any expert advice will be greatly appreciated. Boodreaux PS After studying SEO for 1 month I think the priorities should be #7,#3, #2, #1, #5 (on landing pages) #11, #12,#6, #4, #13, #14, #8, #9, #100 -
Duplicate Content Issue
Why do URL with .html or index.php at the end are annoying to the search engine? I heard it can create some duplicate content but I have no idea why? Could someone explain me why is that so? Thank you
Intermediate & Advanced SEO | | Ideas-Money-Art0
