Website Indexing Issues - Search Bots will only crawl Homepage of Website, Help!
-
Hello Moz World,
I am stuck on a problem, and wanted to get some insight. When I attempt to use Screaming Spider or SEO Powersuite, the software is only crawling the homepage of my website. I have 17 pages associated with the main domain i.e. example.com/home, example.com/sevices, etc. I've done a bit of investigating, and I have found that my client's website does not have Robot.txt file or a site map. However, under Google Search Console, all of my client's website pages have been indexed.
My questions, Why is my software not crawling all of the pages associated with the website? If I integrate a Robot.txt file & sitemap will that resolve the issue?
Thanks ahead of time for all of the great responses.
B/R
Will H.
-
Hi Will,
It'd be impossible to find a solution to the problem without having the domain. If you want to message me the URL, I can take a quick look for ya.
A lack of either of those files shouldn't create any crawl issues. Setting up those files won't fix your problem, but you should add them both anyway.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Shopify Website Page Indexing issue
Hi, I am working on an eCommerce website on Shopify.
Intermediate & Advanced SEO | | Bhisshaun
When I tried Indexing my newly created service pages. The pages are not getting indexed on Google.
I also tried manual indexing of each page and submitted a sitemap but still, the issue doesn't seem to be resolved. Thanks0 -
Google Indexing
Hi We have roughly 8500 pages in our website. Google had indexed almost 6000 of them, but now suddenly I see that the pages indexed has gone to 45. Any possible explanations why this might be happening and what can be done for it. Thanks, Priyam
Intermediate & Advanced SEO | | kh-priyam0 -
Redirecting M Dot Mobile Website to Responsive Design Website Questions
Hi amazing Moz community 🙂 Couldn't find this question anywhere, and knew this was the place to ask! We are helping a client redirect an M Dot website to a Responsive Design website. We want to retain our mobile rankings for keywords. Three questions - We should use 301 redirects from the M Dot website to the new website correct? (not 302s?) How long does it take for Google to understand that we have launched a responsive website? Can we remove the 301 redirects after a few days (if the M Dot website interferes/breaks the new Responsive website)? We have verified an account on Google Search Console for the M Dot website, along with a mobile sitemap that has been submitted and verified. What should we do with this M Dot GSC account? Just delete it? Or keep it and upload the NEW XML Sitemap with the new WWW links (because the website is responsive). THANK YOU!
Intermediate & Advanced SEO | | accpar0 -
Crawled page count in Search console
Hi Guys, I'm working on a project (premium-hookahs.nl) where I stumble upon a situation I can’t address. Attached is a screenshot of the crawled pages in Search Console. History: Doing to technical difficulties this webshop didn’t always no index filterpages resulting in thousands of duplicated pages. In reality this webshops has less than 1000 individual pages. At this point we took the following steps to result this: Noindex filterpages. Exclude those filterspages in Search Console and robots.txt. Canonical the filterpages to the relevant categoriepages. This however didn’t result in Google crawling less pages. Although the implementation wasn’t always sound (technical problems during updates) I’m sure this setup has been the same for the last two weeks. Personally I expected a drop of crawled pages but they are still sky high. Can’t imagine Google visits this site 40 times a day. To complicate the situation: We’re running an experiment to gain positions on around 250 long term searches. A few filters will be indexed (size, color, number of hoses and flavors) and three of them can be combined. This results in around 250 extra pages. Meta titles, descriptions, h1 and texts are unique as well. Questions: - Excluding in robots.txt should result in Google not crawling those pages right? - Is this number of crawled pages normal for a website with around 1000 unique pages? - What am I missing? BxlESTT
Intermediate & Advanced SEO | | Bob_van_Biezen0 -
My site shows 503 error to Google bot, but can see the site fine. Not indexing in Google. Help
Hi, This site is not indexed on Google at all. http://www.thethreehorseshoespub.co.uk Looking into it, it seems to be giving a 503 error to the google bot. I can see the site I have checked source code Checked robots Did have a sitemap param. but removed it for testing GWMT is showing 'unreachable' if I submit a site map or fetch Any ideas on how to remove this error? Many thanks in advance
Intermediate & Advanced SEO | | SolveWebMedia0 -
Google indexed wrong pages of my website.
When I google site:www.ayurjeewan.com, after 8 pages, google shows Slider and shop pages. Which I don't want to be indexed. How can I get rid of these pages?
Intermediate & Advanced SEO | | bondhoward0 -
Bing not indexing website for some weird quality reason
Hi,I have a strange problem. My website www.dealwithautism.com is just 2 months old and have 40+ high quality articles that are already beginning to see some organic traffic from Google without any off page SEO (link building, etc). By quality articles I mean:
Intermediate & Advanced SEO | | DealWithAutism
1. Each article is 1500+ words of unique and highly relevant content with solid on page SEO (images may be reused from Google images). Moz page grader=A for most pages 2. Pretty well structured (with good number of internal links) 3. Entire site (all pages) delivered over https SSL using 301 redirect 4. No malware or spammy backlinks 5. NAP details and social signals available 6. Already ranking top10 in google SERPs for long tail KWs 7. According to Google Webmasters, no crawl errors except for a few (less than 10) 404s 8. Fully responsive - all pages tagged as "Mobile Friendly" by Google However, since day 1, Bing has not indexed a single page on my website (xml sitemap was updated from day 1) even though they are crawling the site. I recently raised an Email ticket and this was their response: "Upon checking, it appears that your site did not meet the standards set by Bing to get indexed the last time it was crawled. However, we will be looking further into this issue along with the Product Group to review the content of your website for re-evaluation. We currently do not have an ETA for the update but please be assured that we will get back to you as soon as they become available." Now based on my previous experience, this could take months. Following are just a few sample pages on the website: https://www.dealwithautism.com/oppositional-defiant-disorder-treatment-and-odd-case-study/ https://www.dealwithautism.com/tourette-syndrome-symptoms-treatment-for-tourettes/ https://www.dealwithautism.com/autism-test-for-toddlers/ I believe the quality of these pages are quite good for a small new website.
Then what does Bing mean by "website not meeting standards"? Am I missing a piece of the puzzle? I would have thought that Google was more quality focused than Bing but my SEO performance in Google is currently exceeding my expectation. Can you experts please help me out here?0 -
URL with a # but no ! being indexed
Given that it contains a #, how come Google is able to index this URL?: http://www.rtl.nl/xl/#/home It was my understanding that Google can't handle # properly unless it's paired with a ! (hash fragment / bang). site:http://www.rtl.nl/xl/#/home returns nothing, but: site:http://www.rtl.nl/xl returns http://www.rtl.nl/xl/#/home in the result set
Intermediate & Advanced SEO | | EdelmanDigital0