Site less than 20 pages shows 1,400+ pages when crawled
-
Hello! I’m new to SEO, and have been soaking up as much as I can. I really love it, and feel like it could be a great fit for me – I love the challenge of figuring out the SEO puzzle, plus I have a copywriting/PR background, so I feel like that would be perfect for helping businesses get a great jump on their online competition.
In fact, I was so excited about my newfound love of SEO that I offered to help a friend who owns a small business on his site. Once I started, though, I found myself hopelessly confused.
The problem comes when I crawl the site. It was designed in Wordpress, and is really not very big (part of my goal in working with him was to help him get some great content added!)
Even though there are only 11 pages – and 6 posts – for the entire site, when I use Screaming Frog to crawl it, it sees HUNDREDS of pages. It stops at 500, because that is the limit for their free version. In the campaign I started here at SEOmoz, and it says over 1,400 pages have been crawled…with something like 900 errors.
Not good, right?
So I've been trying to figure out the problem...when I look closer in Screaming Frog, I can see that some things are being repeated over and over. If I sort by the Title, the URLs look like they’re stuck in a loop somehow - one line will have /blog/category/postname…the next line will have /blog/category/category/postname…and the next line will have /blog/category/category/category/postname…and so on, with another /category/ added each time.
So, with that, I have two questions
- Does anyone know what the problem is, and how to fix it?
- Do professional SEO people troubleshoot this kind of stuff all of the time? Is this the best place to get answers to questions like that? And if not, where is?
Thanks so much in advance for your help! I’ve enjoyed reading all of the posts that are available here so far, it seems like a really excellent and helpful community...I'm looking forward to the day when I can actually answer the questions!!
-
Thanks, Irving! I am trying turning on/off the plugins - the person who designed the site used a WP Boxer plugin and Multiple Content Blocks plugin, and that is how the homepage is designed (feeding info from pages/posts) so I was wondering if that could be part of it...but when I turn them off/on that doesn't seem to help. So I'm trying the other plugins too (there are just a couple), and if that doesn't work, I'll try a fresh install!
I also tried changing the permalink structure to just /sample-post/ and that didn't seem to work either...but I'm going to keep working on it!
I haven't tried the Twitter approach yet - because I don't actually have a Twitter account (I'm trying to keep social media from taking over my life) - but if that's where the answers are, I guess I need to get on there!
-
Did you install plugins that might have caused the issue? I would deactivate all plugins and see if it has an effect then turn them on one at a time to see if you can isolate the issue.
If the plugins are not the issue, it might make sense to backup the DB and do a fresh install of WP which isn't hard.
-
I don't think the site moved hosts - I'm not the person who created it, but his business is relatively new, so if there was a change it would have been done with very little content on the site.
The permalink structure is custom and looks like this: /blog/%year%/%monthnum%/%day%/%postname%/
Would something else be better? Let me know! Thanks!!
-
Hey K,
If you could post a screen shot of the Settings>Permalink structure screen in the Wordpress Dashboard, or just copy and paste whatever is written in there in a reply, that might help diagnose the issue. Also, do you know if the site has moved hosts recently and was re-installed using the Wordpress export & import feature?
-
Thanks, Alan! I'll try contacting those guys!
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will google be able to crawl all of the pages given that the pages displayed or the info on a page varies according to the city of a user?
So the website I am working for asks for a location before displaying the product pages. There are two cities with multiple warehouses. Based on the users' location, the product pages available in the warehouse serving only in that area are shown. If the user skips location, default warehouse-related product pages are shown. The APIs are all location-based.
Intermediate & Advanced SEO | | Airlift0 -
SEO Impact & Google Impact On Removing Product From Category Page for Ecommerce Site
Hello Experts, For my Ecommerce site previously I was showing products at category pages i.e. first all subcategories name after that list all products of all subcateogries. That also approx per category 500 products via load more feature. My query is now I am planning to show products only at Product Listing Page and not on Category pages so what will be SEO impact and how google will treat this? Thanks!
Intermediate & Advanced SEO | | Johny123450 -
After Server Migration - Crawling Gets slow and Dynamic Pages wherein Content changes are not getting Updated
Hello, I have just performed doing server migration 2 days back All's well with traffic moved to new servers But somehow - it seems that w.r.t previous host that on submitting a new article - it was getting indexed in minutes. Now even after submitting page for indexing - its taking bit of time in coming to Search Engines and some pages wherein content is daily updated - despite submitting for indexing - changes are not getting reflected Site name is - http://www.mycarhelpline.com Have checked in robots, meta tags, url structure - all remains well intact. No unknown errors reports through Google webmaster Could someone advise - is it normal - due to name server and ip address change and expect to correct it automatically or am i missing something Kindly advise in . Thanks
Intermediate & Advanced SEO | | Modi0 -
How to ask customers to +1 the page of the service/product they used and liked on Google+ of website
What format to use (how to write) to ask customers to +1 the page of the service/product they used and liked on Google+?
Intermediate & Advanced SEO | | MasonBaker0 -
Investigating Google's treatment of different pages on our site - canonicals, addresses, and more.
Hey all - I hesitate to ask this question, but have spent weeks trying to figure it out to no avail. We are a real estate company and many of our building pages do not show up for a given address. I first thought maybe google did not like us, but we show up well for certain keywords 3rd for Houston office space and dallas office space, etc. We have decent DA and inbound links, but for some reason we do not show up for addresses. An example, 44 Wall St or 44 Wall St office space, we are no where to be found. Our title and description should allow us to easily picked up, but after scrolling through 15 pages (with a ton of non relevant results), we do not show up. This happens quite a bit. I have checked we are being crawled by looking at 44 Wall St TheSquareFoot and checking the cause. We have individual listing pages (with the same titles and descriptions) inside the buildings, but use canonical tags to let google know that these are related and want the building pages to be dominant. I have worked though quite a few tests and can not come up with a reason. If we were just page 7 and never moved it would be one thing, but since we do not show up at all, it almost seems like google is punishing us. My hope is there is one thing that we are doing wrong that is easily fixed. I realize in an ideal world we would have shorter URLs and other nits and nats, but this feels like something that would help us go from page 3 to page 1, not prevent us from ranking at all. Any thoughts or helpful comments would be greatly appreciated. http://www.thesquarefoot.com/buildings/ny/new-york/10005/lower-manhattan/44-wall-st/44-wall-street We do show up one page 1 for this building - http://www.thesquarefoot.com/buildings/ny/new-york/10036/midtown/1501-broadway, but is the exception. I have tried investigating any differences, but am quite baffled.
Intermediate & Advanced SEO | | AtticusBerg10 -
Development site crawled
We just found out our password protected development site has been crawled. We are worried about duplicate content - what are the best steps to take to correct this beyond adding to robots.txt?
Intermediate & Advanced SEO | | EileenCleary0 -
How to generate xml sitemape for an ecommerce site with more than 50000 pages?
Hi, I am new to the forum and struggling hard to work on xml sitemap for an ecommerce site. Site is dynamic and more that 50,000 pages (including product pages). Challenges I am facing should I opt for category wise xml sitemap? how to include new product pages (dynamically) I was wondering if there is any tool that can generate xml site map online (I mean as soon as a new page is added to the site it will pick up automatically). thanks
Intermediate & Advanced SEO | | posy0 -
Negative impact on crawling after upload robots.txt file on HTTPS pages
I experienced negative impact on crawling after upload robots.txt file on HTTPS pages. You can find out both URLs as follow. Robots.txt File for HTTP: http://www.vistastores.com/robots.txt Robots.txt File for HTTPS: https://www.vistastores.com/robots.txt I have disallowed all crawlers for HTTPS pages with following syntax. User-agent: *
Intermediate & Advanced SEO | | CommercePundit
Disallow: / Does it matter for that? If I have done any thing wrong so give me more idea to fix this issue.0