Crawler doesn't discover the links in the main nav
-
Hi Moz Community,
We have a headless ecom (Magento) client that I'm trying to crawl the site. During the crawl, the tool (Screaming Frog) cannot discover the sub-category URLs in the main navigation when I start crawling via homepage.
Similarly, when I start crawling with one of the sub-category page, it doesn't crawl any of the product URLs on the sub-category page itself.
When I inspect product and sub-cat URLs through Search Console, they seem as indexed and if I view how Googlebot rendered the sub-category page, I can see the product URLs on the sub-cat page too.
If you have any idea what's the issue with Screaming Frog and would like to help me out, I'd be so grateful!
Thanks in advance
-
Hi Kate,
Thank you!
I followed you on Twitter, my user name is @curetuvana -
Find me on twitter @katemorris and follow me. Tell me your name and I'll follow you and we can DM.
-
Hi Kate,
Thank you for your time to respond! Is there any way that I can contact you directly?
By the way, I've tried crawling after changing configurations as 'rendering Javascript' however, it still didn't discover product URLs.
Thank you!
-
Ah, I might know your problem. What is your site? We had this issue at my last company, had to do with crawling using JS. If you will send me the site, I can take a look.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can 'follow' rather than 'nofollow' links be damaging partner's SEO
Hey guys and happy Monday! We run a content rich website, 12+ years old, focused on travel in a specific region, and advertisers pay for banners/content etc alongside editorial. We have never used 'nofollow' website links as they're no explicitly paid for by clients, but a partner has asked us to make all links to them 'nofollow' as they have stated the way we currently link is damaging their SEO. Could this be true in any way? I'm only assuming it would adversely affect them if our website was peanalized by Google for 'selling links', which we're not. Perhaps they're just keen to follow best practice for fear of being seen to be buying links. FYI we now plan to change to more full use of 'nofollow', but I'm trying to work out what the client is refering to without seeming ill-informed on the subject! Thank you for any advice 🙂
Intermediate & Advanced SEO | | SEO_Jim0 -
Link Types For Link Building
Hi i have a SEO agency we work with who are building quality guest post links for us, however they are also building forum, profile, blog comments
Intermediate & Advanced SEO | | spyaccounts14
and directory based links. 60% of their links they are building are high quality, relevant guest posts while the other 40% are the other link types. The 40% seem to be relevant directories, forums, blog comments, etc. They said they build other link types because it diversifies the link building and profile rather then just building high quality guest posts. As just building one link type can leave a footprint. What are your thoughts on this? Cheers.0 -
Why isn't the canonical tag on my client's Magento site working?
The reason for this mights be obvious to the right observer, but somehow I'm not able to spot the reason why. The situation:
Intermediate & Advanced SEO | | Inevo
I'm doing an SEO-audit for a client. When I'm checking if the rel=canonical tag is in place correctly, it seems like it: view-source:http://quickplay.no/fotball-mal.html?limit=15) (line nr 15) Anyone seing something wrong with this canonical? When I perform a site:http://quickplay.no/ search, I find that there's many url's indexed that ought to have been picked up by the canonical-tag: (see picture) ..this for example view-source:http://quickplay.no/fotball-mal.html?limit=15 I really can't see why this page is getting indexed, when the canonical-tag is in place. Anybody who can? Sincerely 🙂 GMdWg0K0 -
Monthly Refreshes Aren't Actually Needed, Right?
We get tons of emails from Network Solutions with the following text: To ensure that your website is easily found online it is important that you submit your website to the major search engines and internet directories, including: | Google™ Google Places™ Google Mobile™ Bing™ Yahoo!<sup>®</sup> Twitter<sup>®</sup> | Facebook<sup>®</sup> CitySearch<sup>®</sup> Foursquare™ Angie's List<sup>®</sup> GPS navigation MerchantCircle<sup>®</sup> | To do so, we recommend you go to each search engine and internet directories web page, locate the instructions and then complete a monthly refresh of your listing. If you would like us to complete this process for you please call us at... Everything I've ever read about modern SEO says this isn't necessary and it's just a solicitation to get people to pay them for something they don't even need. We update our social pages regularly and maintain listings on many citation sites using Moz Local (in addition to manually building citations). Can you guys confirm that this is just more spam from Network Solutions?
Intermediate & Advanced SEO | | ScottImageWorks0 -
Submitted a Disavow BUT can't send in a RECONSIDERATION, WHY?
Hi Community! 2 weeks ago, i sent in our first/HUGE disavow list to Google. Out of the 2700 domains we submitted, 1300 of them we successfully removed, but we have nothing to show Google. Reason is because on our reconsideration request page, we can't submit anything because we didn't receive a message from Google (please see screenshot). I know for a FACT we got hit by an ALGORITHM penalty back in March2013. So, I have this wonderful Gdoc to prove that we worked LONG AND HARD to add and remove links in the past year, but we can't seem to message Google and tell them our story on why we should be reconsidered. How do we tell Google our success of removals? It's been 2 weeks, how much longer until we see a change in traffic? Or do we have to wait for the next update of algorithms by google aka REFRESH to see a change? Let me know and thank you so much in advance! Shawn cYGKLVR
Intermediate & Advanced SEO | | Shawn1241 -
Created the content, yet we don't rank for it. Toxic website?
Hey everyone, I'm beginning to think our site is toxic i.e. it'll never rank properly again irrespective of what we do. I recently published some data (2 months ago) in an interactive visual called the "iPhone 5S Price Index". I outreached and got thousands of links from sites including Forbes, Gizmodo (various international versions), Washington Post, The Guardian, NY Times, etc etc. All of these results dominate the Google rankings, all with links pointing to us. YET, we're no where to be seen. What incentive are Google giving content creators, like me, to continue producing content that is obviously popular if we can't even rank for it? The traffic we received was fantastic. In one day the traffic was 40 times our average, which made me smile like a Cheshire Cat from ear-to-ear but we need to improve our rankings overall otherwise the value to us is lost. The traffic wasn't there to buy our service, they were there to see the graphic. Hopefully our brand exposure leads to future sales, but it's a pittance compared to our previous rankings income. I've had this type of success 3 times in the last few months on this site alone. Yet nothing changes. We suffered from a loss of rankings in September 2012, fighting ever since to get it back. Now I'm losing hope it is even possible. Does anyone know why our site wouldn't rank when we're undeniable the source that created the work? Also, why wouldn't the increase in domain authority (which has jumped about 10 points according to OSE) have a knock on effect for the rest of our keywords - or even let us appear within the top 100 for ones we obviously serve? We do Real Company Shit - and we're good at it. But I need these rankings back. It's driving me nuts. Thanks.
Intermediate & Advanced SEO | | purpleindigo0 -
What to do when all products are one of a kind WYSIWYG and url's are continuously changing. Lots of 404's
Hey Guys, I'm working on a website with WYSIWYG one of a kind products and the url's are continuously changing. There are allot of duplicate page titles (56 currently) but that number is always changing too. Let me give you guys a little background on the website. The site sells different types of live coral. So there may be anywhere from 20 - 150 corals of the same species. Each coral is a unique size, color etc. When the coral gets sold the site owner trashes the product creating a new 404. Sometimes the url gets indexed, other times they don't since the corals get sold within hours/days. I was thinking of optimizing each product with a keyword and re-using the url by having the client update the picture and price but that still leaves allot more products than keywords. Here is an example of the corals with the same title http://austinaquafarms.com/product-category/acans/ Thanks for the help guys. I'm not really sure what to do.
Intermediate & Advanced SEO | | aronwp0 -
First Link Priority question - image/logo in header links to homepage
I have not found a clear answer to this particular aspect of the "first link priority" discussion, so wanted to ask here. Noble Samurai (makers of Market Samurai seo software) just posted a video discussing this topic and referencing specifically a use case example where when you disable all the css and view the page the way google sees it, many times companies use an image/logo in their header which links to their homepage. In my case, if you visit our site you can see the logo linking back to the homepage, which is present on every page within the site. When you disable the styling and view the site in a linear path, the logo is the first link. I'd love for our first link to our homepage include a primary keyword phrase anchor text. Noble Samurai (presumably seo experts) posted a video explaining this specifically http://www.noblesamurai.com/blog/market-samurai/website-optimization-first-link-priority-2306 and their suggested code implementations to "fix" it http://www.noblesamurai.com/first-link-priority-templates which use CSS and/or javascript to alter the way it is presented to the spiders. My web developer referred me to google's webmaster central: http://www.google.com/support/webmasters/bin/answer.py?answer=66353 where they seem to indicate that this would be attempting to hide text / links. Is this a good or bad thing to do?
Intermediate & Advanced SEO | | dcutt0