Crawler doesn't discover the links in the main nav
-
Hi Moz Community,
We have a headless ecom (Magento) client that I'm trying to crawl the site. During the crawl, the tool (Screaming Frog) cannot discover the sub-category URLs in the main navigation when I start crawling via homepage.
Similarly, when I start crawling with one of the sub-category page, it doesn't crawl any of the product URLs on the sub-category page itself.
When I inspect product and sub-cat URLs through Search Console, they seem as indexed and if I view how Googlebot rendered the sub-category page, I can see the product URLs on the sub-cat page too.
If you have any idea what's the issue with Screaming Frog and would like to help me out, I'd be so grateful!
Thanks in advance
-
Hi Kate,
Thank you!
I followed you on Twitter, my user name is @curetuvana
-
Find me on twitter @katemorris and follow me. Tell me your name and I'll follow you and we can DM.
-
Hi Kate,
Thank you for your time to respond! Is there any way that I can contact you directly?
By the way, I've tried crawling after changing configurations as 'rendering Javascript' however, it still didn't discover product URLs.
Thank you!
-
Ah, I might know your problem. What is your site? We had this issue at my last company, had to do with crawling using JS. If you will send me the site, I can take a look.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My homepage doesn't seem to be indexed. Any suggestions?
As the title said, I don't think my homepage is being indexed. When I use "site:" search operator it's not there, but it's still ranking for other various keywords. Also the pages of my site I would expect to see with the "site:" search operator aren't there either. Site for reference: three29.com Any ideas what could be causing this? I don't have any errors or penalties in Search Console. Thanks.
Intermediate & Advanced SEO | | Three290 -
Redirect closed shop to main shop, or keep the domain and content alive and use it for link building?
Hello, We used to have two shops selling our products, a small shop with a small selection of only our best quality products (domain smallshop.com), and a big shop with everything (bigshop.com). It used to make sense (without going into full detail), but it's not relevant anymore, and so we decided to stop maintaining the small shop, because it was time consuming and not worth it. There is some really good links pointing to smallshop.com, and the content is original (the product descriptions are different between both shops). So far, we just switch the "add to cart" button on the small shop into a link to the same product on the big shop, and did links from the small shop to the big shop also on categories pages. So the question is: in your opinion, is it better to do that, keep the small shop and content alive and build links to our big shop, or do 301 redirections and shut down completely the small shop ? Thanks for your opinion!
Intermediate & Advanced SEO | | Colage0 -
Incorrect Spelling Indexed In Meta Info - Can't Change It
Hi,It would be great if a member of the community could help me to resolve this issue.Google is indexing an incorrect spelling on of our key pages and we can't identify the reason why.- The page in question: https://newbridgesilverware.com/jewelleryAs you can see from the attached image, the Meta Title is rendered to contain the keyword "jewelry" (the American spelling.) We want this to read as "jewellery" - the British-English spelling. Yet in the page source the word is given in the meta title as "jewellery". Nowhere in the page source or on the page itself does the American spelling appear - yet Google still renders it in the Meta Title.Can anyone identify why this is happening and offer any possible solutions?Much appreciatedDhqJp
Intermediate & Advanced SEO | | Johnny_AppleSeed1 -
Web developer won't 301 redirect to my new website....what can I do?
I want to come away from our third party web developer and use a new website with different web developers, however the web developer wont 301 redirect the old url's to the new ones. Is he required to do this by law? Is there away of doing the 301 redirects without him? Any advice would be much appreciated. Many thanks,Tom
Intermediate & Advanced SEO | | CoGri0 -
Should We Link To Our News?
We just started an "In the News" section on our webpage. We are not sure what would be the best for SEO purposes. Should we link to the news websites that have the stories about our company, even if they have no link bank? Or should we just take screenshots of the news article and only link to articles that link back to us (this is what we a currently doing)? Here is our news page, http://www.buyautoparts.com/News/
Intermediate & Advanced SEO | | joebuilder0 -
What's your Link Building Tactics?
So my question is: What's your Link Building Tactic. I always have a bit of a problem building links for my websites. Also Do you use some kind of a tool? If yes can you reccomend it?
Intermediate & Advanced SEO | | Angelos_Savvaidis0 -
Do links in the nav bar help SEO?
If I am building a Nav bar should I use my keywords or make it easier for the user to find what they are looking for. IMO one should ALWAYS make a site based on user experience. If it Google and other SEs do count Nav links, would it be best to place more important keys first?
Intermediate & Advanced SEO | | SEODinosaur0 -
Link Architecture - Xenu Link Sleuth Vs Manual Observation Confusion
Hi, I have been asked to complete some SEO contracting work for an e-commerce store. The Navigation looked a bit unclean so I decided to investigate it first. a) Manual Observation Within the catalogue view, I loaded up the page source and hit Ctrl-F and searched "href", turns out there's 750 odd links on this page, and most of the other sub catalogue and product pages also have about 750 links. Ouch! My SEO knowledge is telling me this is non-optimal. b) Link Sleuth I crawled the site with Xenu Link Sleuth and found 10,000+ pages. I exported into Open Calc and ran a pivot table to 'count' the number of pages per 'site level'. The results looked like this - Level Pages 0 1 1 42 2 860 3 3268 Now this looks more like a pyramid. I think is is because Link Sleuth can only read 1 'layer' of the Nav bar at a time - it doesnt 'hover' and read the rest of the nav bar (like what can be found by searching for "href" on the page source). Question: How are search spiders going to read the site? Like in (1) or in (2). Thankyou!
Intermediate & Advanced SEO | | DigitalLeaf0