Crawler doesn't discover the links in the main nav
-
Hi Moz Community,
We have a headless ecom (Magento) client that I'm trying to crawl the site. During the crawl, the tool (Screaming Frog) cannot discover the sub-category URLs in the main navigation when I start crawling via homepage.
Similarly, when I start crawling with one of the sub-category page, it doesn't crawl any of the product URLs on the sub-category page itself.
When I inspect product and sub-cat URLs through Search Console, they seem as indexed and if I view how Googlebot rendered the sub-category page, I can see the product URLs on the sub-cat page too.
If you have any idea what's the issue with Screaming Frog and would like to help me out, I'd be so grateful!
Thanks in advance
-
Hi Kate,
Thank you!
I followed you on Twitter, my user name is @curetuvana -
Find me on twitter @katemorris and follow me. Tell me your name and I'll follow you and we can DM.
-
Hi Kate,
Thank you for your time to respond! Is there any way that I can contact you directly?
By the way, I've tried crawling after changing configurations as 'rendering Javascript' however, it still didn't discover product URLs.
Thank you!
-
Ah, I might know your problem. What is your site? We had this issue at my last company, had to do with crawling using JS. If you will send me the site, I can take a look.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My keywords aren't performing for my Umbraco site
A client of mine has just redesigned their site which was pretty small (homepage, about us page and contact us page) and now it includes homepage, about us page, 3 services pages, 5 blog posts and a contact us page. their domain authority is 5, so that gives you an idea of their size. We updated their key pages with keyword optimised content and added the keyword to their meta title and meta description. they're in the process of adding the alt tags and also they need to enable meta tags for the blog posts. Everything is quite in the process at the moment and their organic traffic is low. But I believe that some of the keywords should start moving places for the pages that have been optimised and they haven't. Is there any reason for this? I believe the services pages which have meta tags should have started ranking at least in very low position for the selected keywords. Is there something I'm missing? Thank you!
Intermediate & Advanced SEO | | Chris_Wright0 -
Change Google's version of Canonical link
Hi My website has millions of URLs and some of the URLs have duplicate versions. We did not set canonical all these years. Now we wanted to implement it and fix all the technical SEO issues. I wanted to consolidate and redirect all the variations of a URL to the highest pageview version and use that as the canonical because all of these variations have the same content. While doing this, I found in Google search console that Google has already selected another variation of URL as canonical and not the highest pageview version. My questions: I have millions of URLs for which I have to do 301 and set canonical. How can I find all the canonical URLs that Google has autoselected? Search Console has a daily quota of 100 or something. Is it possible to override Google's version of Canonical? Meaning, if I set a variation as Canonical and it is different than what Google has already selected, will it change overtime in Search Console? Should I just do a 301 to highest pageview variation of the URL and not set canonicals at all? This way the canonical that Google auto selected might get redirected to the highest pageview variation of the URL. Any advice or help would be greatly appreciated.
Intermediate & Advanced SEO | | SDCMarketing0 -
Site still indexed after request 'change of address' search console
Hello, A couple of weeks ago we requested a change of address in Search console. The new, correct url is already indexed. Yet when we search the old url (with site:www.) we find that the old url is still indexed. Is there another way to remove old urls?
Intermediate & Advanced SEO | | conversal0 -
Top hierarchy pages vs footer links vs header links
Hi All, We want to change some of the linking structure on our website. I think we are repeating some non-important pages at footer menu. So I want to move them as second hierarchy level pages and bring some important pages at footer menu. But I have confusion which pages will get more influence: Top menu or bottom menu or normal pages? What is the best place to link non-important pages; so the link juice will not get diluted by passing through these. And what is the right place for "keyword-pages" which must influence our rankings for such keywords? Again one thing to notice here is we cannot highlight pages which are created in keyword perspective in top menu. Thanks
Intermediate & Advanced SEO | | vtmoz0 -
International Targeting | Language > 'fa-ir' - no return tags
I see this error in search console :International Targeting | Language > 'fa-ir' - no return tagsURLs for your site and alternate URLs in 'fa-ir' that do not have return tags.and it is really increasingi do not know what is the problem and what I have done wrong? Originating URL Crawl date Alternate URL 1 /abadan/%D8%A2%D8%A8%D8%A7%D8%AF%D8%A7%D9%86/browse/vehicles/?place=8,541&v01=0,1&saveLoc=1
Intermediate & Advanced SEO | | divar
11/16/16
http://divar.ir/
0 -
301 redirects aren't passing value.
We recently migrated our shop to a new platform. We are using Wordpress for our main website, but we wanted a separate installation of Wordpress for our shop, so we left the main blog where it was, but moved the shop to a /shop/ sub directory with it's on WP installation. So now we have 2 installations of Wordpress. However, since we've done this, none of the pages on the new shop are ranking for anything. Their page rank is 0, and Moz page authority is 1 for every page on the new site. I've set up the proper 301 redirects, and they're redirecting fine, but none of the page value is coming over. It's been about a week now, and despite re-crawls by google, I'm not seeing any change. Also, one of the original (now re-directed) product pages still has a Page Authority of 13 according to Open Site Explorer. I know it's not high, but it had us ranking in the top 5 for a very important keyword, and now that value is being wasted. For example, one of our product pages that was ranking well was startupfashion.com/product/fashion-brand-line-sheet-template
Intermediate & Advanced SEO | | inkyj
That page is now redirected to
http://startupfashion.com/shop/product/fashion-line-sheet-template I've done 301's plenty of times and I've never seen this issue, so i'm wondering if it could have something to do with having multiple installations of Wordpress. I can't see any obvious issues with it... i have the Yoast SEO plugin configured properly on both installations, and all of the pages ARE being indexed by google. Not sure what is going on. Anyone have any experience with this, or have any ideas? Thanks!!0 -
Google can't access/crawl my site!
Hi I'm dealing with this problem for a few days. In fact i didn't realize it was this serious until today when i saw most of my site "de-indexed" and losing most of the rankings. [URL Errors: 1st photo] 8/21/14 there were only 42 errors but in 8/22/14 this number went to 272 and it just keeps going up. The site i'm talking about is gazetaexpress.com (media news, custom cms) with lot's of pages. After i did some research i came to the conclusion that the problem is to the firewall, who might have blocked google bots from accessing the site. But the server administrator is saying that this isn't true and no google bots have been blocked. Also when i go to WMT, and try to Fetch as Google the site, this is what i get: [Fetch as Google: 2nd photo] From more than 60 tries, 2-3 times it showed Complete (and this only to homepage, never to articles). What can be the problem? Can i get Google to crawl properly my site and is there a chance that i will lose my previous rankings? Thanks a lot
Intermediate & Advanced SEO | | granitgash
Granit FvhvDVR.png dKx3m1O.png0 -
Anyone managed to change 'At a glance:' in local search results
On Google's local search results, i.e when the 'Google places' data is displayed along with the map on the right hand side of the search results, there is also an element 'At a glance:'
Intermediate & Advanced SEO | | DeanAndrews
The data that if being displayed is from some years ago and the client would if possible like it to reflect there current services, which they have been providing for some five years. According to Google support here - http://support.google.com/maps/bin/answer.py?hl=en&answer=1344353 this cannot be changed, they say 'Can I edit a listing’s descriptive terms or suggest a new one?
No; the terms are not reviewed, curated, or edited. They come from an algorithm, and we do not help that algorithm figure it out. ' My question is has anyone successfully influenced this data and if so how.0