Welcome to the Q&A Forum

I am working in Magento 2 which has all sorts of difficulties.

My product page is example.com/testproduct the canonical is the same.

But in the sitemap it is example.com/testproduct/

In a perfect world I would get rid of the trailing slash but can't because of this issue- https://magento.stackexchange.com/questions/205337/unique-constraint-violation-found-when-remove-suffix-html-magento-2-2-0

The trailing slash will 301 redirect properly.

Is it an issue having the sitemap urls different with the trailing slash?

Meghan from Moz pointed out I had Disallow: /*? in the robots.txt and that appears to have been the problem. It is the default Magento 2 setting, so not sure why they had that in there.

I am stumped and Moz is being terrible to work with.

This site has about 40k pages

39,800 of them are item pages roughly.

Moz is only finding about 2400 of my pages. So they are missing most but not all of my item pages.

I do not know which item pages they are missing.

The fact that they are finding about 2k but not the rest leads me to believe the crawler is struggling with pagination.

The site is built on Magento 2 and uses the Amasty Layered Navigation extension.

Does anyone have any ideas?

Thanks.

That is what I figured just wanted to double check.

I have a Magento site and just realized we have about 800 review pages indexed. The /review directory is disallowed in robots.txt but the pages are still indexed.

From my understanding robots means it will not crawl the pages BUT if the pages are still indexed if they are linked from somewhere else.

I can add the noindex tag to the review pages but they wont be crawled.

https://www.seroundtable.com/google-do-not-use-noindex-in-robots-txt-20873.html

Should I remove the robots.txt and add the noindex? Or just add the noindex to what I already have?

Crawl stats are improving if anything. The last few weeks our pages crawled per day increased and our total indexed pages has steadily been increasing.

Nope. I didn't touch anything at all, that is why it is throwing me off.

I have about 20,000 items on my Magento Site. I used a plugin to add structured data. If I use the Webmaster Tools Structured Data Tester everything shows up perfect. There are no errors on any page that I have spot checked.

My total items increased to about 2500, but has now started dropping. The numbers have dropped to about 725 over the last few weeks. What can I check?

I installed this extension- https://www.magentocommerce.com/magento-connect/custom-product-urls-seo.html but could not figure out how to get it to work.

In case anyone else runs across this here is what the developer sent me and it worked-

"Second way to set url key using my extension is to tick products on product list in backend and use mass action "Set Custom URL". Then apply pattern to chosen products.
Then again if you wish to chcek created url rewrites go to Catalog > Url Rewrites and see urls containing category names."

I would like my product urls to be /category/manufacturer/name/part#. This would be the only url the item uses and how the product is accessed. It would also be used for product feeds.

My first attempt was to use https://amasty.com/magento-unique-product-url.html This creates a single url but I can not customize it. Sometimes it selects the manufacturer and sometimes the category.

My second attempt was with https://www.magentocommerce.com/magento-connect/custom-product-urls-seo.html I have it installed but it doesn't change the urls.

Has anyone been able to do this successfully?

The best way to crawl a site is Screaming Frog. That should give you the information you are looking for.

What are the pages that are showing duplicate?

Meghan from Moz pointed out I had Disallow: /*? in the robots.txt and that appears to have been the problem. It is the default Magento 2 setting, so not sure why they had that in there.

1. I do not have experience with a Hubspot blog, but I don't think a subdomain will hurt your efforts. I have always read that the best way to do it is in a subdirectory, but the difference between that and a subdomain isn't that big. You can setup a Wordpress blog pretty easy.

2. Google does know how to associate a redirect's link juice and domain authority to the page it redirects to. Gary Illyes said- all redirects pass PageRank now https://www.seroundtable.com/google-redirects-not-seo-22777.html

Looking at your site how sure are you that you need 70,000 pages?

For the sitemap I would stop trying to use a website and do it yourself. It looks like you are running IIS. They have a sitemap generator that you can install on a server easily and run it there. It looks like you have GoDaddy, they catch a lot of crap but I have always found their technical support to be top notch. If you can't figure out how to do it on the server I would give them a call.

Did you check to see if the links are follow or no follow? How old is the domain?

It ended up being my search results. I was able to use the site operator to break it down.

It might work, but eventually Google will catch on. If the site gets penalized for it, trying to get the ranking back up is a pain.

Most of the time this is because it is in the main menu or in the footer. Is it in either of those locations? If it is a blog post do you show recently added posts anywhere? Or related posts?