Welcome to the Q&A Forum

I am working in Magento 2 which has all sorts of difficulties.

My product page is example.com/testproduct the canonical is the same.

But in the sitemap it is example.com/testproduct/

In a perfect world I would get rid of the trailing slash but can't because of this issue- https://magento.stackexchange.com/questions/205337/unique-constraint-violation-found-when-remove-suffix-html-magento-2-2-0

The trailing slash will 301 redirect properly.

Is it an issue having the sitemap urls different with the trailing slash?

Meghan from Moz pointed out I had Disallow: /*? in the robots.txt and that appears to have been the problem. It is the default Magento 2 setting, so not sure why they had that in there.

I am stumped and Moz is being terrible to work with.

This site has about 40k pages

39,800 of them are item pages roughly.

Moz is only finding about 2400 of my pages. So they are missing most but not all of my item pages.

I do not know which item pages they are missing.

The fact that they are finding about 2k but not the rest leads me to believe the crawler is struggling with pagination.

The site is built on Magento 2 and uses the Amasty Layered Navigation extension.

Does anyone have any ideas?

Thanks.

That is what I figured just wanted to double check.

I have a Magento site and just realized we have about 800 review pages indexed. The /review directory is disallowed in robots.txt but the pages are still indexed.

From my understanding robots means it will not crawl the pages BUT if the pages are still indexed if they are linked from somewhere else.

I can add the noindex tag to the review pages but they wont be crawled.

https://www.seroundtable.com/google-do-not-use-noindex-in-robots-txt-20873.html

Should I remove the robots.txt and add the noindex? Or just add the noindex to what I already have?

Crawl stats are improving if anything. The last few weeks our pages crawled per day increased and our total indexed pages has steadily been increasing.

Nope. I didn't touch anything at all, that is why it is throwing me off.

I have about 20,000 items on my Magento Site. I used a plugin to add structured data. If I use the Webmaster Tools Structured Data Tester everything shows up perfect. There are no errors on any page that I have spot checked.

My total items increased to about 2500, but has now started dropping. The numbers have dropped to about 725 over the last few weeks. What can I check?

I installed this extension- https://www.magentocommerce.com/magento-connect/custom-product-urls-seo.html but could not figure out how to get it to work.

In case anyone else runs across this here is what the developer sent me and it worked-

"Second way to set url key using my extension is to tick products on product list in backend and use mass action "Set Custom URL". Then apply pattern to chosen products.
Then again if you wish to chcek created url rewrites go to Catalog > Url Rewrites and see urls containing category names."

I would like my product urls to be /category/manufacturer/name/part#. This would be the only url the item uses and how the product is accessed. It would also be used for product feeds.

My first attempt was to use https://amasty.com/magento-unique-product-url.html This creates a single url but I can not customize it. Sometimes it selects the manufacturer and sometimes the category.

My second attempt was with https://www.magentocommerce.com/magento-connect/custom-product-urls-seo.html I have it installed but it doesn't change the urls.

Has anyone been able to do this successfully?

It ended up being my search results. I was able to use the site operator to break it down.

1. Your first one is interesting. I actually haven't been in there before. There are 96 rows and everyone of them is set to let Googlebot Decide. Do you think I should change that up?

2. Not sure on how many images we have but it is a lot. Not we do not have an image sitemap.

I tried Screaming Frog and it couldn't handle it. After about 1.5 million urls it kept locking up. I just setup a free trial for Deep Crawl. It can only do 10,000 but I will see if it has anything worthwhile.

I wouldn't say it is doing fine. Before I started they launched a new site and messed up the 301 redirects. Traffic hasn't recovered yet.

For Robots I am using the Inchoo robots.txt-http://inchoo.net/ecommerce/ultimate-magento-robots-txt-file-examples/ maybe it is a parameters issue, but I can't figure out how to see all my indexed pages.

I tried doing a search for both inurl:= site:www.site.com and inurl:? site:www.site.com and nothing showed up unless I am missing something.

I can't figure out how to check if some of the canonicalized urls are indexed. The pages are all identical though.

We have less then 100 out of stock items.

Stats-

Webmaster Tools Indexed Pages- 96,995

Site: Search- 97,800 Pages

Sitemap Submitted- 18,832

Sitemap Indexed- 9,746

I went through the search results through page 28 and every item it showed was correct. How do I figure out where these extra 80,000 items are coming from? I tried crawling the site with screaming frog awhile back but it locked because of so many urls. The site is a Magento site so there are a million urls, but I checked and all of the canonicals are setup properly. Where should I start looking?

Looking at your site how sure are you that you need 70,000 pages?

For the sitemap I would stop trying to use a website and do it yourself. It looks like you are running IIS. They have a sitemap generator that you can install on a server easily and run it there. It looks like you have GoDaddy, they catch a lot of crap but I have always found their technical support to be top notch. If you can't figure out how to do it on the server I would give them a call.

1. Yes your domain gets credit for the backlinks.

2. If they change in the future and just have everything on Medium and not your subdomain you would lose the backlinks. I don't see that happening though.

Did you check to see if the links are follow or no follow? How old is the domain?

Most of the time this is because it is in the main menu or in the footer. Is it in either of those locations? If it is a blog post do you show recently added posts anywhere? Or related posts?

What are the pages that are showing duplicate?

Best I could find was-

Unlike disallowed pages, noindexed pages don’t end up in the index and therefore won’t show in search results. Combine both in robots.txt to optimise your crawl efficiency: the noindex will stop the page showing in search results, and the disallow will stop it being crawled

From-https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/

I have had fluctuations like this in the past. Have you seen a drop in organic traffic? Have you looked at your latest links in the Google Search Console to see if there is anything spammy?

I can't tell you which one to focus on, but I would say be careful about spam score. I have a link from a site that has a high spam score, but I know the site is legit. They do several hundred million dollars per year and have been around decades, but for whatever reason Moz thinks they are spam.

I would render the content on the original page if possible. Google can't rank what it can't see.

The best way to crawl a site is Screaming Frog. That should give you the information you are looking for.

It might work, but eventually Google will catch on. If the site gets penalized for it, trying to get the ranking back up is a pain.

1. I do not have experience with a Hubspot blog, but I don't think a subdomain will hurt your efforts. I have always read that the best way to do it is in a subdirectory, but the difference between that and a subdomain isn't that big. You can setup a Wordpress blog pretty easy.

2. Google does know how to associate a redirect's link juice and domain authority to the page it redirects to. Gary Illyes said- all redirects pass PageRank now https://www.seroundtable.com/google-redirects-not-seo-22777.html

I have previously switched sites to https but this one is behaving a little different. On September 19th I switched to https. I did 301 redirects at the .htaccess, added it to search console, and since we are using Magento I changed the base url.

In the past when I have done this the http site index just gradually drops while the https site gradually rises. In early October the http site started to slightly drop but since 10/23 there have been no changes. For the https site it started to increase up until 10/23 then stayed flat.

Why have they stayed stuck like that for a month?

This page is showing up as a 404 in Google Search console- https://www.wolfautomation.com/blog/autonics/ It shows it has been linked from these pages-

• https://www.wolfautomation.com/blog/raffel/
• https://www.wolfautomation.com/blog/new-temperature-controllers-from-autonics/
• https://www.wolfautomation.com/blog/ge-industrial/
• https://www.wolfautomation.com/blog/temp-controller/
• https://www.wolfautomation.com/blog/tx4s/

I never created this page, I don't want this page but it keeps showing up. The problem is the link isn't found on those pages anywhere so I can't delete it. What am I missing? How can I get rid of it?

I have a Magento website. I am going to add a Wordpress blog under /blog. If I setup each with its own webmaster tools to submit a sitemap does it hurt anything?

Which I am ok with, but why am I getting duplicate content?

I figured out what to do. The ID Path can not be the same as the request path. So if anyone else runs into this that is how I fixed it.

It has been used correctly. The site is a Magento site and they have it built in. There are a lot of filters for products so it uses rel=canonical to tell Google which to index.

So I made a mistake it isn't the robots.txt that is the issue. I am getting hit with a ton of duplicate content penalties so I figured that was it. The problem is that I have pages with rel=canonical tags that it is ignoring. Does Roger not read those?

It is crawling the entire site, and there is stuff we do not want it to. Please advise.

I need to do about 6000 redirects for a Magento site. The pages no longer exist. I have tried the URL rewrite module but it isn't working for me and I don't want to do 6000 redirects in the htaccess files. Any suggestions?

It was the menu.

I couldn't recreate that what size did you get it down to?

I am getting the error in search console, I can't find any problems in any of my tests either.

I am getting that error on my product pages. This link is in the errors http://www.wolfautomation.com/drive-accessory-safety-sto-module-i500 but when I look at it on mobile it is fine.