Problems in indexing a website built with Magento
-
Hi all
My name is Riccardo and i work for a web marketing agency. Recently we're having some problem in indexing this website www.farmaermann.it which is based on Magento.
In particular considering google web master tools the website sitemap is ok (without any error) and correctly uploaded. However only 72 of 1.772 URL have been indexed; we sent the sitemap on google webmaster tools 8 days ago. We checked the structure of the robots.txt consulting several Magento guides and it looks well structured also.
In addition to this we noticed that some pages in google researches have different titles and they do not match the page title defined in Magento backend.To conclude we can not understand if this indexing problems are related to the website sitemap, robots.txt or something else.
Has anybody had the same kind of problems?Thank you all for your time and consideration
Riccardo
-
Hi Dan!
Thank you very much for your help and suggestions. I will try to follow your guidelines also.
Riccardo
-
Thank you Linda!
We will try and we will see what happens.
Riccardo
-
However, you should allow Google to crawl your JavaScript and CSS (which is now blocked). Here's some background info on that:
-
Hi Riccardo
Yes to confirm the site is indexed and crawlable. Checking the number of URLs from a sitemap that are indexed isn't the most reliable way to see if you content is indexed. You can do a site: search on your domain in Google like this as probably one of the most reliable ways. Also, you can try jus crawling the site with a tool like Screaming Frog SEO Spider - and if the tool can crawl everything, there may be just a delay on Google's end. But in your case now, all looks good!
-Dan
-
Hi Riccardo,
Since I do not know which pages exist on your site, I cannot be a 100% sure. You can remove this though from your robots.txt and see what happens (in Google Search Console & Bing Webmaster Tools).
Allow: /*?p=
Allow: /catalog/seo_sitemap/category/
Allow: /catalogsearch/result/Good luck!
-
Hi Linda!
Unfortunately we didn't develop the website but we have to work on its optimization. Probably you have right about the robots.txt because the sitemaps looks ok. I will try to remove the crawl delay. On the other hand which disallow rules should i remove or which modifies should i do in particular?
Thank you very much for your help!
Riccardo
-
Hi Josh!
Thank you very much for your help!
So probably there is a delay in webmaster tools data. Unfortunately we didn't develop the site but we only work on its optimization so we are a little bit confused with these data. -
Hi Ricardo,
Your home page is indexed.
It is most likely your problems are because of the robots.txt. -> http://www.farmaermann.it/robots.txt
1. You set a crawl delay of 10 seconds for all bots, which is quite long.
User-agent: *
Crawl-delay: 102. Some of your pages are not allowed to be crawled, like this one in your menu: http://www.farmaermann.it/integratori.html and http://www.farmaermann.it/contraccettivi-e-gravidanza.html
Allow: /*?p=
Allow: /catalog/seo_sitemap/category/
Allow: /catalogsearch/result/My advice is to modify your robots.txt: remove the crawl delay (and check whether your server can handle that) and make sure the pages in your menu can be crawled.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Single topic website or as part of a multiple topic website?
I have content sitting on a site here - https://www.pfizerpro.co.uk/product/xeljanz/rheumatoid-arthritis - domain authority 25 page authority 18 - the pages went live three months ago and the website was launched 18 months. We now have the option to use a brand new domain www.xeljanz.co.uk Which is the better option to stick with the www.pfizerpro.co.uk as it is a larger multiple topic site that should attract more links or to start a new single topic site which google may view as the better source as it is dedicated to the topic? Thanks
Intermediate & Advanced SEO | | Kate_team_DM0 -
Linking to External Websites?
Is it good to link external websites from every page. Since, the on-page grader shows there should be one link pointing to an external source. I have a website that can point to an external website from every page using the brand name of the specific site like deal sites do have. Is it worth having external link on every page, of-course with a no-follow tag?
Intermediate & Advanced SEO | | welcomecure0 -
Google indexed wrong pages of my website.
When I google site:www.ayurjeewan.com, after 8 pages, google shows Slider and shop pages. Which I don't want to be indexed. How can I get rid of these pages?
Intermediate & Advanced SEO | | bondhoward0 -
How many pages should be on landscapers website
Hi Guys, We have a good website strong onsite and offsite seo. A year ago, we had a 15 pages website for all main keywords we needed and we were on top 3 for most of these keywords in google. We were happy but we wanted more.. So we created lots of unique content targeting long tail keywords and created 100 more pages for the website. In next 4-5 months we lost positions for almost all our main keywords but got lots of longtails SERPs. Trafiic grew but the quality and the conversion rate shrinked. Everybody keep saying that it doesn't matter how many pages you have on the website as long as content is unique and I don't think it is true. I see lots of 3-5 paged websites without any seo in top 3 results in google. Does it mean that if I delete all these 100 pages that I created I will have more chances to get my main keywords SERP back? Basically does the seo juice that you have on domain is spreading across all pages and the more pages you have the less juice every page will get?
Intermediate & Advanced SEO | | vadimmarusin100 -
URL strategy mobile website
Hello everyone, We are facing a challenging decision about where our website (Flash Gaming website) is going. We are in the process of creating html5 games in the same theme of the flash games that we provide to our users. Now our main concern is to decide how to show this new content to the user? Shall we create brand new set of urls such as : http://www.mydomain.com/games/mobile/kids/ Or shall we adapt the main desktop url : http://www.mydomain.com/games/kids/ and show the users two different versions of the page depending on whether they are using a mobile device (so they see a mobile version) or a pc/laptop (so they a see desktop version). Or even redirect people to a sub-domain : http://m.mydomain.com/ The main idea we had is to keep the same url structure, as it seems that google is giving the same search results if you are using a mobile device or not. And creating a new set of urls or even a sub-domain, may involve a lot of work to get those new links to the same PA as the desktop URL that is here and know since a while now. Also the desktop page game should not be accessible to the mobile devices, so should this be redirected (301?) to the mobile homepage of the site? But how google will look at the fact that one url is giving 2 different contents, CSS etc, and also all those redirects might look strange... we are worried that doing so will hurt the page authority and its ranking ... but we are trying to find the best way to combine SEO and user experience. Any input on this will be really appreciated. Cheers,
Intermediate & Advanced SEO | | drimlike0 -
Linking Within Website
Hello - I have about 10 landing pages that I am focusing on ranking for and I'm doing okay. My question is should I have all these pages on a drop down menu from my home page or is the innerlinking too much? http://www.kasplacement.com
Intermediate & Advanced SEO | | ksundheim10 -
Splitting one Website into 2 Different New Websites with 301 redirects, help?
Here's the deal. My website stbands.com does fairly well. The only issue it is facing a long term branding crisis. It sells custom products and sporting goods. We decided that we want to make a sporting goods website for the retail stuff and then a custom site only focusing on the custom stuff. One website transformed and broken into 2 new ones, with two new brand names. The way we are thinking about doing this is doing a lot of 301 redirects, but what do we do with the homepage (stbands.com) and what is the best practice to make sure we don't lose traffic to the categories, etc.? Which new website do we 301 the homepage to? It's rough because for some keywords we rank 3 or 4 times on the first page. Scary times, but something must be done for the long term. Any advise is greatly appreciated. Thank you in advance. We are set for a busy next few months 🙂
Intermediate & Advanced SEO | | Hyrule0 -
Optimising My Website Link Containers
Hi, I'm looking at my links containers and trying to optimise them. I would be greatful if anyone can give me some feedback on my plan for perfect optimaisation. My links are constructed as follows: I have a two states:
Intermediate & Advanced SEO | | James77
1/. A Non Hover state which contains an Image and Text
2/. A Hover state which contains a bit more text - I do this as containing full text on the non hover state would not be good for users and would look ugly as well. Here's an example block of the HTML - as you can see from the URL, its quite a deep page level. From the URL and Alt / Titles the Page I am Linking to is about: "The Royal Hotel Accommodation New York Holidays". I Just a bit confused on how I should apply ALT and Title (Titles in particular) attributes given the nested DiV's etc - I can apply these to parent level, or apply all levels, or apply them to a mix. Also is there any obvious thinks you can think of I am missing that may help onsite SEO? Thanks in Advance CURRENT UNOPTIMISED CODE:
The Royal Hotel
New York Holidays Accommodation
The Royal Hotel
MY OPTIMISED CODE (Adding Title and Alt attributes):
The Royal Hotel
New York Holidays Accommodation
The Royal Hotel
0