Is URL appearance defined by crawling or by XML sitemap
-
I am having a problem developing a sitemap because I have long URLs that are made by zend. They go like this: http://myagingfolks.com/professionals/20661/social-workers/pennsylvania-civi-stanger
Because these URL's are long and are fed by Zend when I try to call them all up, to put on the sitemap, the system runs out of memory and crashes.
Do you know what part of a search result, in google, say, comes from the URL? Would it be fine for me to submit to google only www.myagingfolks.com/professionals/20661. Does the crawler find that the URL is indeed http://myagingfolks.com/professionals/20661/social-workers/pennsylvania-civi-stanger or does it go with just what the sitemap tells it?
-
Hi Joe,
THanks for the response. One thing: given that my URL structure gets everything beyond /professional/number/blah blah blah from Zend, does that automatically count as a 301 forward. Meaning, if I get the entire URL in the sitemap, will I still awaken the ire of the google-god?
thanks
-
Google is going to go to the pages submitted in the sitemap and see that they are serving a 301 response code, which they don't want to see in sitemaps. Either find a way to create a sitemap for the URLs you want to use (this is what I'd do) or shorten your URLs so they work with your sitemapping solution (although it is not a good idea to change URL structure because of a software limitation).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How long does google takes to crawl a single site ?
lately i have been thinking , when a crawler visits an already visited site or indexed site, whats the duration of its scanning?
Algorithm Updates | | Sam09schulz0 -
All keywords increasing rank except URL Keyword, whats going on?
Hello, Our website is a private equity firm database, privateequityfirms.com. We rank well for a number of private equity definitions and terms and have been increasing rank in those terms but unfortunately we have been losing ranking in our main keyword and url "private equity firms" .We have ranked as high as 3rd under wikipedia. The only real changes we have made are too the sitemap that is auto generated every time some thing is changed in the database. Does anyone have any ideas what is going on? I have included a Image to help show the problem. Thank you! MozAnalyticsPDF115_zpsddec64fa.png
Algorithm Updates | | Nicktaylor10 -
Reasons for a sharp decline in pages crawled
Hello! I have a site I've been tracking using Moz since July. The site is mainly stagnant with some on page content updates. Starting the first week of December, Moz crawler diagnostics showed that the number of pages crawled decreased from 300 to 100 in a week. So did the number of errors through. So crawler issues went from 275 to 50 and total pages crawled went from 190 to 125 in a week and this number has stayed the same for the last 5 weeks. Are the drops a red flag? Or is it ok since errors decreased also? Has anyone else experienced this and found an issue? FYI: sitemap exists and is submitted via webmaster tools. GWT shows no crawler errors nor blocked URLs.
Algorithm Updates | | Symmetri0 -
Does having a few URLs pointing to another url via 301 "create" duplicate content?
Hello! I have a few URLs all related to the same business sector. Can I point them all at my home domain or should I point them to different relevant content within it? Ioan
Algorithm Updates | | IoanSaid1 -
Bing's indexed pages vs pages appearing in results
Hi all We're trying to increase our efforts in ranking for our keywords on Bing, and I'm discovering a few unexpected challenges. Namely, Bing is reporting 16000+ pages have been crawled... yet a site:mywebsite.com search on Bing shows less than 1000 results. I'm aware that Duane Forrester has said they don't want to show everything, only the best. If that's the case, what factors must we consider most to encourage Bing's engine to display most if not all of the pages the crawl on my site? I have a few ideas of what may be turning Bing off so to speak (some duplicate content issues, 301 redirects due to URL structure updates), but if there's something in particular we should monitor and/or check, please let us know. We'd like to prioritize 🙂 Thanks!
Algorithm Updates | | brandonRT0 -
How could Google define "low quality experience merchants"?
Matt Cutts mentioned at SXSW that Google wants to take into consideration the quality of the experience ecommerce merchants provide and work this into how they rank in SERPs. Here's what he said if you missed it: "We have a potential launch later this year, maybe a little bit sooner, looking at the quality of merchants and whether we can do a better job on that, because we don’t want low quality experience merchants to be ranking in the search results.” My question; how exactly could Google decide if a merchant provides a low and high quality experience? I would image it would be very easy for Google to decide this with merchants in their Trusted Store program. I wonder what other data sets Google could realistically rely upon to make such a judgment. Any ideas or thoughts are appreciated.
Algorithm Updates | | BrianSaxon0 -
Do I nee 2 sitemaps?
Our ecommerce software produces a sitemap.html which is very large. We also use a sitemap.xml file for Google and other main search engines. Is there any point in maintaining the sitemap.html or should we hide it?
Algorithm Updates | | FFTCOUK0 -
Mobi sites and sitemaps
Hi all, How does should one treat mobi sites which have a separate set of files to the main site - with regards to the sitemap? Doe we tell Google about them?
Algorithm Updates | | gazza7770