Are the CSV downloads malformatted, when a comma appears in a URL?
-
Howdy folks, we've been a PRO member for about 24 hours now and I have to say we're loving it! One problem I am having with however is a CSV exported from our crawl diagnostics summary that I've downloaded.
The CSV contains all the data fine, however I am having problems with it when a URL contains a comma. I am making a little tool to work with the CSVs we download and I can't parse it properly because there sometimes URLs contain commas and aren't quoted the same as other fields, such as meta_description_tag, are.
Is there something simple I'm missing or is it something that can be fixed?
Looking forward to learn more about the various tools. Thanks for the help.
-
I won't be too hard on the programmers - I'm a programmer myself. Our small business has developers and designers doing the bulk of the SEO. I can see you've looked in to it as I have - there are many factors involved if I was to decide to "fix" this myself. To be honest, I don't fancy it - I'm hoping the better approach will come from the wonderful SEO Moz developers who might put in a fix. Hint hint.
-
The first rule in this business is "You can't trust programmers"
I should know, I am a programmer and I used to manage teams of them.
You can't trust them to write something perfect, because they will always make huge assumptions, based on what they know.
They should know that URLs can contain commas, and they should quote them.
If they didn't do that in the final field, it is a deficiency in the code and your stuff isn't going to workunless you fix it manually.
What you need to do to fix this is to add a quote after the 10th comma and also add one at the end of each line.
Unfortunately, even that is a problem.
The problem is there are other fields that may not be quoted, some of which can start with http://
There can also be line breaks in the title field, and possibly even in the link text field.
Quotes and other characters are escaped with double quotes.
Titles and link text can also contain commas, so it is very complex.
Some of the fields are a bigger mess because it depends on the link text, and if the link text contains an image, you'll have quotes and equals signs, commas and all kinds of stuff. You can also have upper ascii characters and multibyte characters.
They did actually quote the first URL, if it contains commas.
They really should have quoted every field
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Ooops. Our crawlers are unable to access that URL
hello
Moz Pro | | ssblawton2533
i have enter my site faroush.com but i got an error
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct
what is problem ?0 -
Duplicate content issues with file download links (diff. versions of a downloadable application)
I'm a little unsure how canonicalisation works with this case. 🙂 We have very regular updates to the application which is available as a download on our site. Obviously, with every update the version number of the file being downloaded changes; and along with it, the URL parameter included when people click the 'Download' button on our site. e.g. mysite.com/download/download.php?f=myapp.1.0.1.exe mysite.com/download/download.php?f=myapp.1.0.2.exe mysite.com/download/download.php?f=myapp.1.0.3.exe, etc In the Moz Site Crawl report all of these links are registering as Duplicate Content. There's no content per se on these pages, all they do is trigger a download of the specified file from our servers. Two questions: Are these links actually hurting our ranking/authority/etc? Would adding a canonical tag to the head of mysite.com/download/download.php solve the crawl issues? Would this catch all of the download.php URLs? i.e. Thanks! Jon
Moz Pro | | jonmc
(not super up on php, btw. So if I'm saying something completely bogus here...be kind 😉 )0 -
Long URLs
My website is hosted by Hubspot. When I create a blog, the URL, as an example, would be: http://www.boxtheorygold.com/blog/bid/27061/Manage-By-the-Numbers/ Instead I am getting the URL below. Google Webmaster tools and moz see this as an error and google says it can't crawl because it is a non-existent page. Users cannot see this page, and Hubspot can't figure it out, but google and moz see it. This problem is occurring on about 25 blogs out of 150. Any ideas? And thanks. URL: http://www.boxtheorygold.com/blog/bid/27061/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/Manage-By-the-Numbers URL: http://www.boxtheorygold.com/blog/bid/27061/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/www.boxtheorygold.com/blog/bid/12158/Manage-By-the-Numbers
Moz Pro | | Rong0 -
Crawlers crawl weird long urls
I did a crawl start for the first time and i get many errors, but the weird fact is that the crawler tracks duplicate long, not existing urls. For example (to be clear): there is a page: www.website.com/dogs/dog.html but then it is continuing crawling:
Moz Pro | | r.nijkamp
www.website.com/dogs/dog.html
www.website.com/dogs/dogs/dog.html
www.website.com/dogs/dogs/dogs/dog.html
www.website.com/dogs/dogs/dogs/dogs/dog.html
www.website.com/dogs/dogs/dogs/dogs/dogs/dog.html what can I do about this? Screaming Frog gave me the same issue, so I know it's something with my website0 -
Crawl Errors from URL Parameter
Hello, I am having this issue within SEOmoz's Crawl Diagnosis report. There are a lot of crawl errors happening with pages associated with /login. I will see site.com/login?r=http://.... and have several duplicate content issues associated with those urls. Seeing this, I checked WMT to see if the Google crawler was showing this error as well. It wasn't. So what I ended doing was going to the robots.txt and disallowing rogerbot. It looks like this: User-agent: rogerbot Disallow:/login However, SEOmoz has crawled again and it still picking up on those URLs. Any ideas on how to fix? Thanks!
Moz Pro | | WrightIMC0 -
Crawl Diagnostics Shows thousands of 302's from a single url. I'm confused
Hi guys I just ran my first campaign and the crawl diagnostics are showing some results I'm unfamiliar with.. In the warnings section it shows 2,838 redirects.. this is where I want to focus. When I click here it shows 5 redirects per page. When I go to click on page 2, or next page, or any other page than page 1 for that matter... this is where things get confusing. Nothing shows. Downloading the csv reveals that 2,834 of these are all showing: URL: http://www.mydomain.com/401/login.php url: http://www.mydomain.com/401/login.php referrer: http://www.mydomain.com/401/login.php location_header: http://www.mydomain.com/401/login.php I guess I'm just looking for an explanation as to why it's showing so many to the same page and what possible actions can be taken on my part to correct it (if needed). Thanks in advance
Moz Pro | | sethwb0 -
Links not appearing on Open Site Explorer
My site gained several new inbound links during December and only two of them are not all showing up on the latest Linkscape update. It seems to be the links that were created at the end of the month which are showing up, whereas a handful at the beginning of the month are nowhere to be seen. All the linking pages have been indexed by Google the links are do-follow, and one of the sites in particular is not obsure and has a DA in the 90's. I appreciate the Linkscape doesn't index everything, but I would have thought that more tof the results of my efforts would have shown up in OSE. I'd be really grateful if anyone could explain this to me please. Thanks Ben
Moz Pro | | atticus70 -
We were unable to grade that page. We received a response code of 301\. URL content not parseable
I am using seomoz webapp tool for my SEO on my site. I have run into this issue. Please see the attached file as it has the screen scrape of the error. I am running an on page scan from seomoz for the following url: http://www.racquetsource.com/squash-racquets-s/95.htm When I run the scan I receive the following error: We were unable to grade that page. We received a response code of 301. URL content not parseable. This page had worked previously. I have tried to verify my 301 redirects and am unable to resolve this error. I can perform other on page scans and they work fine. Is this a known problem with this tool? I have verified ensuring I don't have it defined. Any help would be appreciated.
Moz Pro | | GeoffBatterham0