From: Alex Fenton <aff28@...>
Date: 2007-09-24T18:55:15+09:00
Subject: Re: statistical software and data transformation?

M. Edward (Ed) Borasky wrote:
> Alex Fenton wrote:
>> I use Ruby quite a lot at work for data cleaning, transformation and
>> also for generating SPSS syntax. For example, I used it to create a long
>> set of commands for linking together waves of the longitudinal British
>> Household Panel Study.
> 
> Interesting ... how does Ruby compare with other languages for this
> purpose? We might be getting SPSS and if it's as bizarre as I remember,
> I'm going to need some way of preserving my sanity while using it. 

Ruby works nicely for generating SPSS syntax, mainly because of its 
highly functional String/Hash/Array/Regexp classes. For preparing data, 
Excel's also useful, because it includes basic statistical functions (eg 
normal distribution), and because you can copy-n-paste data from a 
spreadsheet into SPSS's Data Editor.

You might want to evaluate Stata as an alternative to SPSS. I haven't 
used it but several more quantitatively-oriented researchers I know 
speak well of it.

>> Not that I know of. I agree re R - on numerous attempts I've never
>> managed to get anywhere with it (I have 8 years programming experience
>> and a postgrad in Research Methods).  It also seems much more geared to
>> the needs of natural rather than social science.
> 
> Outside of "pure statistics", the two most highly-developed application
> areas for R are biology (http://www.bioconductor.org) and quantitative
> finance aka "program trading". Quantitative finance, however, tends to
> jump on bandwagons and jump off onto the "next big thing" quickly as well.

I guess in the "softer" end of social science where I work the data sets 
are relatively small and the analyses not computationally intensive. So 
GUI ease-of-use for occasional users is a important distinguishing feature.

> SPSS has been around a *long* time. As far as I can remember, the only
> thing older was the UCLA Bio-Med package from the early 1960s! Does it
> still read like a hodge-podge of FORTRAN, macro assembler, JCL and such?

Don't know its heritage, but it's a ugly baby... here's some code to 
create a composite key of a four-digit year and a one-digit UK region code:

STRING REGION_YEAR(A6).
COMPUTE REGION_YEAR = CONCAT(
     STRING(Region, F1), '_', STRING(Year, F4)
).
EXECUTE.

> I've been a programmer for a long time and it didn't take me long to
> learn R. In a sense, S and R are dialects of Lisp, so if you're used to
> procedural languages as opposed to functional languages, you'll have a
> steeper learning curve. And if you're used to object-oriented
> programming as done in Smalltalk, Java or Ruby, you'll find R's
> "objects" and "classes" totally different. 

Interesting - I've been programming eight years, but my experience is 
almost all in Ruby, Perl and Javascript, with a bit of C++. Probably why 
I find the SPSS and R syntax so uncomfortable.

alex