Re: re-organize original text data then write in files

I'm putting this at the top of my post because I think the basic problem
here may be intensive numeric calculations, and - even more so - disk (inpu=
t
and) output of about 16 MiB x N bytes of data, where N is 8 bytes (? for
Floating point numbers), so about 128 MiB in total, and other people will
have a better knowledge of some possibly useful links.

On Sun, Jul 25, 2010 at 11:15 PM, Junhui Liao <[e-mail]>wro=
te:

> Actually, I developed two versions of C++ script.
> One is opening 4096 files at the same time. This cost 3 hours.
> Another version is saving all of the data in a big vector,
> then scanning the vector to pick the right items to write
> in files. This cost 2 hours and 45 minutes. :-).
>
Sorry - in my post I misunderstood what you meant by "cost". I think it is
(very?) unlikely that any Ruby (or Perl or Python, etc?) program will run
faster than your C++ scripts. Where Ruby (or Python - I'm not so sure about
Perl, I haven't used it) does have an advantage is that I think development
may be quicker. So there are trade-offs. (Incidentally, I'm not an expert,
but those timings suggest to me that the major processing cost may be in
writing the results out to disk, so changing the language for all or part o=
f
the processing is unlikely to make a large difference?)

But I'm open to correction: there are people who have used Ruby for fairly
intensive large data sets processing, but my understanding is that they use
a mixture of Ruby as "glue" with any intensive calculations in C, etc. For
example, from some limited experience I have the speed of Ruby reading
strings of bytes in from files is similar to the speed of Java or compiled
Pascal, but for calculating CRCs of files the speed of pure Ruby calculatin=
g
the CRCs once the bytes had read in was much slower than Java or compiled
Pascal: so I used Ruby (or rather JRuby) to read in the strings of bytes
from the files, and then called Java code from Ruby to calculate the CRC
from the bytes. Overall the speed of this was similar to a pure Java or pur=
e
compiled Pascal program.

Piet Hut and Jun Makino have been using Ruby to model dense star clusters.
(Note that this is something I know nothing about! I'm just intrigued by th=
e
underlying principle of using Ruby for intensive numerical calculations by
developing in Ruby without worrying about speed by using smaller unrealisti=
c
models, and then using more realistic models by translating part (or all!)
of the Ruby code to a faster language.)

http://www.kira.org/index.php?option=3Dcom_content&task=3Dview&id=3D124&Ite=
mid=3D154
...MODEST is the new name for the Stellar Dynamics workshop. It stands for:
MOdeling DEnse STellar systems
...
The basic idea is to start a kind of N-body wikipedia, as a group's process=
.
It should be self contained, a place to gather all the basic information
that is currently missing from the literature. Up till now, if you want to
write a decent N-body code from scratch, you have to somehow catch the oral
knowledge that is floating around in the community, most of which has never
been written down. We want to change that.

One place to start is the open-source introductory text written over the
last several years by Piet Hut and Jun Makino, "Moving Stars Around"; see
http://www.artcompsci.org/#msa where we give pointers to two versions. The
older one is written using C++ and the newer one, with even more detailed
background, uses Ruby. We propose to use the basic text, and to translate
the (rather short) pieces of code into other languages, starting with LSL,
the Linden Scripting Language, and then to move on to OpenSim. ...

http://www.manybody.org/wiki/index.php/Moving_Stars_Around
http://www.artcompsci.org
http://www.artcompsci.org/kali/pub/msa/title.html
http://www.artcompsci.org/kali/pub/msa/ch16.html#rdocsect123
16.7. Conclusion

Dan: This confirms our earlier conclusions. At least on this particular
computer, that we are now using to do some speed tests, the unoptimized C
version takes 50% more time than the optimized version, the simplest Ruby
version takes about 50 times more time, the Ruby array version about 100
times more, and finally the Ruby vector version takes more than 250 times
more time than the optimized C version.

Carol: But even so, for short calculations, who cares if a run takes ten
millisecond or a few seconds? I certainly like the power of Ruby in giving
us vector classes, and a lot more goodies. We have barely scratched the
surface of all the power that Ruby can give us. You should see what we can
do when we really start to pass blocks to methods and . . .

Dan: . . . and then we will start drinking a lot of coffee, while waiting
for results when we begin to run 100-body experiments! Is there no way to
speed up Ruby calculations?

Carol: There is. By the time we use 100 particles, we are talking about 10^=
2
* 10^2 =3D 10^4 force calculations for every time step. This means that the
calculation of the mutually accelerations will take up almost all of the
computer time. What we can do is write a short C code for computing the
accelerations. It is possible to invoke such a C code from within a Ruby
code. In that way, we can leave most of the Ruby code unchanged, while
gaining most of the C speed.

Erica: I certainly like the flexibility of a high-level language like Ruby,
at least for writing a few trial versions of a new code. In order to play
around, Ruby is a lot more fun and a lot easier to use than C or C++ or
Fortran. After we have constructed a general N-body code that we are really
happy with, we can always translate part of it into C, as Carol just
suggested. Or, if really needed to gain speed, we could even translate the
whole code into C. Translating a code will always take far less time than
developing a code in the first place. And is seems pretty clear to me that
development will be faster in Ruby.

Dan: I'm not so sure about all that. In any case, we got started now with
Ruby, so let us see how far we get. But if and when we really get bogged
down by the lack of speed of Ruby, we should not hesitate to switch to a
more efficient language.


> ...
>
Thanks a lot for your encourage,
> I tried to read something on ruby already.
> Since this language is very simple and beautiful,
> no matter it works for my case or not.
> (But I hope it could be.)
>

To quote something Damon Runyon quoted/adapted: The race is not always to
the swift nor the battle to the strong - but that=92s the way to bet.
http://www.barrypopik.com/index.php/new_york_city/entry/the_race_is_not_alw=
ays_to_the_swift_no_the_battle_to_the_strong_but_thats_t/
Or to continue that analogy, "horses for courses"!
(Translation: for parts of a task that are heavy numerical processing a
"lower level" language is likely to be faster. That said, sometimes the
slower higher level language is fast enough: I was doing some actuarial
programming in JRuby, and intended to have some intensive numerical
calculations in Java. It turned out that JRuby was actually quite fast
enough for those particular calculations, although (for portability reasons=
)
I will be translating that numerical code from Ruby to Java.)

> Maybe it partly depends on what standard methods/functions are
> available: for example, in Ruby you can read a line from a file
> into a String, and then use a builtin method on the String
> to split it into an array of values using a specified delimiter,
> so in your case a space character?

> I need this kind of comment seriously, saying, what are the knowledge
>  which is necessary and enough to do my job. If there are some special
> and powerful methods or stances to do this kind of stuff.
> Or be better, give a example just very close my case.
> I can get the detailed by reading book(s) or googling.

For your particular problem you've already written the C++ code to do the
splitting and the processing, so (ignoring maintainability issues) I don't
think Ruby will speed up the development, and - for the reasons above - I
think it is unlikely to result in faster processing. But you might find it
worth looking at integrating Ruby (or Python or Perl) with C, C++ or Java
for other processing problems. I don't have any experience with C or C++,
but I have found integrating Ruby with Java fairly easily. (Once I found my
way round some initial misunderstandings I had with how to package Java to
be used by Ruby.)

Anyway, good luck with your task(s)!