From: Colin Bartlett Date: 2010-07-28T07:07:10+09:00 Subject: Re: re-organize original text data then write in files --0016364d23abc7814c048c65b89a Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable I'm putting this at the top of my post because I think the basic problem here may be intensive numeric calculations, and - even more so - disk (inpu= t and) output of about 16 MiB x N bytes of data, where N is 8 bytes (? for Floating point numbers), so about 128 MiB in total, and other people will have a better knowledge of some possibly useful links. On Sun, Jul 25, 2010 at 11:15 PM, Junhui Liao wro= te: > Actually, I developed two versions of C++ script. > One is opening 4096 files at the same time. This cost 3 hours. > Another version is saving all of the data in a big vector, > then scanning the vector to pick the right items to write > in files. This cost 2 hours and 45 minutes. :-). > Sorry - in my post I misunderstood what you meant by "cost". I think it is (very?) unlikely that any Ruby (or Perl or Python, etc?) program will run faster than your C++ scripts. Where Ruby (or Python - I'm not so sure about Perl, I haven't used it) does have an advantage is that I think development may be quicker. So there are trade-offs. (Incidentally, I'm not an expert, but those timings suggest to me that the major processing cost may be in writing the results out to disk, so changing the language for all or part o= f the processing is unlikely to make a large difference?) But I'm open to correction: there are people who have used Ruby for fairly intensive large data sets processing, but my understanding is that they use a mixture of Ruby as "glue" with any intensive calculations in C, etc. For example, from some limited experience I have the speed of Ruby reading strings of bytes in from files is similar to the speed of Java or compiled Pascal, but for calculating CRCs of files the speed of pure Ruby calculatin= g the CRCs once the bytes had read in was much slower than Java or compiled Pascal: so I used Ruby (or rather JRuby) to read in the strings of bytes from the files, and then called Java code from Ruby to calculate the CRC from the bytes. Overall the speed of this was similar to a pure Java or pur= e compiled Pascal program. Piet Hut and Jun Makino have been using Ruby to model dense star clusters. (Note that this is something I know nothing about! I'm just intrigued by th= e underlying principle of using Ruby for intensive numerical calculations by developing in Ruby without worrying about speed by using smaller unrealisti= c models, and then using more realistic models by translating part (or all!) of the Ruby code to a faster language.) http://www.kira.org/index.php?option=3Dcom_content&task=3Dview&id=3D124&Ite= mid=3D154 ...MODEST is the new name for the Stellar Dynamics workshop. It stands for: MOdeling DEnse STellar systems ... The basic idea is to start a kind of N-body wikipedia, as a group's process= . It should be self contained, a place to gather all the basic information that is currently missing from the literature. Up till now, if you want to write a decent N-body code from scratch, you have to somehow catch the oral knowledge that is floating around in the community, most of which has never been written down. We want to change that. One place to start is the open-source introductory text written over the last several years by Piet Hut and Jun Makino, "Moving Stars Around"; see http://www.artcompsci.org/#msa where we give pointers to two versions. The older one is written using C++ and the newer one, with even more detailed background, uses Ruby. We propose to use the basic text, and to translate the (rather short) pieces of code into other languages, starting with LSL, the Linden Scripting Language, and then to move on to OpenSim. ... http://www.manybody.org/wiki/index.php/Moving_Stars_Around http://www.artcompsci.org http://www.artcompsci.org/kali/pub/msa/title.html http://www.artcompsci.org/kali/pub/msa/ch16.html#rdocsect123 16.7. Conclusion Dan: This confirms our earlier conclusions. At least on this particular computer, that we are now using to do some speed tests, the unoptimized C version takes 50% more time than the optimized version, the simplest Ruby version takes about 50 times more time, the Ruby array version about 100 times more, and finally the Ruby vector version takes more than 250 times more time than the optimized C version. Carol: But even so, for short calculations, who cares if a run takes ten millisecond or a few seconds? I certainly like the power of Ruby in giving us vector classes, and a lot more goodies. We have barely scratched the surface of all the power that Ruby can give us. You should see what we can do when we really start to pass blocks to methods and . . . Dan: . . . and then we will start drinking a lot of coffee, while waiting for results when we begin to run 100-body experiments! Is there no way to speed up Ruby calculations? Carol: There is. By the time we use 100 particles, we are talking about 10^= 2 * 10^2 =3D 10^4 force calculations for every time step. This means that the calculation of the mutually accelerations will take up almost all of the computer time. What we can do is write a short C code for computing the accelerations. It is possible to invoke such a C code from within a Ruby code. In that way, we can leave most of the Ruby code unchanged, while gaining most of the C speed. Erica: I certainly like the flexibility of a high-level language like Ruby, at least for writing a few trial versions of a new code. In order to play around, Ruby is a lot more fun and a lot easier to use than C or C++ or Fortran. After we have constructed a general N-body code that we are really happy with, we can always translate part of it into C, as Carol just suggested. Or, if really needed to gain speed, we could even translate the whole code into C. Translating a code will always take far less time than developing a code in the first place. And is seems pretty clear to me that development will be faster in Ruby. Dan: I'm not so sure about all that. In any case, we got started now with Ruby, so let us see how far we get. But if and when we really get bogged down by the lack of speed of Ruby, we should not hesitate to switch to a more efficient language. > ... > Thanks a lot for your encourage, > I tried to read something on ruby already. > Since this language is very simple and beautiful, > no matter it works for my case or not. > (But I hope it could be.) > To quote something Damon Runyon quoted/adapted: The race is not always to the swift nor the battle to the strong - but that=92s the way to bet. http://www.barrypopik.com/index.php/new_york_city/entry/the_race_is_not_alw= ays_to_the_swift_no_the_battle_to_the_strong_but_thats_t/ Or to continue that analogy, "horses for courses"! (Translation: for parts of a task that are heavy numerical processing a "lower level" language is likely to be faster. That said, sometimes the slower higher level language is fast enough: I was doing some actuarial programming in JRuby, and intended to have some intensive numerical calculations in Java. It turned out that JRuby was actually quite fast enough for those particular calculations, although (for portability reasons= ) I will be translating that numerical code from Ruby to Java.) > Maybe it partly depends on what standard methods/functions are > available: for example, in Ruby you can read a line from a file > into a String, and then use a builtin method on the String > to split it into an array of values using a specified delimiter, > so in your case a space character? > I need this kind of comment seriously, saying, what are the knowledge > which is necessary and enough to do my job. If there are some special > and powerful methods or stances to do this kind of stuff. > Or be better, give a example just very close my case. > I can get the detailed by reading book(s) or googling. For your particular problem you've already written the C++ code to do the splitting and the processing, so (ignoring maintainability issues) I don't think Ruby will speed up the development, and - for the reasons above - I think it is unlikely to result in faster processing. But you might find it worth looking at integrating Ruby (or Python or Perl) with C, C++ or Java for other processing problems. I don't have any experience with C or C++, but I have found integrating Ruby with Java fairly easily. (Once I found my way round some initial misunderstandings I had with how to package Java to be used by Ruby.) Anyway, good luck with your task(s)! --0016364d23abc7814c048c65b89a--