From: "M. Edward (Ed) Borasky" Date: 2006-11-17T00:42:51+09:00 Subject: Re: *Fast* way to process large files line by line Robert Klemme wrote: > On 16.11.2006 10:51, Neil Wilson wrote: >>> Btw will using something like an mmap extension for ruby speed things >>> up for me ?. >> >> Can't harm to try. I use it for uploading large files and it helps >> there. > > I beg to differ: it can actually harm to apply an optimization measure > without making sure that it actually solves the problem. As long as > you do not know what makes your program slow trying out measures is > pretty much a waste of time. > >> Unfortunately Operating systems don't always behave like you think they >> ought to when doing something like this. Get to know your profiler and >> brush up on your data processing techniques. When you hit a speed >> problem, you *have* to start and understand what the machine is doing >> with each high level instruction you give it and how they all interact >> with each other. > > Exactly. I'd start with vmstat to find out whether we are IO bound or > CPU bound. > > Regards > > robert Just for the sake of amusement, I did a traceroute to see what the output looks like: $ traceroute -n rubyforge.org traceroute to rubyforge.org (205.234.109.18), 30 hops max, 38 byte packets 1 192.168.0.1 4.080 ms 2.845 ms 1.932 ms 2 * * * 3 68.87.218.17 9.540 ms 7.114 ms 7.187 ms 4 68.87.216.29 8.025 ms * 9.980 ms 5 12.116.188.9 21.930 ms 21.446 ms 21.619 ms 6 12.123.12.126 23.060 ms 21.566 ms 21.739 ms MPLS Label=31697 CoS=0 TTL=1 S=1 7 12.123.13.185 22.617 ms 22.220 ms 20.848 ms 8 206.24.211.1 21.786 ms 22.516 ms 20.589 ms 9 204.70.194.50 30.330 ms 28.985 ms 204.70.192.113 106.454 ms MPLS Label=202112 CoS=0 TTL=1 S=1 10 204.70.192.85 62.240 ms 204.70.192.134 87.983 ms 86.297 ms MPLS Label=509440 CoS=0 TTL=1 S=1 11 204.70.192.50 64.302 ms 204.70.192.25 104.547 ms 204.70.192.50 65.496 ms MPLS Label=190688 CoS=0 TTL=1 S=1 12 204.70.192.69 81.608 ms 208.173.10.201 105.080 ms 204.70.192.69 82.155 ms MPLS Label=131056 CoS=0 TTL=1 S=1 13 208.173.50.154 104.882 ms 136.592 ms 204.70.192.62 95.377 ms 14 205.234.109.18 101.528 ms !<10> 206.24.238.218 100.340 ms 100.945 ms 15 205.234.109.18 100.920 ms !<10> 208.173.50.154 100.550 ms 205.234.109.18 100.802 ms !<10> So ... we have *one* traceroute from point a (my system) to point b (rubyforge.org). I'm assuming the original poster has some simple shell script that does a loop forever around something like "date; traceroute -n ip.add.re.ss; sleep 300" to get a log of timestamped traceroutes every five minutes. Then he wants to collect hundreds of these lines from many files (60 GB total) into a huge in-core Ruby structure, then make some kind of report from it. It got so big from the naive approach that he started marshalling it out of core in chunks, leading to the code he posted here. A little arithmetic: that single traceroute above is about 1000 bytes. 60 GB is 60 million times 1000 bytes, which means our collection of 60 GB of traceroute data has about 60 million individual traceroutes, assuming they are as long as my off-the-wall traceroute over the open internet to rubyforge. Obviously they will be much smaller if done on a well-designed corporate intranet, which means there are many more of them. In any event, it looks to me like he is trying to reinvent some network monitoring wheel for which there are undoubtedly both expensive commercial packages and wonderful full-featured open source tools available. I wrote a lot of code like this ten years ago (in ksh, awk and Perl 4) when such tools weren't available. Now they are, and there are Ruby interfaces to most of them. So in addition to buying and learning from "Data Crunching", I'm going to recommend the original poster head over to http://oss.oetiker.ch/rrdtool/ and do some window shopping. IIRC there is a Ruby interface to RRDTool, although I've forgotten whether it's currently actively maintained -- it wasn't a year or so ago when I first started investigating Ruby. And there are definitely tools to migrate a "native" round robin database into an "ordinary" relational database. -- M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P) http://borasky-research.blogspot.com/ If God had meant for carrots to be eaten cooked, He would have given rabbits fire.