From: Robert Klemme <shortcutter@...>
Date: 2009-03-19T01:25:44+09:00
Subject: Re: free RAM problem

--0003255762cafd5dbf0465672bb5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

2009/3/17 Eleanor McHugh <eleanor@games-with-brains.com>:
> On 17 Mar 2009, at 15:23, Edouard Dantes wrote:

>> well, the loops are too basic to create problems, it's fill array with
>> values & write to file.
>>
>> RAM jumps are too erratic (5 or 10% for 2GB RAM)
>>
>> The only processor angry stuff (and possibly RAM too) is the following
>> parsing line, imported from another file.
>>
>> Nokogiri::HTML(page).css('div a').map{|dts|
>> dts.text.gsub(/\s+/,"").gsub(/[^A-Za-z0-9\s]/,'Z') }
>>
>> This line runs thousands of time, might the implied regex feed RAM with
>> datas without cleaning?
>
> I'm not sure it'll make much difference but moving the regex creation int=
o
> the enclosing scope seems neater:
>
> =A0 =A0 =A0 =A0spaces =3D /\s+/
> =A0 =A0 =A0 =A0chars =3D /[^A-Za-z0-9\s]/

This is generally less efficient and AFAIK there are no memory leaks
attached to this.  It's a different story with varying regular
expressions (i.e. based on input or configuration).  But in this case
I'd leave them in place.

> =A0 =A0 =A0 =A0Nokogiri::HTML(page).css('div a').map { |dts|
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0dts.text.gsub(spaces, "").gsub(chars, 'Z')
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
>
> Likewise changing the logic to use gsub! will reduce the number of transi=
ent
> string objects being created and that will reduce memory usage.

True, maybe the first gsub needs to stay in order to avoid aliasing effects=
.

> I'd also be interested to see the code you're using for writing to file a=
s
> the nature of the RAM usage jumps suggests you could be experiencing a
> buffering artefact.

There was also an issue in 1.8.* related to Array#shift or #unshift.

And I agree with your general statement to not explicitly force GC.
Rather look for memory leaks.  Edouard, you might get a first idea
where you spend your memory by dumping instance counts in intervals
(sample attached).

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end

--0003255762cafd5dbf0465672bb5
Content-Type: application/octet-stream; name="mem-stat.rb"
Content-Disposition: attachment; filename="mem-stat.rb"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_fsg898cm0

IyEvdXNyL2Jpbi9lbnYgcnVieTE5CgpkZWYgZHVtcF9zdGF0cyhsYWJlbCwgc3RhdCkKICBwdXRz
ICIgICAje2xhYmVsfSAtLS0iCiAgc3RhdC5zb3J0X2J5IHt8ayx2fCAtdn0uZWFjaF93aXRoX2lu
ZGV4IGRvIHwoayx2KSwgaW5kZXh8CiAgICBicmVhayBpZiBpbmRleCA+PSAxMAogICAgcHJpbnRm
ICIlMTBkICVzXG4iLCB2LCBrCiAgZW5kCmVuZAoKZGVmIHByaW50X3N0YXRzKG9sZCA9IHt9KQog
IHN0YXQgPSBIYXNoLm5ldyAwCiAgT2JqZWN0U3BhY2UuZWFjaF9vYmplY3QoT2JqZWN0KSBkbyB8
b2JqfAogICAgc3RhdFtvYmouY2xhc3NdICs9IDEKICBlbmQKCiAgZGlmZiA9IHt9CiAgb2xkLmVt
cHR5PyBvciBzdGF0LmVhY2ggZG8gfGssdnwKICAgIGRpZmZba10gPSB2IC0gb2xkW2tdLnRvX2kK
ICBlbmQKCiAgcHV0cyAiLS0gU1RBUlQgTWVtb3J5IHN0YXRzIC0tLSIKICBkdW1wX3N0YXRzICdk
aWZmJywgZGlmZiB1bmxlc3MgZGlmZi5lbXB0eT8KICBkdW1wX3N0YXRzICdhYnNvbHV0ZScsIHN0
YXQKCiAgcHV0cyAiLS0gRU5EIE1lbW9yeSBzdGF0cyAtLS0iCgogICMgcmVjYWxjIHRvIGZpbHRl
ciBvdXQgb2JqZWN0cyBjcmVhdGVkCiAgIyBpbiB0aGlzIG1ldGhvZAogIHN0YXQgPSBIYXNoLm5l
dyAwCiAgT2JqZWN0U3BhY2UuZWFjaF9vYmplY3QoT2JqZWN0KSBkbyB8b2JqfAogICAgc3RhdFtv
YmouY2xhc3NdICs9IDEKICBlbmQKICBzdGF0CmVuZAoKcyA9IHByaW50X3N0YXRzCjEwXzAwMC50
aW1lcyB7IFsxLDIsM10gfQpzID0gcHJpbnRfc3RhdHMgcwoyMF8wMDAudGltZXMgeyAiZCIgKyAi
YSIgfQpzID0gcHJpbnRfc3RhdHMgcwpzID0gcHJpbnRfc3RhdHMgcwo=
--0003255762cafd5dbf0465672bb5--