From: Marco Fernandez Date: 2016-03-27T22:22:02+09:00 Subject: [ruby-core:74609] Re: [Ruby trunk Feature#12222] Introducing basic statistics methods for Enumerable (and optimized implementation for Array) --===============1499850245== Content-Type: multipart/alternative; boundary=089e013c60ec84944e052f07ae11 --089e013c60ec84944e052f07ae11 Content-Type: text/plain; charset=UTF-8 unsubscribe On Sun, Mar 27, 2016 at 10:10 PM, wrote: > Issue #12222 has been updated by Kenta Murata. > > > > Especially I want to provide Enumerable#mean and Enumerable#variance as > built-in features because they should be implemented by precision > compensated algorithms. > > Sorry, I don't want to make them be a built-in features. But I want to > make them a standard library features, at least. > > > ---------------------------------------- > Feature #12222: Introducing basic statistics methods for Enumerable (and > optimized implementation for Array) > https://bugs.ruby-lang.org/issues/12222#change-57736 > > * Author: Kenta Murata > * Status: Assigned > * Priority: Normal > * Assignee: Yukihiro Matsumoto > ---------------------------------------- > As python has statistics library for calculating mean, variance, etc. of > arrays and iterators from version 3.4, > I would like to propose to introduce such features for built-in > Enumerable, and optimized implementation for Array. > > Especially I want to provide Enumerable#mean and Enumerable#variance as > built-in features because they should be implemented by precision > compensated algorithms. > The following example shows that we couldn't calculate the standard > deviation for some arrays with simple variance algorithm because we get > negative variance numbers. > > ```ruby > class Array > # Kahan summation > def sum > s = 0.0 > c = 0.0 > n = self.length > i = 0 > while i < n > y = self[i] - c > t = s + y > c = (t - s) - y > s = t > i += 1 > end > s > end > > # precision compensated algorithm > def variance > n = self.length > return Float::NAN if n < 2 > m1 = 0.0 > m2 = 0.0 > i = 0 > while i < n > x = self[i] > delta = x - m1 > m1 += delta / (i + 1) > m2 += delta*(x - m1) > i += 1 > end > m2 / (n - 1) > end > end > > ary = [ 1.0000000081806004, 1.0000000009124625, 1.0000000099201818, > 1.0000000061821668, 1.0000000042644555 ] > > # simple variance algorithm > a = ary.map {|x| x ** 2 }.sum > b = ary.sum ** 2 / ary.length > p (a - b) / (ary.length - 1) #=> -2.220446049250313e-16 > > # precision compensated algorithm > p ary.variance #=> 1.2248208046392579e-17 > ``` > > I think precision compensated algorithm is too complicated to let users > implement it. > > > > -- > https://bugs.ruby-lang.org/ > > Unsubscribe: > > --089e013c60ec84944e052f07ae11 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
unsubscribe

On Sun, Mar 27, 2016 at 10:10 PM, <<= a href=3D"mailto:muraken@gmail.com" target=3D"_blank">muraken@gmail.com= > wrote:
Issue #12222 has been = updated by Kenta Murata.


> Especially I want to provide Enumerable#mean and Enumerable#variance a= s built-in features because they should be implemented by precision compens= ated algorithms.

Sorry, I don't want to make them be a built-in features. But I w= ant to make them a standard library features, at least.


----------------------------------------
Feature #12222: Introducing basic statistics methods for Enumerable (and op= timized implementation for Array)
https://bugs.ruby-lang.org/issues/12222#c= hange-57736

* Author: Kenta Murata
* Status: Assigned
* Priority: Normal
* Assignee: Yukihiro Matsumoto
----------------------------------------
As python has statistics library for calculating mean, variance, etc. of ar= rays and iterators from version 3.4,
I would like to propose to introduce such features for built-in Enumerable,= and optimized implementation for Array.

Especially I want to provide Enumerable#mean and Enumerable#variance as bui= lt-in features because they should be implemented by precision compensated = algorithms.
The following example shows that we couldn't calculate the standard dev= iation for some arrays with simple variance algorithm because we get negati= ve variance numbers.

```ruby
class Array
=C2=A0 # Kahan summation
=C2=A0 def sum
=C2=A0 =C2=A0 s =3D 0.0
=C2=A0 =C2=A0 c =3D 0.0
=C2=A0 =C2=A0 n =3D self.length
=C2=A0 =C2=A0 i =3D 0
=C2=A0 =C2=A0 while i < n
=C2=A0 =C2=A0 =C2=A0 y =3D self[i] - c
=C2=A0 =C2=A0 =C2=A0 t =3D s + y
=C2=A0 =C2=A0 =C2=A0 c =3D (t - s) - y
=C2=A0 =C2=A0 =C2=A0 s =3D t
=C2=A0 =C2=A0 =C2=A0 i +=3D 1
=C2=A0 =C2=A0 end
=C2=A0 =C2=A0 s
=C2=A0 end

=C2=A0 # precision compensated algorithm
=C2=A0 def variance
=C2=A0 =C2=A0 n =3D self.length
=C2=A0 =C2=A0 return Float::NAN if n < 2
=C2=A0 =C2=A0 m1 =3D 0.0
=C2=A0 =C2=A0 m2 =3D 0.0
=C2=A0 =C2=A0 i =3D 0
=C2=A0 =C2=A0 while i < n
=C2=A0 =C2=A0 =C2=A0 x =3D self[i]
=C2=A0 =C2=A0 =C2=A0 delta =3D x - m1
=C2=A0 =C2=A0 =C2=A0 m1 +=3D delta / (i + 1)
=C2=A0 =C2=A0 =C2=A0 m2 +=3D delta*(x - m1)
=C2=A0 =C2=A0 =C2=A0 i +=3D 1
=C2=A0 =C2=A0 end
=C2=A0 =C2=A0 m2 / (n - 1)
=C2=A0 end
end

ary =3D [ 1.0000000081806004, 1.0000000009124625, 1.0000000099201818, 1.000= 0000061821668, 1.0000000042644555 ]

# simple variance algorithm
a =3D ary.map {|x| x ** 2 }.sum
b =3D ary.sum ** 2 / ary.length
p (a - b) / (ary.length - 1)=C2=A0 #=3D> -2.220446049250313e-16

# precision compensated algorithm
p ary.variance=C2=A0 #=3D> 1.2248208046392579e-17
```

I think precision compensated algorithm is too complicated to let users imp= lement it.



--
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:= ruby-core-request@ruby-lang.org?subject=3Dunsubscribe>
<http://lists.ruby-lang.org/cgi-bin/m= ailman/options/ruby-core>

--089e013c60ec84944e052f07ae11-- --===============1499850245== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline Unsubscribe: --===============1499850245==--