Tuesday, August 4, 2009

Precision Loss in to_i Integer Conversion and in Definition Differs (Happens in Ruby and JRuby)

Just noticed this loss of precision with Ruby and JRuby (not sure if it is in language, OS X, etc.):

ruby 1.8.6 (2008-08-11 patchlevel 287) [universal-darwin9.0]

$ ruby -S irb
>> 9.999999999999999.to_i
=> 9
>> 9.9999999999999999.to_i
=> 10

jruby 1.3.1 (ruby 1.8.6p287) (2009-06-15 2fd6c3d) (Java HotSpot(TM) Client VM 1.5.0_19) [i386-java]

irb(main):016:0> 9.9999999999999999.to_i 
=> 10
irb(main):017:0> 9.999999999999999.to_i 
=> 9

A little more deep (but possibly off-track):

>> 9.99999999999999911182158029988.to_i
=> 10
>> 9.99999999999999911182158029987.to_i
=> 9
Even more strangely, that is not where it rounds up without to_i:
>> 9.99999999999999911182158029987
=> 10.0
>> 9.99999999999999911182158029988
=> 10.0
>> 9.999999999999995
=> 9.99999999999999
>> 9.999999999999996
=> 10.0

1 comment:

Jo said...

it looks complicated