Opened 13 years ago

Closed 12 years ago

#2332 closed defect (fixed)

Trivial divide calculations generate results with rounding errors

Reported by: garycmartin Owned by: garycmartin
Priority: Unspecified by Maintainer Milestone: Unspecified
Component: Calculate Version: Unspecified
Severity: Unspecified Keywords:
Cc: Distribution/OS: Unspecified
Bug Status: New

Description

Trivial divide calculations are showing rounding errors (tested in Calculate-33).

89 / 100 = 0.890000000
496 / 100 = 4.95999999

Trying the same calculations in a terminal with python is fine 89 / 100 = 0.89, and 496 / 100 = 4.96.

If I try using multiply instead, the answers are accurate.

89 * 0.01 = 0.89
496 * 0.01 = 4.96

Attachments (1)

0001-Ticket-2332-Rational-format-problem.patch​ (723 bytes) - added by miguelgarcia 12 years ago.
patch for this issue

Download all attachments as: .zip

Change History (5)

comment:1 Changed 13 years ago by greenfeld

Is Calculate using an arbitrary precision numeric engine {python's Decimal type}, or just doing floating point math via eval() or similar?

If it is not doing the former, Calculate probably should do this, as at least a one other software calculator tool I know moved to an arbitrary precision approach to avoid these sorts of quirks.

comment:2 Changed 12 years ago by godiard

This ticket can be related to #2697

Changed 12 years ago by miguelgarcia

patch for this issue

comment:3 Changed 12 years ago by godiard

  • Owner changed from rwh to garycmartin
  • Status changed from new to assigned

comment:4 Changed 12 years ago by garycmartin

  • Resolution set to fixed
  • Status changed from assigned to closed

Thanks again Miguel, this patch is now also in mainline.

​http://git.sugarlabs.org/calculate/mainline/commit/a2cbde202c7450b12b118b688d8399a6c836ca10

Note: See TracTickets for help on using tickets.