Opened 14 years ago
Closed 12 years ago
#2332 closed defect (fixed)
Trivial divide calculations generate results with rounding errors
Reported by: | garycmartin | Owned by: | garycmartin |
---|---|---|---|
Priority: | Unspecified by Maintainer | Milestone: | Unspecified |
Component: | Calculate | Version: | Unspecified |
Severity: | Unspecified | Keywords: | |
Cc: | Distribution/OS: | Unspecified | |
Bug Status: | New |
Description
Trivial divide calculations are showing rounding errors (tested in Calculate-33).
89 / 100 = 0.890000000
496 / 100 = 4.95999999
Trying the same calculations in a terminal with python is fine 89 / 100 = 0.89, and 496 / 100 = 4.96.
If I try using multiply instead, the answers are accurate.
89 * 0.01 = 0.89
496 * 0.01 = 4.96
Attachments (1)
Change History (5)
comment:1 Changed 14 years ago by greenfeld
comment:2 Changed 12 years ago by godiard
This ticket can be related to #2697
comment:3 Changed 12 years ago by godiard
- Owner changed from rwh to garycmartin
- Status changed from new to assigned
comment:4 Changed 12 years ago by garycmartin
- Resolution set to fixed
- Status changed from assigned to closed
Thanks again Miguel, this patch is now also in mainline.
http://git.sugarlabs.org/calculate/mainline/commit/a2cbde202c7450b12b118b688d8399a6c836ca10
Note: See
TracTickets for help on using
tickets.
Is Calculate using an arbitrary precision numeric engine {python's Decimal type}, or just doing floating point math via eval() or similar?
If it is not doing the former, Calculate probably should do this, as at least a one other software calculator tool I know moved to an arbitrary precision approach to avoid these sorts of quirks.