The numeric-precision tag has no wiki summary.
7
votes
7answers
1k views
Why are floating point numbers used often in Science/Engineering?
While investigating the accuracy of floating point numbers, I've seen in a few places a statement similar to
"float and double are (designed for/used often in) engineering and scientific ...
39
votes
2answers
3k views
Why do some languages round to the nearest EVEN integer?
Programming languages like Scheme (R5RS) and Python (see this Question) round towards the nearest even integer when value is exactly between the surrounding integers.
What is the reasoning behind ...
9
votes
1answer
207 views
Implied precision of floating-point functions
While reviewing another programmer's implementation of a function to calculate the normal distribution CDF, I made a suggestion to either replace the entire implementation with Python's built-in ...
2
votes
2answers
86 views
Difference between USP and IEEE rounding
I am trying to understand the difference between different rounding methods: Our application offers two different types of rounding:
IEEE
USP (GMP) rounding. ...
3
votes
2answers
71 views
Trivially Sortable Encoding for Arbitrary-Precision Decimals
I'm looking for a reasonably(*) space-efficient way to encode arbitrary-precision decimals (e.g. BigDecimal), such that when sorting the bit-pattern of the encodings lexicographically, the numbers ...
1
vote
2answers
107 views
How would I change the precision of a variable in Python?
I'm working on a 2D-physics engine, and I need a certain variable to be precise only to the hundredths. Are there any methods for basically shaving off all of that unneeded precision?
I have tried ...
0
votes
4answers
129 views
Do commercial statistical calculators calculate with a high degree of arbitrary precision or just floating point precision? [closed]
Do commercial statistical calculators like Minitab or SAS (and open source like R) perform statistical analysis with very high precision? Or do they just use floating point (or double) precision?
I ...
22
votes
11answers
3k views
Why do we still use floats? [duplicate]
I understand why floats served a purpose in the past. And I think I can see why they're useful in some simulation examples today. But I think those example are more exceptional than common. So I ...
14
votes
4answers
555 views
How to identify unstable floating point computations?
In numerics, it is very important to be able to identify unstable
schemes and to improve their stability. How to identify unstable
floating point computations?
I am working on a very complex ...
7
votes
1answer
303 views
Why normalization improves numerical precision?
I was reading the following article:
Polynomial interpolation of GPS satellite
coordinates, Milan Horemuz and Johan Vium Andersson 2006
and it states the following:
"The estimation procedure of the ...
2
votes
1answer
1k views
Addition of double's is NOT Equal to Sum of the double as a whole
I am aware of the floating point errors as I had gained some knowledge with my question asked here in SE Floating Point Errors.
What I am finding it difficult to understand is, the output of the ...
6
votes
2answers
746 views
Solutions for floating point rounding errors
In building an application that deals with a lot of mathematical calculations, I have encountered the problem that certain numbers cause rounding errors. While I understand that floating point is not ...
0
votes
2answers
2k views
Is this a good way to compare two numbers?
If we have double numbers, let's say I want to see if some double parameter is equal to zero that is passed as double:
public bool AlmostEqual(double x, double y)
{
double epsilon = ...
3
votes
2answers
237 views
Arithmetic Coding vs. Machine Numeric Precision
When subdividing intervals for arithmetic coding, the worst case scenario is that the final interval will have size 2^(1-n), where n is the number of unique symbols you are encoding. This will reach ...
20
votes
6answers
4k views
Why don't computers store decimal numbers as a second whole number?
Computers have trouble storing fractional numbers where the denominator is something other than a solution to 2^x. This is because the first digit after the decimal is worth 1/2, the second 1/4 (or ...
11
votes
4answers
2k views
Handling extremely large numbers in a language which can't?
I'm trying to think about how I would go about doing calculations on extremely large numbers (to infinitum - intergers no floats) if the language construct is incapable of handling numbers larger than ...
6
votes
5answers
8k views
How to handle large numbers?
Well, I've just started doing puzzles and it's so annoying to see puzzles which are easy to do but also need to handle very large numbers. That's the problem.
Say I have to deal with numbers like 10 ...
0
votes
1answer
474 views
Conversion of a number from Single precision floating point representation to a Half precision floating point [closed]
I have a code where I have to work on Half precision floating point representation
numbers. To achieve that I have created my own C++ class fp16 with all operators(arithmetic logical, relational) ...
27
votes
6answers
8k views
What causes floating point rounding errors?
I am aware that floating point arithmetic has precision problems. I usually overcome them by switching to a fixed decimal representation of the number, or simply by neglecting the error.
However, I ...
5
votes
4answers
5k views
Why SQL numeric precision is specified as the number of digits?
I understand that this might be a loaded question.
For example, in Sql Server 2008, doing DECLARE @someNum NUMERIC(10); means that the range of the variable will be limited by 10 digits. In other ...
