The numeric-precision tag has no wiki summary.
0
votes
0answers
310 views
Is this a good way to compare two numbers?
If we have double numbers, let's say I want to see if some double parameter is equal to zero that is passed as double:
public bool AlmostEqual(double x, double y)
{
double epsilon = ...
3
votes
2answers
136 views
Arithmetic Coding vs. Machine Numeric Precision
When subdividing intervals for arithmetic coding, the worst case scenario is that the final interval will have size 2^(1-n), where n is the number of unique symbols you are encoding. This will reach ...
20
votes
6answers
2k views
Why don't computers store decimal numbers as a second whole number?
Computers have trouble storing fractional numbers where the denominator is something other than a solution to 2^x. This is because the first digit after the decimal is worth 1/2, the second 1/4 (or ...
3
votes
6answers
714 views
Programming language for numerical computing [closed]
I am attending a course in numerical computing next semester, so I thought I could prepare this summer. My guess is that all of the stuff in the course is language independent, but it is cool to try ...
11
votes
4answers
1k views
Handling extremely large numbers in a language which can't?
I'm trying to think about how I would go about doing calculations on extremely large numbers (to infinitum - intergers no floats) if the language construct is incapable of handling numbers larger than ...
6
votes
5answers
3k views
How to handle large numbers?
Well, I've just started doing puzzles and it's so annoying to see puzzles which are easy to do but also need to handle very large numbers. That's the problem.
Say I have to deal with numbers like 10 ...
0
votes
1answer
290 views
Conversion of a number from Single precision floating point representation to a Half precision floating point [closed]
I have a code where I have to work on Half precision floating point representation
numbers. To achieve that I have created my own C++ class fp16 with all operators(arithmetic logical, relational) ...
23
votes
6answers
2k views
What causes floating point rounding errors?
I am aware that floating point arithmetic has precision problems. I usually overcome them by switching to a fixed decimal representation of the number, or simply by neglecting the error.
However, I ...
4
votes
4answers
1k views
Why SQL numeric precision is specified as the number of digits?
I understand that this might be a loaded question.
For example, in Sql Server 2008, doing DECLARE @someNum NUMERIC(10); means that the range of the variable will be limited by 10 digits. In other ...
