Dismiss
Announcing Stack Overflow Documentation

We started with Q&A. Technical documentation is next, and we need your help.

Whether you're a beginner or an experienced developer, you can contribute.

Sign up and start helping → Learn more about Documentation →

I have a problem with big iterator in for loop in the code below. It generates floats by reading a string list containing numbers.

def float_generator(tekstowe):
        x = ''
        for c in tekstowe:
            if c != ' ':
                x += c
            else:
                out = float(x)
                x = ''
                yield(out)

I'm getting a "OverflowError: iter index too large". I try to use really big iter numbers (like billions of values in a searched file). Is iter range somehow limited in for loops?

Using Python 2.7 64 bit. Thanks.

share|improve this question
    
How big does float(x) get? – Padraic Cunningham Sep 13 at 12:25
    
They are 0-255 valued and look a bit random like this: 25.75000000 0.340 22.33333397 0.667. I mean these are 4 example floats. – Konrad Gje Sep 13 at 12:28
    

Looks like tekstowe is a sequence type that only implements __getitem__, not __iter__, so it's using the Python iterator wrapper that calls __getitem__ with 0, then 1, 2, 3, etc., until __getitem__ raises IndexError.

As an implementation detail, Python 2.7.11 and higher limits the value of the index passed by the iterator wrapper to LONG_MAX (before 2.7.11, it wasn't bounds checked but it still used a long for index storage, so it would wrap and start indexing with negative values). This doesn't matter on most non-Windows 64 bit builds, where LONG_MAX is 2**63 - 1 (larger than you'd likely encounter), but on Windows, C longs remain 32 bit quantities even on 64 bit builds, so LONG_MAX remains 2**31 - 1, which is low enough to be reached in human timescales.

Your options are:

  1. Change the implementation of whatever class tekstowe is to give it a true __iter__ method, so it doesn't get wrapped by the sequence iterator wrapper when you use it
  2. Upgrade to Python 3.4+, ideally 3.5 (2.7.10/3.4.3 and below lacks the check for overflow entirely, but this could mean wraparound causes infinite looping; 3.4.4/3.5.0 added the check, and they use a signed size_t, testing against PY_SSIZE_T_MAX, which means it will not error until the index reaches 2**63 - 1 on any 64 bit build, Windows or otherwise)

The changes to add the overflow checks were made to resolve Python bug #22939; the type change (from long to Py_ssize_t) for the sequence iterator's index storage occurred in 3.4.0's release, resolving Python bug #17932.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.