When using -e6, iterating TypedArrays or other non-list Iterables doesn't work

[daboross]

The follow code performs correctly when not using the -e6 parameter to Transcrypt, but fails when using -e6:

x = __new__(Int8Array(2))
x[0] = 3
x[1] = 2
for i in x:
    print(i)

When run without -e6, the above code produces two lines of output, 3 and 2. With -e6 enabled, nothing is printed. While this bug is annoying, it might be a bit worse that no error is given when trying to iterate something non-iterable. This is a bug I ran into when trying to switch to using -e6, and it was quite hard to track down where the problem was coming from without any errors for not being able to iterate.

Int8Array defines a length and index-access, and a Symbol.iterator property for standard ES6 iterator access. This can be seen with the following code in a node.js console:

> let x = new Int8Array(2)
> x[0] = 3
> x[1] = 2
> for (let i of x) { console.log(i) }
3
2

I believe this is due to Transcrypt allowing for JavaScript generators, but not "iterables". I don't know too much about the internals of JavaScript nor Transcrypt, but I believe this could be fixed by checking for a Symbol.iterator property in py_iter?

In any case, I hope this information proves useful in improving Transcrypt.

[daboross]

I don't know where to apply this in the repository, but this patch on generated code seems to fix the issue for me:

diff --git a/dist/main.js b/target/main.js
index 46ed629..f0fa741 100644
--- a/target/main.js
+++ b/target/main.js
@@ -778,6 +778,9 @@ function main () {
            iterator.next = wrap_py_next;
            return iterator;
        }
+       else if (Symbol.iterator in iterable) {
+           return iterable[Symbol.iterator]();
+       }
        else if ('next' in iterable) {  // It's a JavaScript generator
            // It should have an iterator field, but doesn't in Chrome
            // So we just return the generator itself, which is both an iterable and an iterator

[JdeH]

Thanks a lot. I'll look into this with priority!

Jacques

[daboross]

This might be a bit of a stretch, but would it possibly be possible to add Symbol.iterator definitions to all types defined by Transcrypt, and just have for loops calling for .. of after creating a shim Symbol.iterator on things which define a __next__ or next method?

It would probably be no practical difference, but it would make more readable output code - working directly with JavaScript iterables seems like it would make it easier to tell what's going on most of the time reading Transcrypted code than when there's a while loop and two variable declarations in output code for every for loop in input code.

[JdeH]

Hi,

Your fix seems to work fine!
I've worked it out a little to generate the Python next attribute.
Also general error handling has been improved, to be less 'silent' on such problems.
Iterating over a non iterable now gives an exception that you may catch.
For compatibility with JS exceptions, this exception has a 'stack' attribute, that can print a stack trace.
Later on maybe (a stripped version of the) 'traceback' module should be added.

So this code

try:
    x = __new__(Int8Array(2))
    x[0] = 3
    x[1] = 2
    for i in x:
        print(i)

    y = 3
    for j in y:
        print (j)
except Exception as exception:
    print (exception)
    print (exception.stack)

prints the following

I'll do some regression testing and then commit your solution + the add ons.

fyi, the relevant code originates from ../transcrypt/modules/org/transcrypt/javascript/builtin.mod.js:

    function py_iter (iterable) {                   // Produces universal iterator with Python '__next__' as well as JavaScript 'next'
        try {
            if ('__iter__' in iterable) {               // It's a Python iterable (incl. JavaScript Arrays and strings)
                var iterator = iterable.__iter__ ();
                iterator.next = wrap_py_next;
                return iterator;
            }
            else if ('selector' in iterable) {          // Assume it's a JQuery iterator
                var iterator = list (iterable) .__iter__ ();
                iterator.next = wrap_py_next;
                return iterator;
            }
            else if ('next' in iterable) {              // It's a JavaScript generator
                // It should have an iterator field, but doesn't in Chrome
                // So we just return the generator itself, which is both an iterable and an iterator
                iterable.__next__ = wrap_js_next;
                return iterable;
            }
            else if (Symbol.iterator in iterable) {     // It's a JavaScript iterable such as a typed array, but not a generator
                iterator = iterable [Symbol.iterator] ();
                iterator.__next__ = wrap_js_next;
                return iterator;
            }
        }
        catch (exception) {
        }
        throw IterableError (new Error ()); // No iterator at all, 'in' may be false everywhere, or throw an exception
    }

Jacques

[JdeH]

@daboross
As for your last remark, I've been considering that and it may be a good idea. I'll have to do some more thinking about the possible consequences, I'll file it separately as an enhancement proposal.

[JdeH]

Fix committed for iterating over e.g. TypedArray, together with general improvements in exception handling, e.g. stack trace.
When using -e 6, iterating over non-iterable now gives exception.

Test case added:

    autoTester.check ('Issue 112')
    try:
        if __envir__.executor_name == __envir__.transpiler_name: # CPython doesn't know Int8Array
            x = __new__ (Int8Array (2))
        else:
            x = [None, None]

        x [0] = 3
        x [1] = 2
        for i in x:
            autoTester.check (i)

        # Since JavaScript 5 gives no exception for for a loop over a non-iterable, following code must only be executed for JavaScript 6
        # Since Transcrypt doesn't get to see all modules loaded by CPython, __ifdef__ cannot be made to do its thing for all modules in an efficient way for CPython
        # But a normal 'if' will work
        if '__esv6__' in __symbols__:
            y = 3
            for j in y:
                autoTester.check (j)

    except Exception as exception:
        autoTester.check ('Detected iterating over non-iterable')

Could you let me know if this works for you.

Kind regards
Jacques

[daboross]

@JdeH awesome, thank you! The fix does seem to work correctly!