Improve for loop explosion success with deferred tokens

[jasmith-hs]

Now that we have better tracking of the PrefixToPreserveState, we can separate the collection expression execution from the iterating of the for loop.
This leads to better explosion success where we can now support exploding for loops even when there are modifications inside for loops with deferred tokens
Additionally, for loop evaluation will be faster as we don't need to re-run the for loop.

For this input:

{% set my_list = [] %}
{% for i in range(2) %}
{% for j in deferred %}
{% do my_list.append(i) %}
{% endfor %}
{% endfor %}
{{ my_list }}

Previously, we'd have this output:

{% set my_list = [] %}{% for i in [0, 1] %}
{% for j in deferred %}
{% do my_list.append(i) %}
{% endfor %}
{% endfor %}
{{ my_list }}

But now we can get this exploded output:

{% set my_list = [] %}{% for __ignored__ in [0] %}
{% for j in deferred %}
{% do my_list.append(0) %}
{% endfor %}

{% for j in deferred %}
{% do my_list.append(1) %}
{% endfor %}
{% endfor %}
{{ my_list }}

[boulter]

Can you explain why this is better practically? What does explosion mean?

[jasmith-hs]

Can you explain why this is better practically? What does explosion mean?

This is better because it means that in a loop item in collection, we don't have to defer item, so we can resolve anything that depends on item.

Exploding just means that we're taking something like:{% for i in range(5) %}{{ i }},{% endfor %} and turning it to 0,1,2,3,4,. So the difference for eager execution is whether we have to preserve the for token, or if we can render each of the iterations out, potentially leaving some deferred tokens in the result.

A practical example would be that:

{% for path in ['path1', 'path2'] %}
Path is '{{ path }}'.
{% include path %}
{% endfor %}

Is more work to evaluate than:

Path is 'path1'.
// Whatever got evaluated inside 'path1'
Path is 'path2'.
// Whatever got evaluated inside 'path2'

Without exploding it, we aren't able to eagerly run the expression tags, include tag, as well as all the logic inside the include tag.