when crawling too fast, backend gets stuck on disallowed sequence even when crawling is allowed again

[katestange]

To reproduce: crawl a lot of sequences (using the API directly from URL bar is fine), until the OEIS complains about you crawling too fast. When this happens, trying the API on the most recent sequence will result in an error in the browser: "Error: Value fetching for {oeis_id} in progress.". (You can verify this by going to oeis.org and asking for something there, and OEIS will say you are crawling too fast, come back later.) Even after the OEIS allows crawling again (as verified by visiting the site after a few minutes pause), the backend seems to be stuck in this state and will not return good data on this sequence. It will keep returning this error indefinitely (even though it continues to work fine on other new sequences, which verifies that the OEIS is talking to it).

[gwhitney]

The underlying problem is in flaskr/nscope/views.py

    if seq.values_requested:
        return LookupError("Value fetching for {oeis_id} in progress.")
    seq.values_requested = True
    db.session.commit()
    # Now try to get it from the OEIS:
    r = requests.get(f"{domain}{oeis_id}/b{oeis_id[1:]}.txt")
    if r.status_code == 404:
        return LookupError(f"B-file for ID '{oeis_id}' not found in OEIS.")
    # Parse the b-file:

because see if it returns that LookupError the database is stuck in the state that it thinks fetching is in progress.

[gwhitney]

@Vectornaut I suspect this is still relevant and needs to be addressed in an updated #127, even in light of #164, but when you get a chance please take a look and update or close accordingly.

[Vectornaut]

I agree that this is probably still relevant. The description of pull request #127 mentions that we need to set up a similar system for repeating value requests. I don't think #164 changes that: we still fetch values from the OEIS, so the code quoted in this comment should still lock out future value requests after a value request fails.

[Vectornaut]

Since #127 should've resolved this issue for metadata, it might be useful to clarify what "trying the API" means, and keep track of which parts of the issue are still active. For example, maybe we could say:

trying API endpoints like:

• get_oeis_metadata (resolved by Allow stale metadata requests to be repeated #127)
• get_oeis_name_and_values
• get_oeis_values

These are just guesses; I haven't tried to reproduce the issue for each endpoint with the current code.