Scraper mostly (if not always) fails to get answers to questions

[satyamtg]

The scraper mostly fails to find answers to the questions while offlining the course https://courses.edx.org/courses/course-v1:edX+edx201+1T2020/course/

This happens in https://github.com/openzim/openedx/blob/master/openedxtozim/xblocks_extractor/Problem.py

WARNING:root: fail to get answers to this problem : block-v1:edX+edx201+1T2020+type@problem+block@1930fe52a0f14aa08b3549607341fcc6 (https://courses.edx.org/courses/course-v1:edX+edx201+1T2020/jump_to/block-v1:edX+edx201+1T2020+type@problem+block@1930fe52a0f14aa08b3549607341fcc6)
WARNING:root: fail to get answers to this problem : block-v1:edX+edx201+1T2020+type@problem+block@1465b47933174ea38a9e5e4e6c0fd2a3 (https://courses.edx.org/courses/course-v1:edX+edx201+1T2020/jump_to/block-v1:edX+edx201+1T2020+type@problem+block@1465b47933174ea38a9e5e4e6c0fd2a3)
WARNING:root: fail to get answers to this problem : block-v1:edX+edx201+1T2020+type@problem+block@2627fa3674dd4724b2ee777f740b920d (https://courses.edx.org/courses/course-v1:edX+edx201+1T2020/jump_to/block-v1:edX+edx201+1T2020+type@problem+block@2627fa3674dd4724b2ee777f740b920d)

[kelson42]

Seems quite critical to me.

[dattaz]

Not so critical IMO, having question is still very useful.

If I remember correctly it depends on the moocs, but some don't allow to see the answers (unless previously answered or previously answered correctly).

[kelson42]

@dattaz @satyamtg What is the impact of this problem? I believe this is broken links (or links pointing to online version?).

[satyamtg]

@dattaz @satyamtg What is the impact of this problem? I believe this is broken links (or links pointing to online version?).

@kelson42 not really. We do offline the quiz, its a fully working quiz. It's just we don't have the answers and hence, we clearly mention in the quiz that we do not have answers for the quiz. Here's a sample of output when we have the answers and when we do not have them.

[kelson42]

Do we have a strategy to retrieve properly the answers?

[satyamtg]

Do we have a strategy to retrieve properly the answers?

At the moment we rely on the show answer button to get the answers. And that is shown by the instances once you try 3-4 times (didn't count that). So, one way is to try several times and then scrape, or other way would be to automate the clicking. (Have to explore more on this, but a selenium based thing is not an option here)

[satyamtg]

Okay, so basicallyI've investigated the issue and have understood the flow properly. We currently rely on the problem_show output (which is where the request goes when the user clicks the show answer button). However, this button is not available every time (for eg. not available in the PHZH problems, as they are self-graded). Also, when it's available, it only appears if the question has been correctly answered by the user.

Thing is, we can do a post request to problem_check with the answer as the payload to get it checked, and hence to get the answers. This would mean more requests. Like say we have a multiple choice question with a single answer correct, we would need to a POST request for every possible answer. It might not seem a big deal for MCQs with a single correct answer, but it really becomes a problem for questions with multiple correct answers. We would need to try with all possible combinations unless we get a correct combination.

Another thing is that the request would return the content for the whole problem xblock, and this content may differ across instances. So, either we can process the whole content and change it when the user clicks submit, (which would allow us to handle and present the content in the same way the instance has it), or we can find the content that's mostly changed in various instances, and change only those. The former method would take longer processing time and would involve larger changes to current problem xblock_extractor. The latter would need us to have a small list of possible changes in xblock content, and finding them in the returned HTML. I think the latter one is a better approach as that would save some processing time and code changes while keeping the scraper generic enough.

What are your thoughts @rgaudin ?

P.S.: It's important to note that we would not be able to fetch answers for problems that require us to pay for the courses (unless you have actually paid for the services), and the problems for which deadlines are over (unless you've reset the deadlines)

[rgaudin]

• Can you reliably detect that you've received the answers?
• Is there a standard way to return/display the results?

For that to work, you need to have a semantic versions of the answers and be able to submit those one after another, and check whether it triggers the display of the responses ? Or do only the combinations of all questions trigger a success?

I don't think tis approach is doable as a simple single choice selection in a 10 questions form is already 3M+ possible combinations…

[satyamtg]

• Can you reliably detect that you've received the answers?

Yup. We can detect that. The request sends the response in a very specific kind of format

• Is there a standard way to return/display the results?

No. The response gives us two things -

1. A boolean value telling whether the answers are correct
2. The HTML to replace the problem div with

For that to work, you need to have a semantic versions of the answers and be able to submit those one after another, and check whether it triggers the display of the responses ? Or do only the combinations of all questions trigger a success?

The options contain a value attribute that is sent in the request. We need not need the semantic answers.

I don't think tis approach is doable as a simple single choice selection in a 10 questions form is already 3M+ possible

I think it's not that many requests (though it's pretty large). requests are sent on a per question basis and not for the whole form. So, if you have 10 questions, each containing 4 options and having only one option correct for each one, we would need only 4 requests at max per question, i.e., we would do in <=40 requests. For other kinds of questions though, that may be higher, like if there's multiple options correct.

I tried doing this approach, but it also seems that the problem xblocks have many different kinds, like there's textboxes, there's dropdown based problems (for which there'll be really large number of requests), and there's checkboxes, and also radio buttons. I tried fixing the problem xblock thing last night but it seems that there's quite a large variety to handle (plus the content changed by the instances).