fetchThreadMessages returns graphQL 500 error limit too high

[KGALLET]

Description of the problem

I'm currently writing a script which fetch every messages of a specific thread_id using your API.
This script is made in order to download every pictures that I sent in the past to the specific thread_id conversation.
My conversation reach almost 100k messages.
When I use my script with a limit parameter for the fetchThreadMessages which is not too high, it's ok. But as soon as I used a limit really high, I got a graphQL 500 error. (example: 50000)

Code to reproduce

#!python3
from os import mkdir
from os.path import isdir, join
from datetime import datetime

from fbchat import Client
from fbchat.models import ImageAttachment
import urllib3

http = urllib3.PoolManager()

def download_pictures(user, password, thread_id, output_folder, before_tstp, msg_limit=100):
    if not isdir(output_folder):
        mkdir(output_folder)

    client = Client(user, password)

    msg_list = client.fetchThreadMessages(thread_id=thread_id, limit=msg_limit, before=before_tstp)
    for msg in msg_list:
        for att in msg.attachments:
            if not type(att) == ImageAttachment:
                continue
            if (att.original_extension) == 'gif':
                continue     
            url = client.fetchImageUrl(att.uid)
            data = http.request("GET", url=url, preload_content=False)
            data = data.read()
            dt_object = datetime.fromtimestamp(int(msg.timestamp) / 1000).strftime('%Y-%m-%d')
            output_filename = join(output_folder, dt_object + " | " + att.uid + "." + att.original_extension)
            print("Downloading %s" % output_filename)
#            last_timestamp = msg.timestamp

            with open(output_filename, "wb") as f:
                f.write(data)

if __name__ == '__main__':
    from argparse import ArgumentParser

    parser = ArgumentParser()
    parser.add_argument("email", help="Email of your facebook account")
    parser.add_argument("password", help="Password of your facebook account")
    parser.add_argument("thread_id", help="ID of your conversation")
    parser.add_argument("--output", '-o', help="Output folder where pictures will be stored", default=".")
    parser.add_argument("--before", "-b", help="Before which timestamp should we download")
    parser.add_argument('--limit', "-l", help="Limit on the number of messages to search for pictures", default=100)

    args = parser.parse_args()
    download_pictures(args.email, args.password, args.thread_id, args.output, args.before, args.limit)

Traceback

Traceback (most recent call last):
  File "download_messenger_pictures.py", line 60, in <module>
    download_pictures(args.email, args.password, args.thread_id, args.output, args.before, args.limit)
  File "download_messenger_pictures.py", line 28, in download_pictures
    msg_list = client.fetchThreadMessages(thread_id=thread_id, limit=msg_limit, before=before_tstp)
  File "/usr/local/lib/python3.7/site-packages/fbchat/_client.py", line 783, in fetchThreadMessages
    j = self.graphql_request(_graphql.from_doc_id("1860982147341344", params))
  File "/usr/local/lib/python3.7/site-packages/fbchat/_client.py", line 185, in graphql_request
    return self.graphql_requests(query)[0]
  File "/usr/local/lib/python3.7/site-packages/fbchat/_client.py", line 177, in graphql_requests
    return tuple(self._post("/api/graphqlbatch/", data, as_graphql=True))
  File "/usr/local/lib/python3.7/site-packages/fbchat/_client.py", line 134, in _post
    content = check_request(r)
  File "/usr/local/lib/python3.7/site-packages/fbchat/_util.py", line 156, in check_request
    check_http_code(r.status_code)
  File "/usr/local/lib/python3.7/site-packages/fbchat/_util.py", line 171, in check_http_code
    raise FBchatFacebookError(msg, request_status_code=code)
fbchat._exception.FBchatFacebookError: Error when sending request: Got 500 response.

Environment information

• Python3
• fbchat 1.7.3

[madsmtm]

For your specific use case, you should have a look at #434 (will be included in the next release), which implements the feature using Facebook's APIs.

But otherwise, this is a bug, probably just the endpoint which fails, since it's handling such a large amount of data. I'll investigate it further at some point!

[mlodybercik]

If you don't really need it right now you might want to use facebook's archivizer tool. It'll take a few days to process but you'll get all images, files, sounds and attachments in one go. I used it some time ago to collect all my messenger data from 2013 till recent. It took about 7 days to collect, the zip with all my files, in low quality took 7GB.

[KGALLET]

@madsmtm thank's for your fast answer. I will look at the MR today.
Do you know what the roadmap of the api ? (I mean, when the next release will be available?)

@krzesu0 I already thought about this solution, but in this case, it will download EVERY data from every conversation. My use case is to download pictures (and maybe videos later) of a specific one conversation.
And as we send lots of pictures day per day with the people of the thread, I will use my script with a cron to download pictures day per day, and to send it to my server which host my private cloud. So this solution may be cool for one day, but as I need to do it often as I can, it is not the best solution imo. But thank's for your advice :)

[KGALLET]

For those who come in this issue and want a solution, you might try to find the timestamp of the first message of your thread, and loop on the until your timestamp reach this one using my script using the before args of the fetchThreadMessages. A bit tricky but should work.

[mlodybercik]

For now you could temporarily clone my fork and use it for this purpose. Something like this should work fine:

urls = []
for i in client.fetchThreadImages(id):
    if isinstance(i, ImageAttachment):
        url.append(i.big_preview_url)

[KGALLET]

@krzesu0 imma give it a try asap :) thank's for your work

[KGALLET]

Ok, did what I want using your script @krzesu0
Thank's @krzesu0 @madsmtm

[madsmtm]

@KGALLET, sorry for the long delay, I've been on summer holidays, but I can probably find the time to make a new release in at least the coming week 😉

[madsmtm]

Aaand just notifying you that I've now made a new release, v1.8.0, in which this addition is included 😊

[KGALLET]

Just saw your message @madsmtm , love it :)
Thanks