gguf : embed files to gguf model file

[katsu560]

see Embed yolo files 831 ggml-org/ggml#831

Some app like ggml/yolov3-tiny needs additional files to execute such as label(coco.names) and alphabet labels(100_0.png, ...) files.
If these files are embedded to a model(gguf) file and the app read them from the model file, the app is more portable.

I added below

• expanded gguf-py to support NAMEDOBJECT, constants.py, gguf_reader.py, gguf_writer.py
  • please see pull request to llama.cpp
• added gguf-addfile.py script to add files to gguf file
  • add files as NAMEDOBJECT (general.namedobject.N) or add files as NAMEDOBJECT array (general.namedobject[N] with --array option)

see ggml ggml-org/ggml#831

• added new GGUF_TYPE_NAMEDOBJECT with name(file path) and value(file body) for adding files to gguf
• expanded ggml to support NAMEDOBJECT, ggml.h ggml.c
• expanded yolov3-tiny to read coco.names and alphabet labels from gguf file,
  • at first read from gguf, then read from file if failed from gguf

NAMEDOBJECT constructed from name(file path) and value(file body)

    struct gguf_nobj {
        uint64_t nname;  // length of name
        char   * name;   // name in utf8
        uint64_t n;      // length of data in bytes
        char   * data;   // data body (file body)
    };

script usage:

python3 gguf-addfile.py [--array] input-gguf-file output-gguf-file files ...

• add files as NAMEDOBJECT (general.namedobject.N)
• add files as NAMEDOBJECT array (general.namedobject[N]) with --array option

[CISC]

Why make a copy of gguf-new-metadata.py (and merging it with gguf-dump.py for some reason) instead of just adding this functionality to it?

[katsu560]

finally, I added file data as follows;

• store the file path as key with starting '/' to avoid from conflicts to other key names.
  ex. storing file 'data/coco.names' as '/data/coco.names'
  if storing absolute file path '/a/b/c' as '//a/b/c'
• store the file contents as GGUF_TYPE_STRING's value.
  So, I deleted all NAMEDOBJECT part.

I wrote script based on gguf-new-metadata.py.
I deleted dump code from gguf-addfile.py

[katsu560]

Why make a copy of gguf-new-metadata.py (and merging it with gguf-dump.py for some reason) instead of just adding this functionality to it?

my idea is adding files as metadata.
so, gguf-new-metadata.py is good starting point for me.
now, I deleted dump code from gguf-addfile.py.

[CISC]

Why make a copy of gguf-new-metadata.py (and merging it with gguf-dump.py for some reason) instead of just adding this functionality to it?

my idea is adding files as metadata. so, gguf-new-metadata.py is good starting point for me.

Sure, but why make a new script when you can just add this functionality to gguf-new-metadata.py, it's what it's for. :)

[katsu560]

move to Embed files #8121