Skip to content

Added script to invoke alpaca model#273

Closed
nullhook wants to merge 2 commits into
ggml-org:masterfrom
nullhook:master
Closed

Added script to invoke alpaca model#273
nullhook wants to merge 2 commits into
ggml-org:masterfrom
nullhook:master

Conversation

@nullhook

@nullhook nullhook commented Mar 18, 2023 •
edited
Loading

Copy link
Copy Markdown

Resolves #240

WIP

This needs to be able to:

  1. Configure custom model folders.
  2. Adjust settings for running variants of the Alpaca model and make corresponding changes in the C++ side.
  3. Provide the ability to configure parameters.
  4. Possibly hide the extraneous template it returns.
  5. Ability to run interactive mode

@ggerganov

ggerganov commented Mar 19, 2023

Copy link
Copy Markdown
Member

I made some changes to on master already to support the interactive mode where we inject ### Instruction: prefix and ### Response: suffix in the input. I don't know if this is really needed, but this is what the alpaca.cpp guys are using, so I added it.

I think our original --interactive mode actually works better, but don't have too much evidence.

If you adapt this python script to the new changes and update the README with usage instruction - we can merge.

@gjmulder gjmulder added enhancement New feature or request model Model specific 🦙. labels Mar 20, 2023
ggerganov
ggerganov requested changes Mar 21, 2023
View reviewed changes

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adapt to recent changes. See alpaca.sh for example how to turn on the --instruct mode

@ggerganov

ggerganov commented Apr 13, 2023

Copy link
Copy Markdown
Member

I think this got a little outdated.
If this is still of interest, please move the script to the examples folder and add some brief usage instructions to the README file and reopen

@ggerganov ggerganov closed this Apr 13, 2023
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
@ikawrakow
FlashMLA-3: the best of both worlds (CPU only) (ggml-org#273)
24e780b 
* Repack a model with the quantize tool

* WIP

* Fixed various issues

As we don't have a way to tell if a repacked quant has been modified,
I had to remove the modification at the expense of a slight decrease
in performance. This affects q8_0_r8, q8_KV_r8, q8_k_r8 on Zen4, and
q4_0_r8 on ARM.

* Create wk_b and wv_b as Q8_0_R8 if the wkv_b type is interleaved

* Fix GCC 13.3 compilation error

* Another one

* Add missing include

* FlashMLA-3: the best of both worlds - CPU only

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
@hydra-z hydra-z Bot mentioned this pull request Jun 26, 2026
Closed
@ddvnguyen ddvnguyen mentioned this pull request Jun 26, 2026
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov requested changes

Assignees

No one assigned

Labels

enhancement New feature or request model Model specific

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add instructions for using Alpaca

3 participants

@nullhook @ggerganov @gjmulder