Skip to content

infill mode for main and server#3296

Merged
ggerganov merged 9 commits into
ggml-org:masterfrom
vvhg1:master
Oct 2, 2023
Merged

infill mode for main and server#3296
ggerganov merged 9 commits into
ggml-org:masterfrom
vvhg1:master

Conversation

@vvhg1

@vvhg1 vvhg1 commented Sep 21, 2023

Copy link
Copy Markdown
Contributor

This adds support for the infill tasks in ./main and ./server.
There are still a few things that should be improved, but it works as is.
TODO:

  • update Main Readme
  • if context is too big in infill mode, we probably want to cut from both sides, need to make sure that neither prefix nor suffix get cut completely
  • entering new prompts in ./main is working but a bit awkward, user should be guided a bit more and prompting layout/feedback should be improved

PS hope it helps, thanks for the great work!!

@cebtenzzre cebtenzzre changed the title vvhg-code-infill infill mode for main and server Sep 21, 2023
@cebtenzzre cebtenzzre added the enhancement New feature or request label Sep 21, 2023
@ggerganov

Copy link
Copy Markdown
Member

Should we make a separate infill example instead of adding to main?

@vvhg1

vvhg1 commented Sep 29, 2023

Copy link
Copy Markdown
Contributor Author

Should we make a separate infill example instead of adding to main?

It shares a lot with main but it might well make sense to not clutter the functionality of main too much.
I can have a look over the weekend. Shall I split the server infill part into a separate pr or do you want to keep both in the same pr?

@ggerganov

Copy link
Copy Markdown
Member

I think a separate minimal infill example would be better.
The server changes can remain as they are.

@Green-Sky

Copy link
Copy Markdown
Collaborator

whats the difference between infill/this pr and fim/#2934 ?

@ggerganov

Copy link
Copy Markdown
Member

I think they are very similar, but AFAIK #2934 does not work yet for some reason and I haven't had the time to check why. If we can make a similar example in this PR that works, then we can close #2934 and keep just the current one

@vvhg1

vvhg1 commented Sep 29, 2023

Copy link
Copy Markdown
Contributor Author

I think they are very similar, but AFAIK #2934 does not work yet for some reason and I haven't had the time to check why. If we can make a similar example in this PR that works, then we can close #2934 and keep just the current one

So more or less taking over the structure of #2934 and fill it with this example? Sounds good. What about the --interactive flag, is that valuable or do you just want the bare minimum? As it's already working with --interactive it shouldn't be any extra work to keep it.

vvhg1 and others added 3 commits September 30, 2023 13:47
* reverted changes to main and added infill example
@vvhg1

vvhg1 commented Sep 30, 2023

Copy link
Copy Markdown
Contributor Author

Ok, I'm done, no more changes in main.cpp, all moved to infill.cpp and added readme.
The conflict is easily resolved but for some reason I can't resolve it here, so I'd like to kindly ask you to do it on your end when merging.
Happy to work on it if you have further suggestions!

@cebtenzzre

Copy link
Copy Markdown
Collaborator

I tried to merge master into this PR, but your branch is protected so it won't let me:

remote: error: GH006: Protected branch update failed for refs/heads/master.
remote: error: Changes must be made through a pull request.
To github.com:vvhg1/llama.cpp.git
 ! [remote rejected]   pr-3296 -> master (protected branch hook declined)
error: failed to push some refs to 'github.com:vvhg1/llama.cpp.git'

@vvhg1

vvhg1 commented Sep 30, 2023

Copy link
Copy Markdown
Contributor Author

I tried to merge master into this PR, but your branch is protected so it won't let me:

remote: error: GH006: Protected branch update failed for refs/heads/master.
remote: error: Changes must be made through a pull request.
To github.com:vvhg1/llama.cpp.git
 ! [remote rejected]   pr-3296 -> master (protected branch hook declined)
error: failed to push some refs to 'github.com:vvhg1/llama.cpp.git'

could you try again, I unprotected my master

@vvhg1

vvhg1 commented Sep 30, 2023

Copy link
Copy Markdown
Contributor Author

Done, should all be working now.

@ggerganov
ggerganov merged commit c97f01c into ggml-org:master Oct 2, 2023
yusiwen pushed a commit to yusiwen/llama.cpp that referenced this pull request Oct 7, 2023
* vvhg-code-infill (ggml-org#1)

* infill in separate example (ggml-org#2)

* reverted changes to main and added infill example

* cleanup

* naming improvement

* make : add missing blank line

* fix missing semicolon

* brought infill up to current main code

* cleanup

---------

Co-authored-by: Cebtenzzre <cebtenzzre@gmail.com>
@amanda-peng

amanda-peng commented Oct 10, 2023

Copy link
Copy Markdown

Hi, is there any plan to implement non-stream mode for infill in server side?

@ggerganov

Copy link
Copy Markdown
Member

Hi, is there any plan to implement non-stream mode for infill in server side?

Contributions are welcome

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
* vvhg-code-infill (ggml-org#1)

* infill in separate example (ggml-org#2)

* reverted changes to main and added infill example

* cleanup

* naming improvement

* make : add missing blank line

* fix missing semicolon

* brought infill up to current main code

* cleanup

---------

Co-authored-by: Cebtenzzre <cebtenzzre@gmail.com>
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
* vvhg-code-infill (ggml-org#1)

* infill in separate example (ggml-org#2)

* reverted changes to main and added infill example

* cleanup

* naming improvement

* make : add missing blank line

* fix missing semicolon

* brought infill up to current main code

* cleanup

---------

Co-authored-by: Cebtenzzre <cebtenzzre@gmail.com>
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
* vvhg-code-infill (ggml-org#1)

* infill in separate example (ggml-org#2)

* reverted changes to main and added infill example

* cleanup

* naming improvement

* make : add missing blank line

* fix missing semicolon

* brought infill up to current main code

* cleanup

---------

Co-authored-by: Cebtenzzre <cebtenzzre@gmail.com>
AlexiAlp pushed a commit to minghaop/llama.cpp that referenced this pull request Jun 2, 2026
* vvhg-code-infill (ggml-org#1)

* infill in separate example (ggml-org#2)

* reverted changes to main and added infill example

* cleanup

* naming improvement

* make : add missing blank line

* fix missing semicolon

* brought infill up to current main code

* cleanup

---------

Co-authored-by: Cebtenzzre <cebtenzzre@gmail.com>
AlexiAlp pushed a commit to minghaop/llama.cpp that referenced this pull request Jun 2, 2026
* vvhg-code-infill (ggml-org#1)

* infill in separate example (ggml-org#2)

* reverted changes to main and added infill example

* cleanup

* naming improvement

* make : add missing blank line

* fix missing semicolon

* brought infill up to current main code

* cleanup

---------

Co-authored-by: Cebtenzzre <cebtenzzre@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants