Support -ins for alpaca model in tcp server - #349
Closed
vonjackustc wants to merge 7 commits into
Closed
Conversation
Signed-off-by: Thiago Padilha <thiago@padilha.cc>
Signed-off-by: Thiago Padilha <thiago@padilha.cc>
Signed-off-by: Thiago Padilha <thiago@padilha.cc>
Signed-off-by: Thiago Padilha <thiago@padilha.cc>
The goal is to allow running llama_main while connected to other streams, such as TCP sockets. Signed-off-by: Thiago Padilha <thiago@padilha.cc>
This new mode works by first loading the model then listening for TCP connections on a port. When a connection is received, arguments will be parsed using a simple protocol: - First the number of arguments will be read followed by a newline character. - Then each argument will be read, separated by the 0 byte. - With this we build an argument vector, similar to what is passed to the program entry point. We pass this to gpt_params_parse. Finally `llama_main` will be executed with the input/output streams connected to the socket. Signed-off-by: Thiago Padilha <thiago@padilha.cc>
Change multiple printf to fprintf(outstream
Contributor
|
@vonjackustc can you change the target branch to |
Contributor
|
I did same thing but in windows, here socket stream is a nice idea, but I use thread ThreadSafeQueueto impl it. |
Contributor
|
@vonjackustc I missed these new extra |
Author
You can change LLAMA_N_PARTS from { 5120, 2 } to { 5120, 1 } to support quantized alpaca-13b-q4.bin here: https://github.com/antimatter15/alpaca.cpp#getting-started-13b |
Contributor
I have no idea what these parameters mean, but isn't this what the |
tarruda
force-pushed
the
tcp_server
branch
3 times, most recently
from
March 22, 2023 17:35
0635dc0 to
3a0dcb3
Compare
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
ddvnguyen
pushed a commit
to ddvnguyen/llama.cpp
that referenced
this pull request
Jun 25, 2026
…l-org#349 review round 2) PR #10 round-2 review found a real 🟡 P2: The naive impl caches `LLAMA_HYDRA_BUILD_TYPE` (the *derived* label) in the CMake cache. After a first configure of a fresh build dir with `-DBUILD_SHARED_LIBS=ON`, the cache has `LLAMA_HYDRA_BUILD_TYPE=shared`. On a *reconfigure* of the same build dir with `-DBUILD_SHARED_LIBS=OFF`: if(DEFINED LLAMA_HYDRA_BUILD_TYPE) # TRUE — comes from cache set(_hydra_build_type "${LLAMA_HYDRA_BUILD_TYPE}") elseif(BUILD_SHARED_LIBS) # never reached ... The first branch takes the stale cached value. The new build is *actually* static but the [shared] label survives, sails through the ggml-org#351 check-build-type gate, and the hang that ggml-org#349 exists to prevent is back in production. Fix: cache the *override* (only set by the user), not the derived label. The override is an empty string by default. The derived label is a normal (non-cache) variable recomputed every configure from BUILD_SHARED_LIBS (or the override). configure_file() picks up the freshly-derived label at configure time. Tested: -D override=experimental → "experimental" (persists) -D override=experimental, no flag → "experimental" (persists) -D override=experimental, -DBUILD_SHARED_LIBS=OFF → "experimental" (override wins) cmake --fresh → "shared" (override cleared, re-derives) -DBUILD_SHARED_LIBS=ON, then -DBUILD_SHARED_LIBS=OFF → re-derives to "static" Also: unify the label style in llama_print_build_info(). Was `type=shared`, now `[shared]` to match --version and llama_build_info() so all sinks of the label are grep-friendly with the same regex (\\[\[^\\]\\]\\]). The 🟢 P3 about llama_build_info() format change (b<N>-<sha> → b<N>-<sha> [shared]) is kept as-is. This is a private fork; the user-facing parser (--version) still works. If a downstream consumer ever breaks, the fix is to read the new field separately rather than parsing the historical format. Tracks ddvnguyen/hydra_vortex#349. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5 tasks
ddvnguyen
added a commit
to ddvnguyen/llama.cpp
that referenced
this pull request
Jun 25, 2026
fork: surface HYDRA_BUILD_TYPE in --version (hydra_vortex ggml-org#349)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change multiple printf to fprintf(outstream