Speed compare with vllm?

[lucasjinreal]

Speed compare with vllm?

[jdemouth]

We do not plan to publish performance numbers that compare TensorRT-LLM with vLLM.

Our internal measurements show that TensorRT-LLM’s in-flight batching and paged KV cache features work well and TensorRT-LLM can deliver great performance. We’d be happy to provide you with performance numbers for relevant cases.

Is there a particular use case that you’d be interested in?

[michaelfeil]

@lucasjinreal Found this tweet - https://twitter.com/HamelHusain/status/1719872352694174093 comparing tensort-llm + triton and vllm + triton. That said, there are a lot of aspects, so any engine would find a scenario (batch_size x latency / throughput x hardware) where it beats all other engines.

[StephennFernandes]

does TensorRT LLM also support contiguous batching like vllm does ?

[dongs0104]

@StephennFernandes

trt llm support in-flight batching same as contiguous batching

[parano]

BentoML team recently did a benchmark comparing vLLM, TRT-LLM, TGI: https://www.bentoml.com/blog/benchmarking-llm-inference-backends

With the current releases running Llama3 8b and 70b quantized, TRT-LLM shows much better token generation rate overall, but vLLM has better time to first token.

[datalee]

BentoML team recently did a benchmark comparing vLLM, TRT-LLM, TGI: https://www.bentoml.com/blog/benchmarking-llm-inference-backends

With the current releases running Llama3 8b and 70b quantized, TRT-LLM shows much better token generation rate overall, but vLLM has better time to first token.

LMdeploy niubi！