Large Language Models up to 4x Faster on RTX With TensorRT-LLM for Windows | NVIDIA Blog
blogs.nvidia.com
Generative AI on PC is getting up to 4x faster via TensorRT-LLM for Windows, an open-source library that accelerates inference performance.
  • korewa@reddthat.comEnglish
    3·
    9 months ago

    Dang I need to try these for now only the stable diffusion extension for automatic 1111 is available.

    I wonder if it will accelerate 30b models that doesn’t fit all in the gpu vram.

    If it only accelerates 13b then it was already fast enough