TensorRT-LLM adds a slew of new performance-enhancing features to all NVIDIA GPUs.
Just ahead of the next round of MLPerf benchmarks, NVIDIA has announced a new TensorRT software for Large Language Models that can dramatically improve performance and efficiency for inference processing across all NVIDIA GPUs. Unfortunately, this software came too late to contribute to the company’s MLPerf benchmarks, but the open source software will be generally available next month. We will opine on how the impact of this software could impact MLPerf results when they are released.
Software can have a massive impact on the performance of GPUs, and TensorRT has been the optimization engine for NVIDIA inference processing for years. Now, the company is applying new techniques specifically for LLMs to TensorRT, and the impact is dramatic. While the H100 is four times the performance of the previous A100, based on benchmarks for the GPT-J 6B LLM inferencing, the new TensorRT-LLM can double that throughput to an 8X advantage for JPT-J and nearly 4.8X for Llama2....
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
CVS Health Shuffles C-Suite Responsibilities, Adds New Roles And New HiresCVS Health shook up its executive suite, adding responsibilities for several members of the company’s executive leadership team.
Read more »
Nvidia is the reason why AMD's new GPUs are so goodAMD's new RX 7800 XT and RX 7700 XT have been met with a mixed reception, so why do we still recommend them? It's because of Nvidia.
Read more »
Visinema, Indonesian Studio, Sets Busan Double Bill, Adds New Management; ‘This Is Just the Beginning’ (EXCLUSIVE)Indonesian production firm is setting its sights on becoming a diverse studio operation with the injection of high-profile management and the acquisition of a significant new animated series “Nussa…
Read more »
Nvidia Stock’s Blockbuster Gain Is a ‘Big Market Delusion’Rob Arnott, a quantitative-investing pioneer known for steering investors out of complacency, says the chip maker is “a great company priced beyond perfection.”
Read more »
AMD RX 7800 XT vs. Nvidia RTX 4070: a clear winnerAMD's RX 7800 XT is here to rival Nvidia's RTX 4070. How do these two graphics card compare, and which one is the best? We know the answer.
Read more »