ExLlamaV2
A fast inference library for running LLMs locally on modern consumer-class GPUs - turboderp-org/exllamav2
There was an error while loading. Please reload this page. ExLlamaV2 is an inference library for running local LLMs on modern consumer GPUs. The official and recommended backend server for ExLlamaV2 is TabbyAPI, which provides an OpenAI-compatible API for local or remote inference, with extended features like HF model downloading, embedding model support and support for HF Jinja2 chat templates. See the wiki for help getting started. The dynamic generator supports all inference, sampling and speculative decoding features of the previous two generators, consolidated into one API (with the exception of FP8 cache, though the Q4 cache mode is supported and performs better anyway, see here.) The generator is explained in detail here. See the full, updated examples here. Some quick tests to compare performance with ExLlama V1. There may be more performance optimizations in the future, and speeds will vary across GPUs, with slow CPUs still being a potential bottleneck: To install from the repo you'll need the CUDA Toolkit and either gcc on Linux or (Build Tools for) Visual Studio on Windows). Also make sure you have an appropriate version of PyTorch, then run: A simple console chatbot is included. Run it with: To install the current dev version, clone the repo and run the setup script: This will install the "JIT version" of the package, i.e. it will install the Python components without building the C++ extension in the process. Instead, the extension will be built the first time the library is used, then cached in ~/.cache/torch_extensions for subsequent use. Either download an appropriate wheel or install directly from the appropriate URL: A PyPI package is available as well. This is the same as the JIT version (see above). It can be installed with: ExLlamaV2 supports the same 4-bit GPTQ models as V1, but also a new "EXL2" format. EXL2 is based on the same optimization method as GPTQ and supports 2, 3, 4, 5, 6 and 8-bit quantization. The format allows for mixing quantization levels within a model to achieve any average bitrate between 2 and 8 bits per weight. Moreover, it's possible to apply multiple quantization levels to each linear layer, producing something akin to sparse quantization wherein more important weights (columns) are quantized with more bits. The same remapping trick that lets ExLlama work efficiently with act-order models allows this mixing of formats to happen with little to no impact on performance. Parameter selection is done automatically by quantizing each matrix multiple times, measuring the quantization error (with respect to the chosen calibration data) for each of a number of possible settings, per layer. Finally, a combination is chosen that minimizes the maximum quantization error over the entire model while meeting a target average bitrate. In my tests, this scheme allows Llama2 70B to run on a single 24 GB GPU with a 2048-token context, producing coherent and mostly stable output with 2.55 bits per weight
Recall.ai
Recall.ai provides an API to get recordings, transcripts and metadata from video conferencing platforms like Zoom, Google Meet Microsoft Teams, and mo
Record video conferences through a bot joining the call. Best when explicit recording consent is needed, or for building AI agents. Record video conferences and in-person meetings through a desktop app, without a bot in the call. Best for a stealthier recording experience. “Legal tech is high stakes. Working with Recall.ai, we had scalable Zoom meeting support ready in under two months, rather than six." “Recall.ai allows us to build meeting recording features without worrying about infrastructure. It has helped us move faster than we could have with an in-house build." "We’re building an AI Scribe for our doctors, and Recall.ai was the first piece of infrastructure I pushed to bring in. I’d seen how seamlessly it handled meeting data at my last company, which made choosing it again an easy call." "Recall.ai's Meeting Bot API saved us from months of pain. One integration, extremely reliable, and we launched our meeting bot feature in days." “Once we started using Recall.ai's Desktop Recording SDK to power Mem’s meeting recording experience, the painful edge cases that we had to chase on the support side went to zero." “Recall.ai allows us to operate reliable, enterprise-scale meeting transcriptions without worrying about infrastructure or security."
ExLlamaV2
Recall.ai
ExLlamaV2
Recall.ai
Pricing found: $38
Only in ExLlamaV2 (10)
ExLlamaV2
Recall.ai
ExLlamaV2
Recall.ai
