Machine Learning Engineer | Product Maker@atlantiq AI | Building Human-in-Loop AI Assistant for Ship Designers
2,270 followers
I am deploying my own LLM Mistral-7B-instruct with supercharged inference As I work on building a chat assistant with Mistral-7B to help customers navigate complex SAAS platform, I run into an important consideration, how will I scale and serve the LLM running the assistant. Let's look at a scenario: Using one GPU-A100 for deployment, our LLM Mistral-7B can generate 17 tokens per second. Now, lets say, if we have 1000 customers using our assistant at the same time, and average length of response from assistant is 150 tokens, putting the numbers together, our assistant will take 2 hours to process requests at anytime. An average reader's speed is 240 words per minute which we should match so our readers don't get bored but with the above setup, more than half the customers could even be waiting 1 hour to get any text at all. Not good at all for User Experience!! First, lets define the metrics we will use to assess the performance of LLM in the context of deployment: - Latency : Total time taken to process one user query. Important for better UX - Throughput: The number of tokens generated per second by the system. Important for scalability We are going to use a popular framework vLLM for optimization and benchmarking but lets look at the basic principles that vLLM leverages: 1. KV caching: - Transformer decoder architecture generates tokens sequentially and to generate a token, it uses all the past generated tokens. For each new token, a key-value vectors are generated which measures the relevance of the token to previous tokens. - So lets say, if we want to predict xth token, we will need KV vectors for 1...(x-1)th tokens, these vectors can be cached instead of regenerating them for every token, leading to time optimization with a memory trade-off. 2. Continuous batching our main optimization: - We parallelly process batches of customer queries, enhancing throughput. - However, differing response sizes in generative text lead to inefficient GPU memory use. - For examples: lets create a batch of two queries: - 'Delhi is the capital of which country?' -'Tell me about Harry potter' The first requires a brief response, while the second could be lengthy. With equal memory allocation per query, the GPU waits for the longer response to complete, leading to underutilized memory for the shorter query. This results in a hold-up of memory resources that could have been used for processing other queries. vLLM allows the efficient use of GPU memory to cache KV vectors, such that when a query in a batch is finished, another query can start processing in that batch. Observations on using vLLM on a batch of 60 queries: 1. Latency decreased more than 15x with vLLM 2. Throughput increased from 18 tokens/s to 385 tk/s 3. Throughput shows significant boost on large batches Link to reproduce results on colab: https://lnkd.in/ew_S_2WD If you are working on a similar project, you are welcome to share your experience :)