My Adventures With Local LLMs

After more than two months of radio silence, here I am, back with a bang! Or should I say, back with a byte? A lot has shifted in the AI landscape since my last post. Back then, OpenAI was the only game in town for LLM (Large Language Models) that met our needs. Fast-forward to today, and it’s a whole new ball game, just around a week after writing previous post, Claude released worldwide, and with Gemini Models, Bard is no longer a laughingstock. And Perplexity’s integration of GPT-4, Claude, Gemini in main product. And their labs, are hosting so many models, including Llama2, Mistral and their own pplx-online models for us to try with. The best part? These models are at your fingertips, ready to run on any hardware beefy enough to handle them. And in today’s blog, we’ll dive into the hows and wheres of downloading these models, and explore some of the options available. Disclaimer: I’m no ML engineer. My journey through the world of LLMs is from the perspective of an avid user and enthusiast.

Parameters, Weights and Quantization

In the world of LLMs, size does matter! Models come in various sizes, measured in billions of parameters - 3b, 7b, 13b, up to a whopping 70b¹. But what does this mean for us mere mortals with regular computers? It’s like choosing between a scooter, a sedan, and a monster truck - each has its place, but you wouldn’t take a monster truck for a quick spin to the grocery store, right? The larger the model, the more computing power you need. So, unless you’re sitting on a data center, you might want to stick to the ‘scooter’ or ‘sedan’ models.

Quantization is another buzzword in the LLM universe. Think of it like packing for a vacation - you can either take two large suitcases or compress everything into a carry-on. Quantization is like that carry-on - it crunches down the model size (4bit, 5bit, etc.) so it can run on less powerful hardware without losing too much performance. It’s a balancing act between size and efficiency.

For end users and enthusiasts, understanding ‘b’ in LLMs is crucial. The ‘b’ stands for billion parameters - a parameter being a part of the model that has learned from the training data. More parameters typically mean a more nuanced understanding and generating capability², but also a heavier computational load. It’s a bit like having a more detailed map of a city – the more details, the better your navigation, but you’ll need a larger wall to hang it on!

Some tools to use models locally

The world of LLMs is like a digital library waiting to be explored. HuggingFace is like the librarian, offering a plethora of models from all kind of creators. Ranging from official providers like OpenAI, Microsoft, Meta, Replit etc and then there are people like TheBloke who takes the models and convert them using format and tools which can take advantage of llama.cpp.

llama.cpp 🦙💻

Imagine a world where your laptop becomes an AI powerhouse, crunching numbers and spitting out AI magic. That’s what llama.cpp offers. It’s a plain C/C++ implementation that allows the LLaMA model and their derivatives to run on your computer laptop. Regardless of the machine and chipset you’re using, whether it’s Apple’s silicon, Nvidia, ARM or even mobile chipset. It’s written in C/C++ ensuring that it caters to a broad range of hardware. It’s like turning your laptop (or mobile) into a mini-AI (Or Micro-AI) lab, minus the lab coat and goggl​​es.

You can see and play with Llama.cpp here

KoboldCpp 🐲👨‍💻

Next up, we have KoboldCpp. Think of it as the friendly neighborhood AI toolkit. It’s a simple, one-file way to run various GGML and GGUF models. KoboldCpp is like that handy multi-tool in your drawer - easy to use, and it gets the job done without the fuss. It’s perfect for those who want to dabble in AI text generation without getting bogged down by complex setups. You know, like having a personal AI butler at your command, ready to churn out text at your wish, in various modes like story generation, chat, adventure modes etc. And behind the scenes it can connect to Stability AI or Google’s TTS models to give life to your generated text.

gpt4all 🌐🤖

Lastly, gpt4all, which was my first foray into trying localized LLMs and playing with it. It’s easy to download. You can just pick one of the model they offer, along with clearly mentioned system requirements and whether your laptop is able to run it or not. It was also my first time when I (unsuccessfully) played with Retrieval Augmented Generation (AKA instead of giving a big chunk of text, you understand query, pass only relevant text after performing the search and doing the summarization, in a very simple way). Though it’s been much longer since I used it, it will still be remembered for being my first foray into localized LLMs. You can download GPT4All from here

Ollama 🦙📚🌐🔧

Then there’s Ollama, kind of like the cool, offbeat bookstore/librarian down the street. The beauty of Ollama is a simple server which can be downloaded via a single command, another command to download and start using the model of your choice, thus getting all hassles away from setting up llama.cpp or koboldcpp. And for someone who runs a lot of command line tools, getting started with Ollama is a second nature.

Hosting ourselves vs using a hosting provider

Whenever a backend is handled, there has been a debate of server-full or serverless, i.e. Hosting your own servers and getting sure it’s up and running vs using a cloud-provider’s service and just take care about the code. The server-full versus serverless debate is akin to a tech version of the classic home cooking versus dining out conundrum. When you decide to host your own AI models, it’s a bit like preparing a gourmet meal from scratch. Yes, it demands more time, resources, and a bit of technical know-how (think deciding the right hardware specifications to handle the model’s computational appetite). The satisfaction, however, comes from having complete control and customization. You’re the chef in your AI kitchen, tweaking and seasoning your models to perfection.

On the flip side, there’s the pay-as-you-use, serverless approach. Imagine it as the convenience of ordering your favorite dish from a restaurant. No muss, no fuss – just a straightforward transaction. Here, the heavy lifting of keeping the AI models up and running is handled by the provider. You don’t have to worry about the complexities of hardware requirements, maintenance, or even unexpected downtimes. It’s about trading a bit of control for ease and simplicity. Plus, you pay only for what you use – like paying for a meal, rather than buying all the ingredients and cooking tools. As I navigate through these options, I’m finding that each provider offers a unique flavor to this AI feast.

APIs for these models

Ideally Locally available LLMs are kind of Server-full deployment. We chose our machines, chose models, host it and call it a day. But there are some services which allows us to use these models without having to worry about hosting it ourselves. They host Llamas, mistrals and many models available in HuggingFace for us, in some cases with APIs compatible with OpenAI’s API.

The Perplexity is one such provider who host LLaMAs, Mistral and their own PPLX models. Then there is Openrouter who host many models. For people who like to be in AWS Ecosystem, keep a look on Amazon Bedrock hosting Llama, Claude and even Stable Diffusion models as APIs.

Some models I use regularly, with workflow

Unlike my use with OpenAI models. My daily workflow with locally hosted model is limited for a specific task, specially dealing with highly sensitive data. Llama is my go-to for analyzing private documents - think of it as my digital magnifying glass. CodeLlama is like a pocket-sized senior coder, helping me generate snippets of code. For those moments when I want to step out of the serious world and into a playground of possibilities, I turn to Mistral, Zephyr, and Dolphin. These models and modification of existing models are like my personal AI playground, each offering a unique way to push boundaries and explore the ‘what-ifs’ of AI capabilities. It’s like having an amusement park of AI at my fingertips, where I can ride the rollercoasters of innovation and see just how far I can stretch the limits of these models. This playful exploration not only adds a dash of fun to my day but also sparks new ideas and perspectives in my work with AI.

Update on tools I can’t live without

Aider has emerged as a formidable force in my coding arsenal, even surpassing GitHub Copilot in certain aspects (Ability to use GPT 4 and GPT 4 Turbo being the main aspect). This AI pair programming tool, designed for command-line use, allows for seamless integration with GPT-4 for code editing and generation. Aider stands out not only for its ability to edit code in your local git repository but also for its knack for crafting sensible commit messages, ensuring a smooth workflow. It’s a versatile companion in diverse coding scenarios. The real charm of Aider lies in its simplicity and effectiveness; it’s like having a coding wizard at your fingertips, always ready to assist when Copilot isn’t available. Whether starting a new project or refining an existing one, Aider proves to be an invaluable asset, transforming complex coding tasks into manageable, even enjoyable, experiences​​.

Closing Remarks

So, there you have it - my whirlwind tour through the land of local LLMs. It’s been a journey of discovery, learning, and a lot of fun. As I continue to explore this fascinating world, I’m reminded of the endless possibilities that AI holds. The recent drama with OpenAI Leadership and occasional outages have highlighted the importance of local LLMs. While hosted models like OpenAI, Gemini, or Mistral Medium still outperform many openly available LLMs, it’s in those moments when “the parks are closed” that these local models prove to be our true friends. They offer autonomy, control, and continuity in our AI adventures. So dive in, explore, and let your creativity flourish. The world of AI is vast and full of wonders, and it’s all yours to discover. I hope you discover it that will catapult your AI journey to new heights and with that

May the Force (of LLMs) Be With You