Last week I set up my desk, I was about to finish a project I'm working on for a couple of weeks, and as I was wrapping things after with this tiny script that simply won't run. I was trying to make my package version bump on every release. I did the same thing countless times before, but this one just refused to work. I pushed through and kept fighting with it to exhaustion after two hours into the night. Looking back, I thought, God, this shouldn't have taken that long. Noting 2023, understood I've been going through the same process more times than I've noticed, so today I'm solving that for good. I'm going to use one of the most exciting projects out there to run my local language model and speak to it directly, and the best part about it, it's completely free. Meet old llama. To install, go to old llama ai, hit download and extract the file, it'll open a simple UI to greet you and help you install the CLI. When you're ready, go ahead and test it out and find your first model. Let's zoom on models for one sec. You can find a bunch of them on the old llama website and choose what fits the workload or train your own. Mistral and llama too are the most commonly used, but, the astral claims it's even stronger than llama too, which meta claims it's even better than chachi bitti. You can also find cool projects like llama to unsensored, where the author removed common obstacles when trying to ask the model for data it's not supposed to share. This doesn't have to be like how to make a bump type of stuff. Some models will be reluctant to help you create a social media bot, claiming that social media automation is wrong. I guess they didn't hear about both statistics on Instagram in any case, not that every model offers an API and CLI access, so you can easily wrap your own solution, stick around because we're going to see something that does exactly that. What I love about old llama is that it resembles docker in many, many ways. It's basically mimics the environment of docker and brings it into the world of AI. Now, as an ops person, I can really appreciate that. That's basically taking something completely out of my field of expertise and bringing everything that I feel familiar with. I can list, run, attach, even delete, start and stop models as if I'm running containers. It's basically the same commands, just using a different CLI, and it feels like I'm basically just using containers on my local machine, which is amazing. After listing the running models, I can delete the one that I'm not using and then attach to llama too, so we can start working with it. Let's ask a rather naive question, whether I should be trying out its uncensored model or not. And of course, my very responsible model is parenting me with the empathy. As with any other language, I can play with it and ask random questions and get the random answers that I was asking for. Like whether it's possible to leak your own elbow or not, nevermind. Before settling into the idea of running my AI model locally on a normal basis, I'd like to first make sure it's not taking too much resources. I was trying to visualize what was going on with Prox, and it seems like it's not moving the needle. Not sure whether I'm missing a critical parameter or that the model is already consuming its resources simply because it's being held running. Please comment if you think you know. For now, I think I'm okay with keeping it locally and I'll show you why. Remember how I said that everything within llama resembles Docker for me? Well here's another great reason. llama has the concept of using model files, which may sound familiar or be familiar for you from Docker files. They look the same, they feel the same, they even use the same notions to build different files and run models. I'll show you how. Let's go ahead and create one. I'll call this one the magician. Just like in Docker files, we use a from statement so it will inherit from something else, in this case llama too. I'm adding other parameters like the temperature, which basically directs the model how much it should be created with its answers and then a context for the answers I'm expecting. In this instance, I want it to be a magician that entertains its users. As you can see, I'm having hard time keeping a straight face and not being drawn into a conversation. How about we do something useful with it? Let's run llama's model that's trained specifically for coding tasks. Just like with Docker, the run command also pulls the model if it's not present locally and there's labels that you can use for different revisions. Unlike container images, LLM models are quite heavy and several gigabytes take a while to find their way. Once it's ready, I can start asking it code related questions, like for example, how to bump my package JSON. Let's try the suggestion to know whether it's trustworthy. After bumping, I'll reload my package JSON and the version was indeed bumped to the next major v2.0. You can also see my Starship prompt now shows package and the updated version. If you want to learn more about Starship, click the video above. If you recall, from the beginning of the video, I was presented with a weird error when publishing my bumped version. Let's see if code llama can handle my error. Like all good devs, I'll just throw the error right at it. It offers forcing my way through, which honestly is not the best idea, although it will work or an alternative solution to pump on minor and then push that. That's more like it. I don't know about you, but it feels like it's not quite smooth. David from DevonDuty, shout out to David, I'll leave his channel below. He built a plugin called JEN and VIM, and that's basically interacting with old llama under the hood to provide answers or grab the context from within your VIM and then offer the solutions back to my working space. To install JEN and VIM, I'll head over to my lazy plugins list and edit. On my next new VIM reload, lazy takes care of the installation and the JEN command is available to me with all the prompts baked into it. If you're unfamiliar with lazy, check out the video above to learn more. We'll get into these options in a sec, but let's see how JEN works beautifully with real context. Let's start simple by just writing one sentence, which I can then mark and send to JEN to enhance my writing. Many of the prompts are configured to replace the original code slash text that sent to them, so JEN replaces my line with its better wording. Additionally, I can make a list or a markdown table out of my text. This is very useful when writing readmies and markdown-based documentation files. To make sure the table is marked and valid, I'll shoot it over to below to see how it's printed in my terminal. Looking good? I think we're ready to extend JEN and VIM with our own prompt and actually useful one. So you can add just like this as many prompts as you want to suit daily requirements while working in new VIM. To add a prompt, I start by telling my main plugins file that there is a new AI file to load. Check out my lazy plugin setup if you're interested in more than that. After that, I use the template from the JEN and VIM's plugin prompt file. I'll give it a title that will show up on my menu within your VIM when we're done. The two other settings it has is the prompt, which will shortly provide, and replace Equidestrew, which for the sake of example I chose to live on. For my prompt, I'll be using the nice example for a DevOps engineer from the Olamma Model Finance Library. I can grab the prompt used for its context and use it in my own setup. I'm adding a break line followed by dollar text, which the plugin then interprets as a required input. Let's see this in action. I find my DevOps me command, and since we've mentioned it requires an input, then the plugin asks me for a prompt. I've actually changed the name here as it's not really a prompt. The context was already given to it by the model. We just add an input that goes with it. Let's ask for a terraform file that deploys Olamma in the cloud for us in case we want to run it elsewhere. The input line might need a break line or some UI fixes, but it takes my payload eventually. After some required cleanup of comments and explanations, I'm left with the code. And what can I tell you, code Llamma isn't great with terraform. Lots of either missing configuration or improvements in code have done better. For example instead of forcing me to find an image for my instance, just bring something that's generally considered as best practice. I might have to train my own DevOps model eventually, but see, I'm going to give it another go with Mistral. This time, I want to focus on having Olamma on the instance when it starts. So I'll rewrite my prompt to support that. And sure enough, Mistral comes back with a script that's embedded in the template. It could have been simpler, but this should work or at least provide a framework to work in, which is kind of nice to be honest. I think this is worth a spotlight. With DevOps specifically when it comes to AI prompts are everything. Whatever you provide, this is what the system knows. You can see the results. This just goes to say that nobody or no system is going to replace all engineers anytime soon. What you can be replaced with is another engineer that is making use of an AI correctly. As you can see, the output will never be perfect. As long as the model is not trained on your system in your environment and your routing and wiring and networking and everything that goes within your environment, you have to provide the context in which it works. You will still have to take the result and tweak it to your needs. I'm going back for CodeLama for my last and best attempt by asking both a good solution for the image, including the script I got from Mistral. It's not perfect and I do find myself having to combine the two for it to run. That said, it is deployable eventually with Terraform and I did get a remote instance that I can log into and work with in the cloud. By the way, if you're interested in the Terraform example code to deploy your own, please comment below and I'll share a good one that works with best practices and batteries included as far as accessing the instance and the model API goes. OLama works great as is, but as with many thriving communities, there is quite a few projects you want to check out. Go to the GitHub page and scroll down to the community integrations where you'll find open source UI options, SDKs for other languages, and the coolest part in my opinion, plugins for common tooling like Obsidian, Discord and so on. The one that immediately caught my eye was the Raycast extension. If you're a Mac user and still don't know about Raycast, please check it out. I'll leave a link below, but in a nutshell, it's the best spotlight replacement. It can do an incredible amount of things. It's free and it has an extension store beyond throwing confetti on my screen, which is great. We can install any existing extension with just a few clicks. I'm going to the extension store within Raycast and look for OLama. I hit enter and it's installed. Simple as that. Now I can start interacting with OLama directly from Raycast. When it opens, it lets me choose the existing models. I have on my machine, it runs them if required, and opens a UI for conversing with the model. Now, I'm using Raycast to toggle Google searches. Imagine the ability to get the answer directly without even having to open a browser. Thank you, Mistral. Someone should have named you that jokes AI. Yep, not improving. I think that's enough. So this is exciting, but it doesn't mean much if you don't master your VIMS case. Once you do, it's a bulletproof combination with Jen and VIM. So check out the next video right here so you can start using Envim like a pro. Thank you for watching and I'll see you on the next one.
