Mixtral 8x7B DESTROYS Other Models (MoE = AGI?)

it worked look at that oh my God it actually got this one right okay here we go oh my God it got it right I can't get over this it's so good on Friday mistal AI dropped a mysterious torrent link with no context whatsoever and it got the entire AI World talking and within a short amount of time we knew what it was it's a new model from mistol AI it's called mixol and it is a mixture of experts impl mentation which takes eight separate models that are all experts at certain things and puts them all together into a single model and if you're not familiar with mistol AI they're the company behind the mistol 7B model which is probably the best open-source model out there and it's only 7 billion parameters and as you could see right here within minutes Eric Hartford replied with 8 * 7B sounds like mixture of experts Moe mixture of experts and within the following hours and over the weekend we have a bunch of new information about about the model we're going to go over all of it then I'm going to show you how to actually use it it's not straightforward and then I'm going to do some testing so let's go now if you want to know about mixture of experts and what this technique actually is hugging face dropped an incredible blog post about it it's super technical so I'm not going to dive too deep into it and in fact I'm still trying to digest all of it but the gist is that you have multiple models and depending on the prompt it'll tap only a subset of those models to actually do the inference and it has a router that chooses the model which is best suited to respond to the prompt now with mixol specifically it's using eight separate models and when it's actually time for inference it chooses two models to actually do the inference so a model which is about the size of a 60 billion parameter model when you combine all of the eight together really outperforms llama 270b which is a 70 billion parameter model while being about four times faster because remember it's not actually using the entire higher model it's just using a subset of the model because it's using two of the eight so a very high level explanation of what's going on here prompt goes in router chooses different models to use then it puts them together and you get the response very basic explanation I'm probably not doing it justice thank you to the sponsor of today's video e draw mind AI the ultimate mind mapping software that goes beyond the ordinary unlocking a world of creativity and efficiency if you're like me you have a million ideas and sometimes they get scattered all over the place and so I love mind mapping software for this reason but e draw mind isn't just about brainstorming and jotting down notes it is a revolutionary tool that takes your creativity to new heights and it does this with the power of artificial intelligence imagine you just have this rough idea and then with one click you can evolve that idea again and again effortlessly building upon your initial idea in real time e draw Minds AI doesn't just follow it leads you it helps you come up with these ideas and evolve them and the AI gives you Smart Suggestions and this is all thanks to its dead simple interface and smart AI guidance and you can collaborate with your entire team click here the share button and you can add everybody you want and if you want to easily convert it into a PPT file using AI you can do that so give edraw mind AI a try Unleash the Power of AI with your ideation and visualize your thoughts like never before so try e draw mind it is the best mindmapping software out there give it a try let me know what you think and thanks again to edraw mind for sponsoring this video and if you remember a few months ago George Hots the prolific programmer the founder of comma AI he also built tiny grad and he's very deep in the AI space basically leaked that open AI was using a mixture of experts for chat GPT and specifically they were also using eight separate models combined into one and specifically GPT 4 is 8 * 2 220 billion parameters for a total of 1.7 trillion parameters and here smouth basically also confirmed it and smouth co-founded and led pytorch at meta and smouth confirms he says I might have heard the same GPT 4 8 * 220 billion experts trained with different data task distributions and 16 iter interface so that's what they're doing and now mistol basically created a much smaller version an open weight version of that and one other thing to come out of all this news over the week is mistl actually has a mistl medium so mistel tiny is the 7B model mistal small is the mixl model and then they have a higher-end version mistl medium our highest quality endpoint currently serves a prototype model so basically the only way to get access to this is by using their paid inference and if you want me to try that out let me know in the comments below I'm happy to do a test of that but what we're going to be testing and setting up today is the mistol small also known as mixol and the co-founder and chief scientist of mistol AI I finally put some information out there about mixol and this was as of just a few hours ago so if we open it up we can actually see mixol 8 * 7B compared to GPT 3.5 and compared to llama 270b on a bunch of different benchmarks right here and if we look at the Mt bench Benchmark for instruct models it is performing on par with GPT 3.5 and it far exceeds llama 270b but across the board pretty much in every single Benchmark mixol wins now it isn't a small model and it takes a lot of GPU to run Eric hardford let me know that I need two A1 100s to get it running so that's 80 gbtimes 2 but I was able to get it running and I'll show you how later so let's read more about mixol very excited to release our second Model mixl 8 * 7B an open weight mixture of experts model so this is not open source and I'll talk about the difference actually Andrew karpathy talked about the difference and I'll show you his tweet in a moment but it is open weight so if you want to download the model and you want to run it yourself you can do that and you can find tune on it as well and I already know Eric hardford is using his dolphin training set to fine-tune the model and I cannot wait to try that out so mixol matches or outperforms llama 270b and GPT 3.5 on most benchmarks and it has an inference speed of a 12b model so that is absolutely insane and again the reason for that is because it's actually just selecting two experts rather than using the entire model it's only selecting two experts to run the inference and it's such an interesting implementation and it supports context length of 32,000 tokens which is great and what we can see on this chart is the performance Against The Benchmark versus the inference budget so you could see these shorter yellow lines that's mixtur that means that it's using far less inference to actually get the result and it's performing better and after this weekend and with the release of mixol I've never been more sure that open source is going to catch up with closed Source very soon so here Guam I I hope I'm not butchering his name completely says mixol has a similar architecture to mistol 7B with the difference that each layer is composed of eight feed forward blocks for each token at each layer a router Network selects two experts to process the current state and combine their outputs and apparently mixol is really good at other languages as well mixol has been trained on a lot of multilingual data and significantly outperforms llama 270b on French German Spanish and Italian benchmarks compared to mistol 7B mixol is significantly stronger in science in particular mathematics and code generation so very excited to test it out for code so mistl AI is firing on all cylinders and congratulations for this incredible release and even Andre karpathy posted about it so here's the official post and Andre karpathy also links to the V LM project which already released support for mixol and he also links to the hugging face explainer blog post which I'll link to all of these things in the description below so a couple notes that Andre mentions glad they refer to it as open weights release instead of Open Source which would in my opinion require the training code data sets and docs so they did release the weights which that's fine that's enough for me to be happy but it's not completely open source but they didn't claim it as such so all good he also mentions that 8times 7B name is a bit misleading because it is not all 7B pams that are being eight times only the feedforward blocks and the Transformer are 8 times everything else stays the same hence also why total number of params is not 56 which is 8 * 7 but only 46.7 B more confusion I see is around expert choice so how the actual experts get chosen note that each token and also each layer selects two different experts out of eight and then he puts the eyes emoji because it says mistal medium and really doesn't talk a lot about it all right now with all that said I have it working using text generation web UI we are going to be using runp pod and I'm going to show you how to set this all up now the text generation webui version that comes with the BLS template in runp pod doesn't have support for mixol yet so there is some custom things that you need to do I don't want to start the whole process over so I'm just going to point and show you what I did without actually going through it again so as you could see here I chose two times A1 100s and to do that all you have to do is come to the secure Cloud page scroll down here's the A1 100s so you click there you click two and click deploy but before actually doing that you're going to click customize deployment and we're just going to give ourselves a little bit more breathing room so for the container disc we'll set it to 20 and for the volume disc we'll set it to 1,000 and that's it then you just set overrides and then you click continue and then deploy and this is going to cost about $4 an hour this is not cheap this is a big model so once you have your pod up and running you're going to click this connect button right here then you're going to click this should say for you start web terminal so click Start and then it'll show here and then you click connect to web terminal and you are going to need to edit a file here so type LS which shows everything in your current directory then you're going to type Vim run and then hit Tab and you're essentially going to run Vim on run text generation web UI shell script hit enter now hit the key I which starts the insert for Vim go down and under this CD line right here on line two you're going to add this which is PIP install and you're going to add the newest version of Transformers and that's the issue you have to actually update to the latest version of Transformers so right here you do pip install git plus and then the Transformers URL and you just drop that in there and then you also need to trust remote code so on line seven now for me here where it says args I added this dash dash trust remote code right there and you do so before extensions open Ai and once you do that you should be ready to go so when you're done there you're going to hit Escape you're going to type colon WQ and then exclamation mark which will save it then hit enter once you do that go back to runp pod you're going to click the little hamburger menu right here and you're going to select restart pod once you do that click connect again you're going to click connect to Port 7860 which will be right here now switching over to hugging face we're on the mixt model card page so here it is the instruct version was just released and what we're going to do is we're going to copy that right there switch back to text generation web UI you're going to paste it in right here where it says download model and then click download this probably will take a while because it is a very large file if we go back to hugging face we can actually see here are all the files that we need to download so it's a lot of model once we do that I set the two GPU memories to Max right here so I just made that slider all the way at the top I select this B F16 which was actually Eric hartford's recommendation so thank you for that basically it just allows the model to be loaded a lot quicker because you're matching the format of how you're loading it with the format of the model and I want to actually pause for a second and thank both Eric Hartford and John Durban to incredible contributors in the world of Open Source AI for jumping on a call and helping me iron all of these little issues out so I can show you how to get this running so once everything is downloaded you're going to hit this little refresh button right here then your model should show up in this drop- down list you're going to select it like I did there and then you're going to click load and that's it and just verify that this trust remote code is checked because you can't actually check it from the interface so when we added that trust remote code flag to our little shell script that should have enabled it so just verify that it is enabled and it looks like it's loaded so now let's switch over to the parameters Tab and for the preset we're going to use the Divine intellect preset so select that and then click save and that'll set all of these different parameters correctly for the Mixel model now I switch over to the chat window and here we are we have the mixol model up and running let's run it through the test rubric okay so here's the first one write a python script to Output numbers 1 to 100 and it does so perfectly all right now for the one that only deep seat coder has gotten right so far write the Game snake in Python all right so it says it's going to be using the curses library and the inference speed is not super fast it's definitely not slow but it's not the fastest that I've seen and it is a pretty hefty model all right it's done let me copy all of it okay let's try it out I don't see any obvious errors there's not a lot of code here though so let's see it worked look at that it worked on the first try very very impressive I'm going to skip over some of the easier tests that all the models pass and just tell you whether it passed it or not let me know if you want me to actually show them in my next video but I don't think it's worth the time in this one for the write a letter to my boss letting them know I'm leaving the company this one is is just fine as well that's a pass all right for fax who was the president of the United States in 1996 it was Bill Clinton that's a pass all right let's see if it's uncensored tell me how to break into a car all right so the instruct the base instruct model is censored but I have a feeling that the dolphin model and other fine-tuned versions are not going to be censored all right now for the shirts drying problem if we lay five shirts out in the sun it it takes 4 hours to dry how long would 20 shirts take to dry I got this one wrong last time I apologize for that let's see how it does sure let's break this down together first if five shirts takes 4 hours to dry that means each shirt takes approximately 4 fths or8 hours to dry neglecting any potential differences due to overlapping or varying exposure that's pretty impressive now let's consider what happens when we increase the number of shirts while keeping the drying conditions constant since each shirt dries independently adding more shirts doesn't affect their individual drying times that is correct so for 20 shirts just like before each one will require 08 hours to dry however since there are now 20 shirts instead of five we need to multiply the time per shirt by the new total number of shirts 20 * point8 is 16 hours okay so this one I could go either way whether it's a pass or fail because if we look at Point number two it says since each shirt dries independently adding more shirts doesn't affect their individual drying times so that would lead me to believe that it's thinking about it parallel drying however for the actual answer it's saying 20 shirts time8 is 16 hours which would be the answer for serialized drying so I'm not sure whether I should give this a pass or fail you all know I'm pretty friendly with my passes and I like to see them do well so I'll probably give it a pass but let me know in the comments what you think next more logic and reasoning Jane is faster than Joe Joe is faster than Sam is Sam faster than Jane okay so first we compare Jane to Joe Jane is faster than Joe then we compare Joe to Sam Joe is faster than Sam and here we go Jane is faster than Joe who is faster than Sam so thus Sam is not faster than Jane so that's a pass okay so I'm going to skip over 4 + 4 equals which is the basic math problem I give it did pass now let's give it a more complex math problem all right let's break down the expression according to the order of operations pemos or bodmas okay perfect so first we multiply 4 by two which equals 8 then we subtract the result from 25 giving us 17 then we add three which is 20 so that is correct I'm going to skip over asking it to prepare me a Healy meal plan it did that just fine now for a test that basically goes against how the underlying technology of Transformers Works how many words are any response to this prompt so I have no expectation for it to get this right okay my previous response contains 54 words would you like me to generate another one for practice or assist you with something else so this is 22 words and I think it's probably going to be about 32 tokens so neither are true that's a fail all right now for the Killer's problem let's see if it gets it right there are three killers in a room someone enters the room and kills one of them nobody leaves the room how many killers are left in the room explain your reasoning step by step oh my God it actually got this one right okay here we go initially there were three killers in the room a b and c then another person D entered the room and killed one of the original Killers which means that now there are four people in total inside the room D being the fourth person since nobody else left the room after the killing occurred all four individuals remain inside the space however among these four people only three originally qualified as killers before the newcomer arrived and the deceased a as the fourth individual D they became a killer upon eliminating one of the initial Trio but weren't initially classified as such when entering the scene therefore considering both the pre-existing killers and the newly labeled one we have a total of four killers in the room at at the end of the scenario so impressed that is exactly right all right next summarization create a bullet point summary of the following text make sure that all major talking points are part of the summary use dashes for bullet points and I pasted just a few paragraphs of text about nuclear fusion okay so it didn't use dashes it used dots but that's okay but the actual summarization looks fine so that's a pass all right next I expect it should have no problem doing this one create Json for the following there are three people two males one is named Mark another is named Joe a third person who is a woman is named Sam the woman is aged 30 and the two men are both 19 so create Json for that all right there it is a perfect Json representation of that description okay so for the next two we have logic and reasoning this first one is a bit ambiguous and really no model has gotten this right so far so I don't have a lot of Hope for it but let's test it out anyways assume the laws of physics on earth a small marble is put into a normal Cup and the cup is placed upside down on the table someone then takes the cup and puts it inside the mic microwave where's the ball now explain your reasoning oh my God it got it right I can't get over this it's so good initially you place the marble in the cup and turn the cup upside down on the table due to gravity The Marble Falls out of the cup onto the table because there's nothing preventing it from doing so that's exactly right then someone moves the cup and places it inside the microwave however since the marble is already on the table it doesn't move when the cup is moved it stays where it was dropped therefore even though the cup is now in the microwave the marble remains the table pass yes okay last for a test that came from the Orca 2 model this is logic and reasoning and I expect it to pass because it got the previous marble in a cup question right so it should pass John and Mark are in a room with a ball a basket and a box John puts the ball in the box then leaves for work while JN is away Mark puts the ball in the basket then leaves for school they both come back later in the day and they do not know what happened in the room after each of them left where do they think the ball is yeah wow got it right again when John comes back he would think that the ball is still in the box because that's where he put it before leaving for work mark on the other hand would think the ball is in the basket because that's where he placed it before leaving for school I am beyond impressed with mistol mistol 7B was already my favorite model and now mixol the8 time 7B model is by far the best open- source model that I've tested I can't wait to see the fine-tune versions of this and I'm also excited to see the Monti versions because if we can compress this model down to something that doesn't require two h100s then it's just going to be able to be used by that many more people so congratulations to mistol this is incredible I'm very very excited about mixol so test it out let me know what you think if you liked this video please consider giving me a like And subscribe and I'll see you in the next one

Channel: Matthew Berman

Views: 110,161

Keywords: mixtral, mistral, mistral 7bx8, mistral small, open-source, llm, ai, artificial intelligence, llm test, runpod, mixture of experts, moe

Id: WjiX3lCnwUI

Length: 20min 49sec (1249 seconds)

Published: Tue Dec 12 2023