Cerebral Valley: Adam D'Angelo (Quora) + Harrison Chase (LangChain) with Miles Grimshaw (Benchmark)

[Music] foreign [Music] excited uh thank you Adam and Harrison for joining us today um uh Adam is the founder of Cora and also on the board of openai and Harris and the founder of Lang chain maybe we'll start just by introducing um yourselves and maybe Adam start with you uh what how did you end up started working on Poe which is certainly one of the more interesting kind of consumer experiences for interacting with a bot bot experience sure so first of all thanks for thanks for hosting and thanks for having me here so I've been interested in AI for a long time and and I've just always been and very optimistic about the the potential for a long time the AI was useful for things like recommender systems which were very important for for quora but we knew that someday we'd get to the point where AI was able to generate text we just didn't really know when it was going to be and so at Cora we've been watching this very very carefully and early last year we came to a realization that hey this is basically this is the time when we need to to start to move and and there's a real opportunity here if we can take advantage of it and so we started off and we did some experiments using AI to generate answers on quora and the initial conclusion we had from that was that AI is it's got it had gotten pretty good but it was still not usually as good as the best human that our machine Learning System could find to write an answer and but the cases where it really was an advantage was if someone needed an immediate answer to a new question or if someone wanted to go back and forth and get follow-up questions answered and so after experimenting with that for a while we realized that actually a chat UI Paradigm is the the right thing for this new technology rather than trying to kind of shoehorn it into the the existing product that we had and we also wanted to just be able to iterate at a faster pace so this this technology wave is moving so fast that we really need to just be able to adapt and move very very quickly and so one other advantage of building a separate product aside from the UI considers considerations is that we'd be able to move faster and so so around August last year we ended up starting a new project which ultimately was called Poe and and launched in December and so it's a it's a fast chat interface to a variety of AI models currently we have access to Chachi BT the anthropic models a few different versions of Claude and we have recently added gpt4 so that's uh yeah that that's what we're doing now it's a big Focus for the company although core is also continuing to go well and and adapting to some of these new technologies at the same time awesome and I think around Harrison I think around August of last year is when you started dabbling with the idea of Lang chain maybe you share some background on on what instigated working on it and deciding to make it a company yeah absolutely and again thank you thank you guys for having us here um yeah I started uh around August I started going to a bunch of uh I think that's when I mean gendered fairies kind of just gone like woo over the past like a year even but I started paying attention in August went to a bunch of meetups in SF and chatted with a bunch of folks um and and I noticed that they were doing a lot of the same things over and over again and they were kind of like patterns in terms of the applications that they were building and I think at that point you know Jasper had been out copy AI had been out these kind of like text Generation Um Applications had been out but the new the new wave of applications let's say which were a bit more complicated and involved more kind of like orchestration of the language model with other sources of data and computation were starting to be built and there were some common kind of like abstractions and patterns that I was noticing so you know as a side project I I and there's also some really cool research papers that were coming out and so I saw some common patterns put them in a python package threw it out there um and then yeah I've kind of like iterated on it since um and yeah we've been we've been a company for about uh I think two months at this point um two and a half months um with the idea of again continuing to just you know abstracting out all these things was done with the goal of making it easier for everyone to develop these types of applications and so that's the focus of the company as well awesome um Adam one thing that you do in Poe is expose all the different under different language models that people could use as opposed to sort of pick the best one maybe for the user and say hey for your experience like here use this this is the best one right now we're going to constantly switch out state of the Opera you actually expose that to the user with the idea of several different Bots what's the what's the thinking there behind that yeah so I remember hearing an anecdote a long time ago that in the early days of the web Yahoo was the the dominant dominant website really and and there was a pretty big population of Internet users who thought of Yahoo as the same thing as the web and they would open their web browser and the default homepage was Yahoo and and they really didn't understand a distinction between Yahoo and and other sites and but you know eventually you fast forward the web is much more developed today and people have more of an understanding of the different sites they want to go to they might use Google to to navigate that but it's useful for people to have an understanding of where they're going what they can trust uh and and have these different brands for different services that they access and so with Poe we want to be similar to a web browser and that we want to enable people to access many different forms of AI and we think that some understanding on the part of the user about where their request is going and and who they're talking to is is going to be a useful thing we do want to get into recommending people the best bot to talk to for their use case but right now we're focused on just enabling access to as many different AI models as possible including these foundational models like chat gbt or Claude but also we want to provide access to any kind of application that someone might build on top of these through through something like Lane chain we expect there to just be this this very wide diversity of AI chat experiences that developers are able to build so we're in the process of building an API and soon we'll we'll access to to all developers to to create these these Bots and we're really looking forward to what everyone can create so you imagine in the not too distant future developers individuals will be able to create bot experiences alongside some of the the initial ones you've you've offered up with the large language models yeah that's right what might be some of the examples that you'd hope happen and emerge in the next few months or quarters depending on launch I mean I I think there's going to be this massive ecosystem that's you know similar to what the web is is like today but some of the early examples that we've played with that I think are promising uh what one is language tutors so if you're if you're trying to learn a new language there's if you prompt GP chat gbt the right way you can you can get some help but you if you imagine a a good language tutor it might have a memory of what you're weak at and what you're strong at and might try to intentionally prompt you to to answer questions about areas that you're weaker in it might it might have some kind of progress tracking it might it might remind you how much you know whether you didn't show up the day before to do your practice there's a whole kind of experience you could build around that and this is just language learning I think learning any kind of topic you could build a similar kind of product I think there's going to be a lot of bots that are essentially games or hosts of games that there's a ton of really exciting stuff that can be built there I think a lot of companies have unique data sets that they wanna move into a a chat experience and and so you know I could imagine a world where most companies have their own bot that they provide to the public hopefully through Poe but also through through other interfaces I think this is going to become a a common pattern that that basically every product every company out there wants to wants to provide Harrison you were I think one of the first places many who've tried to build these bot experiences and connect them up to their own information that's not in the model obviously or other tools to take action to retrieve additional information and have made that easier for folks where are you investing time now in that where are you seeing developer interest in that what's sort of the focus at the moment of connecting these models to to more power yeah I'd say I'd say there's like two main areas and I think both tie pretty nicely and with what Adam was talking about I think the two are basically uh personalization and then kind of like agentic action taking stuff and so personalization can mean a variety of things it can mean uh you know exposing all your company's data in a personalized chat bot that that can talk about your company um it can also mean remembering things and remembering things about the user um and kind of like adjusting that way and then the agendic part comes into taking actions and so and so being able to interact with other resources and and sometimes you know sometimes you have data that's in that's in text form and you can interact with directly other times you have data behind apis and you want a natural language layer on top of that API and so adding the ability for the language model to First generate a query then taking action to to query that get it back and then respond in a synthesized human way I think those are kind of like the two main areas that we've seen a lot of developer interests in and and from the beginning those have been kind of like you know Lang chains not focused on uh you know we have stuff around prompts obviously those are kind of like the new uh uh Paradigm for for coding in these applications but a lot of the the the tools added have been aimed at like personalization and then agentic things obviously tools you call them tools chat GPT just came out open AI just came out and called them plugins but the idea of connecting up the llms to other services um how do you think about that that announcement how do you think about Tools in Lang chain and and more broadly this sort of idea of llms connected to external resources and where you think about that that heading yeah I think it's uh I think it's great I think there's been a huge swell of kind of like excitement around this and I think it's been uh really uh interesting it it's connected well to a lot of the concepts that have been out in academic Academia from from the react paper in in November or something like that the self-ask paper also out in October and so I think it's really I think it's great to draw attention to this new or to this idea of connecting it to external things I think it's also great to see how open AI is approaching it they're one of the best like research organizations in the world and they're dedicating a lot of resources to trying to get this to work well and I think you know they they launched something really impressive and I think there's still a lot of stuff to be done in that space and I think it's also really interesting to see how people want this not just with uh chat GPT or gpt4 or whatever fine-tuned model they're using but with other models as well so to hook it up to like anthropic and have that model do it people already asking about like llama in alpaca and can you use those to control your own and have have more uh you know yeah you have your own models run and do these types of things so I think it's been really interesting and uh to see the the increase definitely like an increase in excitement and awareness around these really powerful ideas and abstractions so you both sit over in a kind of large language model agnostic in in many ways as you're saying connecting up tools to other other Foundation models people might want to use maybe it's in a private use case and so they don't want to you know private information they don't want to give it as much to open AI um Adam you're exposing it to users to be able to pick and be aware of the different Foundation models have you seen that what's your emerging views on sort of the foundation model fight in some way and and are you sort of surprised and impressed with the quality that's coming out of um anthropic and and some challenges as well as open source like alpaca yeah I I think I mean the field is moving incredibly quickly um you know I I think it's pretty clear right now gpt4 is in the lead um but the entire field is moving so fast that it's very hard to know where things will be in in the future um I think I also I think we've gotten to the point where these Foundation models are so powerful that it's it's sort of good enough for many applications and it's at the point of you'll have diminishing returns to to continued progress for some applications for other applications you're going to want just to always be on the frontier and have the most powerful thing um and and so to me a lot of the most interesting progress to be made from here is is more around things like plugins or just better ways of fine-tuning the models as opposed to just continuing to to scale the the models and you know also with scale comes greater expense and and so things get not economical for for many users um but it's it's an incredible field um it's it's moving so fast I I think I think we're all going to be shocked at where things are a year from now I say that just extrapolating from from past Trends um and I think there's there's going to be a diversity of different providers of these models exactly how it'll all shake out as an industry I think it's very hard for anyone to say you know there's a camp that's saying it's going to be commoditized there's another Camp that's saying you know it should be like a winner take all Market I think it's it's very hard to say at this point how it's going to to play out yeah I agree with that and and adding on I think you know gpd4 is clearly the best one out there at the moment and there's some Delta between them and anthropic and then you know anthropic and the next like open short small or something like that um but I think it it kind of comes down to like what's in that Delta and there's a lot of things that you can do that are that you can do with open source and there's even more there's a really like the anthropic model is really good and so there's a lot that you can do with the anthropic model and so for the plugins for example like you know can you use plugins as easily with the anthropic model as you can gpt4 probably not but I think there's some like uh with a little bit of like clever prompt engineering you can get it to work and I think that's also another like interesting thing that I've seen is that the prompts don't really transfer between models and as I said there's like this really fast pace of development and so prompts that worked really well like for example the react prompts that came out in the paper in November or October or something like that um you know those aren't necessary or they don't work as well for the current state of the art models and so so it's tough to predict like what you know in you know in a year will all the models be roughly converged and prompts will work the same maybe will they have different styles that work for different malls I think that's another like interesting thing that developers are having to Grapple with so you know it's it's not it's kind of simple and that you can just like change the model name but it's not really that when you're doing these more complex things so obviously it's a very different programming Paradigm in that it's not deterministic it's sort of Fairly stochastic and and as you're saying as you switch out the model it changes um but even in general as you uh you know you you allow people to use tools is the agent going to use the right one at the right time and in the perfect way um it's sort of goal oriented and I sometimes will joke that um it's almost like a college freshman that occasionally gets drunk on Franzia you know so like I was a little awry at times um like how do you think about helping developers in that Paradigm um where it where it's somewhat random like how are you thinking about supporting developers building application experiences that they want to have a high amount of reliability like meet user expectations uh but but could uh there's a hallucination side but also could just like not perform the the the the desired goals yeah I think that's the main thing that we're thinking about I think right now Lane chain in its current form is really good for prototyping applications it makes it really easy to get started build them I think the next thing is bring it into production and production not only comes with kind of like system challenges but also with like reliability issues and I think you know it's easy to get something that you can cherry pick an example for and tweet out on Twitter and get lots of likes but getting that to work well in production is a whole other thing and so one of the main things that we're thinking about is evaluation of these things which you know you can't maybe for some applications you can use like accuracy and other things like that but for a lot of these you know it's it's longer textual answers there's maybe there's a few different axes that you want to evaluate them on is it like polite is it is it is it factually correct and all of those are you know not not nearly as simple as traditional machine learning methods or metrics so where uh yeah we're I'm very bullish on the idea of actually using language models to evaluate themselves I think anthropic's done a lot of great work in that area and so that's something we're looking into a lot but yeah this is I think evaluation has emerged as a big pain point and I'm actually curious to ask Adam how you guys think about evaluation because you know we yeah you guys have an actual product and so it's much more top of mind for you probably yeah so I think as much as possible we want to let developers build the the AI experiences and then we want to provide developers with support that'll help them do their own evaluation as opposed to us getting into the that game ourselves we want to get into a position where we're good at evaluating the differences between the different Bots after they're created so we can recommend the right one to a user for for any given need they have but for now you know we we've found you know we we do we do these systematic evals um we we do we do it by hand we found it's we had to we have to do a very different process for this than what we had been used to for quora and evaluating um the quality of of human written answers because the the answers are often they seem right to someone who's evaluating them but the answer could actually be be wrong um and so we've had to set up some different processes this is not something we've done a great job of yet but we want to be in a position where we can really help developers evaluate their own models so we're going to be working with them to to make sure they have the data and the support they need to like run an A B test for example when when that's appropriate I think I think that's a really interesting point as well the idea of like running a b testing and stuff like that's obviously pretty common in kind of like software and product development but in machine learning like you generally rely more on like you know MSE or accuracy or something like that but now with these end-to-end applications that are really difficult to evaluate I think those types of a b testing more traditional uh you know testing of things is is really interesting so expect more in Lang chain in this area yes um I'll ask one more and then maybe open up to the audience if there are any questions um an area that's obviously getting a lot of discussion as as these scale up as the compute scales up but folks try to scale up the data um uh into them and find additional sources I'm curious Adam how you think it's about two twofold maybe one is um uh uh sort of you know unique data sets like the core data set as as as it applies to building some of these models and then two maybe there's a you know not a not a today problem but a next few years problem maybe where these ugc sites have been enormously impactful for training data but maybe as users move away from sort of going and seeing who wrote the answer and moving to a world where model outputs and answer the sort of social status that would have come from being a thought leader on stack Overflow or in core it might diminish and so there might be less incremental content um generated that won't sort of keep us up and I'm curious if you have a lens on that as it pertains to to training these models yeah so so first of all we haven't gotten into training models on the quora data set yet that's something we want to try at some point but but it's not something we've we've done yet and we actually committed to our users that if and when we do that we will allow any core user to opt out of having their content uh used for for training of large language models um and I I think that's that's an important commitment because people have a choice about where they're gonna share their knowledge and and whether to share their knowledge and if if they don't have the the certainty that it might not effectively be um be taken and and reused by AI in ways that they didn't want then they might be less likely to to share on quora and then we think that that commitment is also going to give us the right motivation to make sure that any kind of training we do respects authors in in the right way so it might be that we have a bot in the future that that can like quote from core answers or or cite sources and I think we should be able to find a good relationship where most authors on quora are are happy with the credit that they're getting similarly to how they're they're generally happy with the distribution that they get on quora itself today um you know in our case we haven't really seen any reduction in the amount of people coming to Cora as a result of all these you know this advancement of of chat UI and AI um it it may happen in the future but we'll we'll see I think there's always going to be a pretty big segment of people who want to go to the source and they want to they want to hear a story from a person who actually experienced that thing and so we're gonna just kind of see how things play out and and react but so far the the motivation for people to write answers on quora has been less about the fact that someone in the future might look up their answer and read it and more about short-term uh critique and and comments and and reception from the community Encore itself which is something that I don't see um getting disrupted in the same way that people looking up answers a year later might might get disrupted hey it's Eric I'm going to jump in I mean you guys touched on this a little bit but I feel you know we're talking about like chat gb10 I feel gpt10 I think there's this fear that like open AI could just be dominant you know they're ahead they keep learning more like Poe is predicated on people shopping between different ones and you know somebody winning in a different use case than chat gbt and similarly like the plug-in case I think there are people who say oh you know Lang Chang's in trouble now like can you address sort of the fear of like if the world is dominated by open AI are your offerings still relevant and like how do you think about that so you know as a board member when I asked a hard question for me to answer um but I guess I will say you know I'm I've I've been on the board and so I've I've I have a perspective on opening eyes strategy and and where things are going and if if it was just that they were gonna totally dominate everything then I wouldn't be building po so so that that's the first thing I'll say I think the other thing I think you can see in in opening eyes actions is that they especially over the last month there's been a big focus on developers so there's these all these different developer apis that are available and and a lot of progress on on improving things like the terms of service for developers and that's hope that's happening because open AI I think recognizes that there is going to be a wide diversity of use cases that people want to do with this technology um and it may be that we end up in a world where openai is providing this very powerful model and then you have hundreds of companies building different things on top of it and then all of those products are available through Poe or many of them the ones that were the where the UI is appropriate and so A diversity of AI applications is not necessarily even in conflict with a world where openai has the best model and and effectively everyone is using that and I think like similarly yeah like you know A diversity of applications does mean that there will probably need to be tooling and other things built to help enable those I think you know it largely depends on what open AI decides to do right like if they decide to to go really hard into this area yeah that's you know that might not be great if they decide to go really hard into consumer products right I think like I'm not on the board of open AI can I be on the board of open a um so so I don't know what they're gonna do I I I strongly believe that one people will always want some optionality I think there's definitely like uh especially yeah I think there's a big value in kind of like being open and having interoperable systems and then two I'm also a belief that that there's going to be a really really long tale of applications um and uh yeah I think openai will probably remain a bit more focused on trying to provide the best possible model and and you know they have done a lot to make the developer experience better and I'm very grateful for that but I do think there'll be tooling needed around that maybe time for one last one I'm the guy behind Eric um there's this open letter out there from Milan and was at all and I you know because you're on the board It's probably hard for you to talk about that too but the uh and I actually like to hear what miles thinks but I'd just like to hear a reaction about why you think they had to publish that open letter and say hey let's stop everything for six months and take a big breath and think about this great question yeah yeah I I obviously I'm aware of the the letter and um I don't think I can say too much more I I think there's a I think there's a variety of different motivations that that people have for um for for suggesting that um I I think that I guess I will say I think AI safety and and some of the long-term issues that are going to emerge are incredibly important and you know in my role as a board member it's it's very important to me that that is something that I'm looking out for and that the the whole board is is looking out for um I think that whether a six-month pause is the right answer and and whether it's practical you know that that's a different question but yeah I don't really want to get into it any more than that here I haven't spent too much time on on uh on the fear-mongering of like AGI taking over the world and like what will all do and that that sort of future I think that's spend a lot more time um with with developers who are still trying to figure out what what can be possible here there's fissile material and we're still finding uh amazing new applications and capabilities um you know it was only six months ago or so that the react paper came out with sort of Chain of Thought prompting the ability to use tools like we barely started working on that you know probably only six months old and so I think with the with the clay that's already on already there to be used um uh there's a lot for uh amazing experiences to get creative that seep into all pockets of of the economy and that's where I spend my time um focused on so no thoughts there great thank you guys so much really appreciate it thank you thank you thank you [Music]

Channel: Newcomer

Views: 6,830

Keywords:

Id: CUCBXyJym3M

Length: 29min 21sec (1761 seconds)

Published: Fri Mar 31 2023