Microsoft Dataverse Overview

everyone so hey everyone this is daniel christian hey everyone this is reza durrani and welcome welcome to the rnd show awesome so there's that tell me how your week was man um busy busy week i am actually conducting some power platform governance based workshops for for one of my customers so something interesting something different and of course preparing for my next video as well so i think i'm having i'm having a good time inside the power platform space what about you you know it was it was so i let's go back before last weekend was a three-day weekend but i took the friday off so i had a four day weekend and i had such a blast like the friday um my wife and i we went on a date we enjoyed our brunch uh but then after you have a four day weekend you still have to do five days worth of work in four days um so that was a little bit hectic for me but i'm glad and it's all done um and we are here and we're gonna have this great show so i'm i'm not pretty happy about this it was it was you know it is what it is well you said great show i think i and you have kind of gotten a sneak peek of it 15 minutes before the call so i think i know we are very excited i'm sure the audience is equally excited and we are up around 50 right now and and growing so so thank you everyone for for joining us today and uh so so then let's let's get started with this man i'm gonna uh what do you have uh what do you have for us right now let's see awesome all right so um today is is huge day because we've got the one and only ryan jones drum roll please he's he's gonna talk a lot he's gonna i know he's gonna teach reza and me a bunch about microsoft database we'll get to that that's you know we'll get to that in a minute what i do want to talk about is thank you so much for all the followers for the rnd show you've got 125 for two followers uh thanks so much um and on the right side over here you see our youtube channel you're already on reza's youtube if this is the first time you come to this channel please click on the subscribe and hit that bell notification it's literally helping us help you um so you know just thought i'll drop that in all right now i resent i want to give a big shout out to bjordin uh he's a big friend on the social media but this guy goes above and beyond i mean i'll i'll be honest pure and your slime presentation is like 100 times better than what we built so we want to literally give you a big shot i mean look at this first one he pushed pushed that i think about a yeah ago i mean that i mean this is like top notch man i mean yeah he grabbed my picture from one of my videos i think i i'm looking okay there nice and here's the next one too is that uh i think he pretty much nailed it on this one and then for those of you who haven't figured this out he has literally switched our names and and you know what resa and i literally are brothers and we are like twin brothers like if you take a picture of us from the exact same angle um i remember when we had that time um you know where we both and i had not cut our hair for a long time we literally looked like brothers at that time so hey burden you know uh i mean this was an epic and you're not the only one who actually thinks about that so i mean thanks thanks so much for doing this thank you and uh i think we need to we need to bring him on to to cut our promos and thumbnails for our future videos i think he's doing a better job than us i don't know full disclosure we are not professional at this so we are we might be pinging you for uh for some of your artistic talent cool all right so now let's get back to the whole you know agenda of this show is that we are going to talk about database overview stuff about you know what is in the backend architecture relational data in and out and then by the one and only which is which is ryan jones so i'm not going to steal the thunder he does a much better job introducing himself but there's a few things that i wanted to talk about ryan ryan is a man of many talents and the one thing that i want a big shout out is ryan does musical holiday lights as a hobby this is a picture of his house i mean and it's not a video otherwise if it was a video you would see all of this dance to us a musical sequence and he's got a mobile phone on the top that is epic so none you know i had to shout out because i personally do the same thing i've been doing for years this takes a lot of work it takes time to plan it and then you got to build musical sequences on that so big thing is like you know he's definitely a community person he does goes above and beyond just to bring happiness to people's face and that is a sheer sign that ryan is a big community person so let's bring ryan in man the guy's here i'm excited let's bring him in yay thank you so much for having me it's it's an honor to be able to be here today it's an honor to be able to to talk with everyone about dataverse and you know i i lead the product team for dataverse and so i hope i'll be able to share uh you know things that i know and understand about dataverse but i want to also give credit to all of you guys i mean when i see the content that you create when i see the amazing apps everyone builds it blows my mind seeing the way that customers are are using these tools to build amazing things uh it it's awe-inspiring and humbling every day i mean to maybe use a silly metaphor you know i feel like maybe i'm like manufacturing hammers or something like that i try to make the best hammers i can but the majesty of the the hammer relative to you know the sculpture or the home that is constructed using that hammer is is just uh it's amazing and humbling so thank you so much for having me it's an honor thank you thank you ryan for ryan for being on the show and i just want to say something ryan is extremely humble here for me he is mr dada verse that's that's the name i've coined for him so i actually wanted to introduce him that way but here you go mr dataverse is right here with us to talk about database itself so awesome yeah let's get started took it you're going to stamp it hashtag mr so cool all right brian so we've got a question for you and after that the show is all yours okay so the premise of today's session is what is databurst is it cosmos db is it azure sql is it azure blob storage or is it just a squirrel running in the back end charging all your stations like what what is dataverse and kind of take us through the whole process on the path so ryan the show is yours take it and help us understand what data verses awesome thank you so much daniel and i think before we start going into uh what dataverse is i think it's super important for us to start by talking about data right data is is what we know and that data comes from a variety of places right uh that data is central to how we build our applications and when i say applications i i don't necessarily mean power apps uh in that sense i mean are applications that are composed of apps of workflows dashboards and with bots and the thing is all of these different components to our applications they all orbit data okay and that data it can come from two different types of places in the power platform that data can come from connectors which allows me to connect to my data in sharepoint lists in excel in sql or 475 different other places where that data already lives but then we also allow you to build that data on top of dataverse within the power platform both of these allow you to take action against your data in power apps where you're pointing clicking touching on your phone or on your desktop to interact with that data through uh through a pixel perfect or a model driven experience it allows you to automate against that data where you're able to you know get get the cloud to go through and execute a set of predefined steps or workflows for you automatically we have power bi that allows you to analyze that data to uncover and unlock new insights and then we have virtual agent which allows you to build bots that allow folks to chat with that data i couldn't find a word that began with an a other than agent for virtual agent so i i'm sorry i couldn't keep going with the a theme there but you know when we think about these these two types of data we have external data data that lives outside of the power platform and then we have internal data data that lives in within the power platform and of course we connect to that external data using our connectors and we use dataverse for storing data within the power platform and so let's talk about why we might use these different types of data when we build our applications so why might we use external data why might we use connectors and i think there are two really good reasons here number one is it can be really really easy in that if you already have your data living in a sharepoint list you already have that data living in an excel worksheet you already have that data living in a sql server it's way way easier to get started building your apps and building your workflows just by building your apps against that data where it already lives okay you don't have to move data you don't have to understand data types it's just easier and it's cool okay to be sure the other thing that we see is sometimes this data that place where it already live is that data is in its home you know i always think of happy gilmore when he's shouting at the golf ball saying go to your home the thing is some of this data already lives in a system of record where a bunch of other apps and a whole ecosystem of business logic and business processes are built around that data in a place where it already lives and in those cases the switching cost of moving that data to somewhere else be it to dataverse or somewhere else it just doesn't make sense and so it's totally cool totally fine to leave that data right where it lives and leverage connectors to connect to that data now when we think about the capabilities that we have in connectors and gosh i apologize guys this slides out of date we actually have built-in connectivity to over 475 plus cloud services content services databases apis and more we have seamless hybrid connectivity where you can leverage our on-premises data gateway to connect to data that lives within your company's network boundary and for systems which do not already have a connector that exists you can create custom connectors to connect to those data sources once again where they exist where they reside today now well you might be saying well gosh ryan you really sold us on on connectors there like why the heck would we ever want this dataverse thing totally fair question so let's talk about why would like that internal data why why do we think that there's value in putting data into database and the first thing i would say is dataverse is and i know i'm biased here but dataverse is smart okay dataverse understands because it allows you to express how different pieces of data relate to one another if you have products and you have vendors you're able to describe to dataverse how those two different pieces of data relate to one another and the thing is it's not just creating a relationship it's also that dataverse in our data runtime based on the way you describe the shape of your data we determine the best way to index it the best way to store it and the best way to make that accessible for you in your applications and the different surface areas that you have for your applications and i'll come back to that here in a little bit the second thing that i would say is dataverse scales okay we have customers with north of 20 terabytes of data within a single dataverse environment okay when when you go to uh certain retailers i won't say which one and you say i would like a screen door installed or i would like a new window or i need someone to install a tile floor i'm not going to say which which home improvement retailer but every single one of those jobs get runs through our system some of our customers in the technology sector if the hard disk in your laptop or your service or your server fails and they need to dispatch someone to swap out that hard disk for you because it's under warranty and you have a service contract that flows through this system in fact that particular system services hundreds of millions of requests per day okay so it's really really built to support very very large mission-critical systems for for our customers and then the final thing that i would say is dataverse is built from the ground up to secure cr uh business critical data it's designed from the ground up to secure business critical data when we think about the history of dataverse uh while the dataverse name is is fairly new you know before that we called it uh well something i won't mention and then we called it common data service um but before that this came from you know a lot of the the foundation or the platform that was central to operating our dynamics 365 sales customer service field service marketing and talent businesses and so the thing is kind of the the lineage of this system is storing and protecting and securing a company's most valuable data assets and so we have a ton of capabilities we'll talk more about them here in a little bit that allow you to really make sure that your data is is secured only accessible by the people that should be able to work with it provide great auditing trail so on and so forth now i mentioned earlier that one of the ways that dataverse is smart or clever is that it allows you to use your data in all sorts of ways okay and one of the ways that i like to express this is in terms of the types of apps lowercase apps not uppercase apps that you can build using dataverse and so as you can see when you use dataverse you can build canvas you can build model driven apps you can build power portals you can build flows you can leverage ai builder you can build dashboards and power bi you can build power virtual agents you are able to maximize the types of apps the types of experiences that you are able to build over the top of your data and this allows for all sorts of very very interesting scenarios where for example as a part of covid response we had uh governments that were leveraging power platform and database for vaccine administration and distribution empower portals was being used by the public to sign up for their vaccination appointments and then canvas apps and or model driven apps were being used in the back office or in some cases in the field to track folks showing up for their appointments administering the vaccines so on and so forth and so you have these very very rich apps where a common set of data is being used across a bunch of different experiences to enable an end-to-end business process and so once again dataverse allows you to do more with your data by allowing you to build the broadest set of different types of apps or components over the top of that data and so gosh i've talked all this time but like really what is dataverse in a sentence the question you asked me at the start of this uh daniel dataverse number one easily structures data right we want it to be as easy to structure data as it is to drag and drop controls onto a powerapp canvas or as easy as it is to add steps to a workflow number two is we do this for a variety of data and business logic because people have relational data things that would typically show up in tables but they also have images they also have files they also have log history um they also have other needs that drive other types of storage behind the scenes and we want to make sure that dataverse can support all of those different pieces or types of data and this extends not only data but also business logic as well where once again through clicks and sometimes code if you want to you can describe business logic that is applied consistently and universally across all of those apps workflows dashboards and everything else that you build and we do this to support interconnected apps and processes we like to think of ourselves as silo busters we are not a fan of app over data source app over data source app over data source because what you end up with is a bajillion approximately uh different pockets of data none of which are in sync and none of which agree with one another and then finally doing so in a secure and compliant manner so that is dataverse in in one sentence and so if we look at what's in the box for dataverse and we look all the way over at the left side of our screen first of all everything we do in dataverse starts with an api this is the api that is consumed by powerapps by power automate by power bi by visual studio or xamarin or the tool of your choice if you're a professional developer and so as soon as you hit that odata standards-based api the first thing you see or you hit is our security layer where we ensure that your data is protected through rich and robust authentication and authorization powered by azure active directory with role-based access control and auditing you then get to our logic tier which is where we start being clever where we have things like calculated roll up and formula fields where we have plugins business rules duplicate detection we then have our data capabilities that allow you to describe the shape of your data allow you to discover and validate your data allow you to do advanced things such as multi-language or multi-currency applications for apps that have to be consumed by people around the globe we then have our storage tier where we get into the bits and bytes of how we store your different pieces of data into azure sql into files and blobs semi-structured data going into cosmos db uh relevant search data going into azure cognitive search and then finally we have integration which allows you to connect dataverse to all of the other systems that that you oftentimes need to do so i mean this is beyond what we do in power platform in that we have web hooks for invoking arbitrary web points we have application lifecycle management with actions or tasks for both azure devops as well as github we have data export facilities through synapse analytics and data export service and so what i'll do is i'll kind of give an overview of each of these so when we think about security and dataverse first and foremost in dataverse you're always authenticated and authorized based on what's going on in azure active directory okay uh when we look at azure active directory it's in use by the overwhelming majority of of the the enterprise or corporate world and what we're able to do is using azure active directory as a system of record for users and groups you are then able to assign roles role-based access control that grant your users and or groups access to different pieces of data within dataverse and this can be super super granular okay you can do row and field and action and table level security so that if for example you wanted one azure active directory group to be able to update one field on one record in one table i have no idea why you'd ever want to do that but you can do it the system is expressive enough to do that and this can be done in the context of complex hierarchical security models as well okay we also when we provision dataverse dataverse always lives within an environment which can act as a security and compliance boundary for you it also defines data residency so if you have certain data that lee needs to live in europe you have other data that needs to live in japan you have other data that needs to live within the united states you can define where that data lives and we take care of the rest and then we have very very rich and robust security models that in fact in some ways you can customize so that once again you are able to secure the data that is critical to your business sorry guys i'm having a little bit of trouble advancing my slides let me try once more no worries no worries i mean i just love what you've already covered so far it's just from you know the security standpoint from the uh ease and you know shared across the locations i mean so far i personally i'm just blown out of my mind so thanks man i'm learning a ton over here thanks wonderful and it looks like i managed to crash powerpoint with all my pictures so just give me one second guys no worries so while you do this uh from what i got from this it's not just a database there's a lot more to microsoft database than just being promoted or sold as a database you're exactly right raza like one of the things that we say sometimes is it's data a database plus plus or it's more than a database um maybe for me one of my favorite uh restaurants um is a place that makes a thing called the smokehouse burger and the reason i love it is because it's not just a burger yeah it has it has you know multiple hamburger patties and there's cheese and there's sauteed onions and then there's barbecued brisket and then there is uh smoked uh pulled pork and then there's bacon like that is that is where it's at as far as as far as i'm concerned um and so and the most important ingredient right it is made with love that is yes yes um yes okay the i'm almost back up guys give me just one second i'm so sorry to our audience members no worries we've got a lot of good feedback already about your slides and the most asked question right now is can we get this light so i've i've promised the audience once once ryan shares the slide deck with me i will make it available to to all of them yes yeah i'm happy to i'm happy to do that um thank you cool we're getting really really close here all right i tell you what i was going to do my demo at the end but while my slide deck reloads how about i just go ahead and do a demo now um because it actually gets into the the next part of my my presentation which was really starting to get into the logic capabilities of of dataverse okay um and one of the one of the things about logic in in dataverse is that you know we want to make sure that business logic can be applied at a data level okay because it really wouldn't be it wouldn't be good if you had business logic that was applied if you worked with the data through your powerapps experience but then if someone tried to automate that same action the logic didn't get applied right like this could be a very bad thing it can lead to inconsistent data it can lead to confusion for for your users and so we we definitely do not want to uh you know do that anymore uh any more than absolutely necessary okay cool i just wanted to make sure that i was streaming again can you guys see my screen okay now yes we can okay awesome and so the thing is we allow you to express this logic in a bunch of different ways okay these can be rules around duplicate detection they can be business rules they can be synchronous workflows or more and so you can create these using low code facilities like the synchronous workflow designer that's in the slide you can do it using power fx and this is new and this is super cool in fact we haven't blogged about it yet but i'm going to show everyone that's tuned in today what that looks like or for pro devs you can create plug-ins and by the way like this is super cool you can literally take a dll you can take a little snippet of c-sharp code and it runs in between our api that's up here and the data tier that is down here and so this allows you to do literally like in transaction custom code i'm sorry i'm geeking out a little bit here um but i think this is like the coolest thing ever um and then of course like as i mentioned you can also create synchronous workflows um that also operate kind of in that um with it within the transaction but let me demo for you since i teased it earlier when i when i crashed my presentation um let me demo what this new capability looks like okay and so i'm gonna go over to here and so imagine for a moment since i'm a holiday lighting fanatic that i wanted to start a business that um that sold holiday lighting related things and one of the important things when you are in these sort of like e-commerce uh businesses you have to ship ship stuff and so let's assume that i wanted to have a table for all of my different boxes where i could uh track the different size of those boxes and so i'm going to create this table and so what i'm now going to do is i'm going to start adding columns and so what i'm going to do is of course we need to know the height of the box we'll make that a decimal and we'll also you know i think it's kind of difficult to have negative space so we'll set the minimum value to zero and then i'm going to do the same thing again for width and then i'm going to do the same thing for depth and you know one thing i would point out is that i'm doing this like behind the scenes we're doing all sorts of stuff in sql we're scaffolding apis we're doing all sorts of stuff but here all i'm doing is pointing and clicking and adding columns and so now this is very similar to a modern sharepoint list experience where you can add columns on the fly yes meant to be super simple and straightforward now what i want to do now is add a formula column where we're able to calculate a column within dataverse using powerfx and so volume in this case would be my height times my width note the awesome intellisense as i'm right now wow by depth and then i'm going to create okay so now my volume and so here if i wanted to have my small box and this is one inch by uh three inches by four inches maybe for shipping something like a raspberry pi you'll see that it calculates the volume for me and so there in just a few seconds i went through and defined a table i defined all its columns i then defined a formula column or like a it's kind of like a calculated uh column in in sql server use power fx to define that formula and it just worked like that that quickly so i mean the idea is super super powerful but also super super easy as well good grief i'm getting goosebumps over here i'm so excited so this that feature i like is one of my my favorite features that that we've shipped recently so i am i'm super stoked about it so anyway logic and so we'll keep rolling um oh dear nothing not again okay sure we then have data right and as you saw we provide facilities for you to be able to define the shape of your data so that your data models match your needs once you've done that you can very very easily connect power bi desktop to author power bi reports and then upload those to power bi.com and as you do that we actually have very very special integration with power bi we support what's called power bi direct query which allows you to get instant real-time analytics over the top of your dataverse data in the context of the person who is running the report so all that security stuff that we talked about earlier all that gets represented in your power bi reports when you leverage direct query mode and then data versus smart and intelligent too in that many of our ai builder capabilities are actually enabled through dataverse whether we're talking about text recognition business card reader entity extraction and more these things are leveraging dataverse behind the scenes now from a storage perspective and this kind of gets to your is it this or is it that question that you asked earlier daniel right we use all sorts of goodies in azure to make this thing work we use sql we use cosmos db we use azure storage we use azure cognitive search we use azure data lake and we glue all this stuff together behind the scenes so that we can store your data in the best way and this lets us do very unique and cool things like for example smarter search that allows you to have really really simple easy fast and high quality search to quickly and easily find data within your system and we are able to abstract away all of the interesting things going on behind the scenes with all those pile of azure services that i mentioned earlier you don't have to think about it or worry about it and then of course dataverse supports all sorts of integrations we have apis that allow you to consume or communicate with dataverse from custom code if you want these are the same apis that power apps power automate and all the other tools communicate with we support invoking web hooks pushing events to azure event hub and service bus as well as data export through synapse link to azure synapse and data lake we'll come back to this in a little bit we have great data integration capabilities to move data in and into dataverse um and not just move that data but transform it along the way and then the other thing that we have is this capability called virtual tables that allows you to kind of like mapping or mounting a drive in windows or linux you can map or mount a table that lives outside of dataverse into dataverse and once that's done you can start using it through many of the applications that we talked about earlier once again when your data is in dataverse you can do more with that data and so maybe to give an example first of all everything starts with a table right and here i have a products table that shows many of the products if i were to start my holiday light uh supplier company um many of the products that i have and you can see we have things that are numbers we have things that are strings we have things that are decimals we have vendor that's actually a relationship between our product and vendor table but the thing is this would not be a compelling set of information to put on a website right we're visual people and so of course we want to support images for this right and so the thing is we do some clever things in dataverse in that when you create an image column you can enable it to store full-size images and then what we do is we create a little thumbnail we store that in sql because you're wanting to render a big list like you see on the screen all the time but then we take the full size image and we put that in azure storage behind the scenes you do nothing for this other than tell us you want a full-size image and we take care of all the heavy lifting behind the scenes okay now you also want to be able to find that data and so the the thing is we also are able to you go in and say you know what i'd like to make name searchable because that seems like a thing that you'd want to search it probably doesn't make sense to make like weight searchable or vendor vendor id here searchable maybe in the vendor table it is searchable but you can go and configure that and all of a sudden you get super super powerful search over the top of this data once you've defined that table we expose not only the apis for you to be able to interact with the product but also api that describes the shape of that data as well okay now just having data isn't enough as we talked about earlier you have to have that business logic as well and so we provide mechanisms within dataverse for you to define or describe your own business logic methods not only or this is above and beyond uh the data logic that you can describe that executes as a part of create read update and delete operations and so what this can do is as for example you create another product maybe uh we're adding a new receiver in this then what we're able to do is emit a data of a product created a bit that pushes to power automate so that we're able to run um run a workflow as as as quickly as as possible and with as high a fidelity as possible now we've also been more recently working on capabilities around business events which allows you to describe not only a data event like hey a product was created but also an event that has specific meaning to your business such as like hey this product my raspberry pi they just went out of stock okay and then we also now allow you to define custom apis as well so that whether there are business events or other things that you want to push into dataverse you can do that as well now of course this is very sensitive important holiday lighting information so we have that enterprise class security uh infrastructure that we talked about earlier um because i'm sure let my left up to my wife uh she tolerates my holiday lighting hobby but i'm sure she would want inventory of all these things to go to zero very quickly um or maybe she'd like me to sell them all fire some jewelry uh instead but uh anyway and then finally it's important that we are intelligent as well and so what we're able to do is we're able to take this data and shadow it into azure data lake what that does is that gives us a super high read read throughput replica of the data that we can then use to train ai models okay this is what lets our ai models provide better inferences okay and so once that model is trained from the data that is shadowed in the lake we are then able to perform inferences through ai builder once again plumbed together through uh through dataverse and we run on azure okay we are one of azure's largest customers okay we have a single endpoint we hide all this infrastructure from you but we have a load balancer that sits over the top of roughly uh 17 billion different uh servers that operate in vm scale sets for for my azure folks that are that are joining us today and these handle things like posting our apis running our async processing and jobs running custom potentially hostile code in our sandbox servers uh generating paginated reports in our reporting servers and then this layers over the top of all of those different storage technologies that i mentioned earlier but then you can also extend with azure as i talked about earlier web hooks can invoke your azure functions service endpoints can push events to event club service bus those same web hooks can be hosted in azure kubernetes service and then we export data left right and center to azure data lake and uh azure synapse and maybe to talk a little bit about this so we have some cool capabilities with dataverse synapse link around how we're able to create a gateway between insights and usage okay and so through azure or i'm sorry through dataverse synapse link you can configure export of your data like that product catalog that i showed you into azure data lake you can also export your product meta the metadata about that product catalog into azure synapse now the thing is a lot of times you don't do this because that data lives by itself what you want to do is you're wanting to join that data with other data that comes from other places and so let's assume that i had a legacy like support ticketing system running in sql server maybe even on prim i can use azure data factory to take that data move it into azure data lake and then what i can do is i can run sql queries and spark queries over the top of that joined data within azure data lake but using the sql language or the spark primitives that i'm most familiar with as a data scientist then finally if i wanted to make that data from that legacy support system visible through dataverse perhaps to use in a powerapps portal i can leverage virtual tables to mount that table or map that table into dataverse so that it looks and acts like other dataverse data now if you're looking to get started we have a light version of dataverse that's available in dataverse for teams it's a subset of the capabilities of what we talked about today but for going in and quickly and easily defining a table and going in and building an app or workflows over the top of it it's there it's today it's included for anyone practically anyone that has teams and it's fully governorable by central i.t it's built on the core of dataverse so it gives you a little bit of everything that we've talked about today uh supporting up to a million rows okay and of course fully upgradable to to dataverse and so guys with that i think i'm three minutes behind schedule i apologize for that but we want to do some q a wow yeah my goodness i mean i'm kind of speechless a little bit over here that was that's impressive man one of the favorite things which i liked was about the flexibility to scale um take it from you know the existing database and then scale it that that's that's huge because now you've also factored in factored in those enterprise level companies where they can consume that database space so fast it's like no you're all good you can go ahead and now scale up at a higher level and you've given them options to scale so i feel like you guys have really have put some thought into this process and that's what really impressed me a little but you know there's a what were your thoughts on this i think i'm speechless right now i'm just trying to just trying to get my head around what what's like how powerful this the service truly is backed by azure uh using all of azure services and the best part is the back end architecture is controlled so as things change if as azure improves i am very sure that those services will also automatically come into into their into database because that os is going to utilize those features scalable billions of data you're talking about security at a column level hierarchical security which which data source offers these kind of security options quick creation experiences uh relational data i think i think this is the full spectrum and the beauty is if you want to start small you have dataverse for teams so you can always begin there start exploring database learning more about it and then you have a magic button upgrade it just takes you from the small capabilities of that over to the full feature set wherein you can explore and and utilize the complete power of the service so i think that's my key takeaways from from ryan's presentation today yeah awesome so ryan you've got if some fun but yet meaningful questions for you so um reza if you want to go ahead and show my my slide so you know okay we've all now established it is definitely more above and beyond this is not the back end of database uh ryan has convinced me i'm good so ryan i got some questions for you all right so let's go let's go with this i want you to tell me what is your first impression there's going to be some pictures coming up take a look at it give me your first impression all right picture number one sharepoint is not a relational database what is your first impression go i i agree um i would also say and i hope uh you know this is consistent with what i shared earlier the thing is we want you to build apps we want you to build workflows and so if your data already lives in sharepoint and it's working just fine then go build your app or your workflow on sharepoint okay um they're going to be cases where you need more complex relational data modeling they're going to be places where you need larger numbers of rows and records they're going to be places where you need more complex um security models they're going to be places where you need more complex logic we've got dataverse for you when you need that okay but sharepoint's awesome we have lots of customers building power uh powerapps and power automate workflows over the top of sharepoint today um and so you know we talked about like when to use connectors and all that earlier i think that applies here as well but the thing is totally right sharepoint is [Laughter] awesome all right question number two or slide number two what is your first impression on this one take a look at it and it's it's kind of funny right there's a the database is the new kid on the blog coming in and say yay database you know everybody come hug me and then all these other sequel cosmetics banging it up and the key is down on the ground sore but way in the back there is excel hiding sweating bullets and right next to it is sharepoint impressions go i have i have a few thoughts like number one is sometimes people ask us all the time like sql and dataverse right and and the big thing i would say here is we love sql like we are one of the world's largest sql customers um we are rapidly uh growing to be one of the world's largest uh azure cosmos db customers um we're pretty far down the list on some of the other uh like azure um but by storage volume um we are one of the world's largest azure cognitive search customers and so the thing is we love all these technologies okay i think that's super super important like i have uh our team has weekly discussions with azure sql you know i'm on a first name basis with you know rohan our our cvp of azure azure data like we work very closely together okay and so we love all these tools and technologies and i mean i think to to our discussion earlier if you already have a system of record that's running well on top of sql we have connectors for that okay but i think that if you know you have data that needs a home it doesn't yet have a home and you're looking to maximize the number of apps that you can build against it and you want to do so in a way that's friendly for uh non-pro developers i think you should really take a a good long hard look at dataverse um and so you know i think my version of of the uh of the diagram might be that a bunch of the characters are holding hands and we're playing red rover okay uh i think i think that might be that might be my my mental image that that i have for that okay but i don't worry i over answered your question so it was it's all in good humor but there was you know some good content over there cool so we've got some time and one of the things that um reza and i already did was we actually got some questions both from our community um so we will go and ask them and i can already see some questions coming in on the chat so you know go ahead reza you've got the first one yeah so i'll i'll start with my question i just want to uh point this out that all the folks on the call this is your chance to ask your questions put the letter q colon post your question we will try and answer as many questions as we can okay so please post your questions in the chat in the meanwhile we're gonna start with some common questions that dan and i uh received from y'all plus questions that dan and i have so uh the first question in fact i think we have four questions then i've added one more in my head once ryan went through so i'll start with my question first dataverse is a part of the power platform we have data connectors that allow us to connect to other data sources correct me if i'm wrong here when power apps or power automate makes a query against my data source through the data connectors it goes through a double hop process the request goes to azure uh it goes to a service in azure that re-routes it to the underlying connector to get the data and bring it back and there are there are challenges there in terms of you know delegation and and and other sort of is not issues but challenges on the other side because database is a part of this platform when the power platform queries data was it directly queries the api so there is no double hop that is taking place there so is that is that a fair uh assumption that i'm making right here it is with one important qualifier okay when you you pick your connector when you're building your app or your workflow and nowadays we do more things to kind of steer you to pick the right one but you want to make sure that you use the dataverse connector okay we have a bunch of other connectors that's i think sometimes they're labeled classic or legacy uh those actually go through that double hop infra that you're talking about but if you use the new dataverse connector or if you go through the inline table creation and powerapps experience that does that direct connectivity it tells powerapps and it tells powerautomate to talk directly to those apis that i showed you in the slides earlier today and that makes it super super fast it makes it super super reliable and what it also means is pretty much as soon as we add new capabilities to dataverse those things light up and power automate and power apps as opposed to like us needing to uh you know tap dance to update the connector and and everything else so i hope i answered your question theresa you did thank you all right second question is from dan all right so one of the things which was um you know inquisitive to me um is this we kept hearing is there was a version one like a v1 and a v2 of data wars so what what is this because i felt like the switch got flipped overnight everything still worked but what what happened like first of all did i just hear rumors you know what is this v1 v2 yeah okay yeah so a long time ago back when dataverse was called the common data service there was a thing cds1 and cds2 um and they were different things they were completely different like platforms everything else and the thing is we kind of had to make a decision cds v2 was actually the runtime that sat beneath dynamics 365 customer engagement now sales service marketing and the thing is we had a pile of data in that system we didn't have a whole lot of data in in cds one and so what we did is we merged the two and so we completed those migrations gosh i think it's been two or three years ago now we've never had a dataverse of e1v2 dataverse has always just been that cds2 and continuous innovation so um we you know there was a thing but like i don't think there have been any cds1 environments around now for like two years or something like that so there was an earlier incarnation of this but we it's no longer around and we went through great lengths for to help folks that needed uh to move to help them move to help move all their data if there are issues working through that like we really believe strongly and like no customer left behind um because we realize how important this data is to our customers and how important the apps are that are built on top of it and ryan is that one of the reasons why uh many a times we used to see default environment upgrade is that the upgrade upgrading from yes yes that's exactly exactly it but i don't think if you're seeing that today please send me an email uh ryjones microsoft.com uh because you should not be seeing that button anymore um because we upgraded to all the environments like i said a couple years ago now i'm sorry pandemic has turned my brain to mush so i don't really remember how long ago it was all right let's move on to the last question from me and dan and then we will open the floor up to the audience so the last question is around file storage so sharepoint traditionally is a document management system file storage versioning um you can store up to millions of documents so what what would you want to add to this if i was to say all right i need to store files and i and i want to store it in sharepoint because it's a storage space it's i won't call it free but it's included with my license office license uh database for teams is limited in terms of storage it's only two gigs if i start adding files i might you know consume that capacity pretty quickly so what would you like to to to like what points would you like to add on on this specific topic and i hope i'm not putting you on the spot here no no not at all so when i think about like where sharepoint is super super strong and like amazing i mean use this feature all day every day is around collaboration on those documents right when you need to have folks that are working together on the same word document on the same powerpoint deck on the same excel sheet i mean we're going through semester planning right now guess what all our documents live in sharepoint not in in dataverse okay like that's kind of like point number one that i would make like there are a set of scenarios that where sharepoint is super super strong and i can't imagine using anything else okay now the the second like thing that i would say is there's lots of times where those files they aren't really files by themselves right like imagine in that products table that i showed earlier that i had like a schematic for each of those products maybe it was wiring diagrams or like a true electrical like component schematic um the thing is that's not really like a separate file with its own lifecycle right really that's like a file that's a part of the record itself and security and everything else should follow that of the record okay and that's where we see that dataverse is is stronger which is when like literally a file can be treated as though it is just a a column on on my data and i interact with that very broadly in fact like i mentioned some of our big customers earlier that use this like in the the home improvement services space or in the um it space like one of those customers in particular actually just moved tens of terabytes of data from sharepoint into dataverse um for this reason as well as when folks scale up to be really really large when you have like hundreds of thousands of um you know users that are going to interact with the system you know many tens of thousands of monthly active users that's really not what uh like a single you know site in sharepoint is is designed for makes sense um and so that that's kind of where we see hopefully i answered your question no you did you did and in fact now we're heading to the audience questions and i think the number one question kind of is interrelated with this so let me pop this up on screen this question is from mr omit agarwal and the question is all right we can we want to get data into dataverse what is the option how can i just bring a sharepoint list directly into dataworks do we have any options available for that so the most common way that we see people bring data into dataverse is through power platform data flows okay power platform data flows has the ability to connect to a bunch of different data sources and you're able to connect it to those data sources you're then able to either create a new table or map like the existing data source to um to you know an existing table you can perform inline transforms it's actually a really really cool technology so the most common way that we see folks do it is is through power platform data flows another common mechanism for smaller um data volumes is um and i don't know how well this would work in the sharepoint specific scenario for forgive me you guys know more about sharepoint than i do okay uh but like another common thing that we see folks do is if it's a small data set that they can open up in excel then they will uh we have an excel connector power query connector for dataverse and people will copy paste the data in excel and then save from there because you know lots of folks are familiar with and know excel so that's another common mechanism as as well um actually both using power query behind the scenes can i can i also use that table designer experience that you showed earlier and just copy and paste can i do that oh gosh for small data i haven't tried it okay so i i because i'm thinking you would want to like select a whole bunch of yellow select just put it in there yeah i apologize i haven't ever tried it before uh it makes me want to try it now all right okay let's go to the next one so the next one is from uh krishna krishna vandana he's one of our mvd mvp community folks so the question is in in data in database changing data type is not possible but there was a new article yesterday and i did read about this that that allows you to change the data type by specifying the nature of the data uh but it needed an api call could you please explain this in a bit more detail yeah and i'll give maybe kind of a two-part answer here uh part one is you know remember behind the scenes we dataverse is using all these storage subsystems and they are what we call like strongly typed systems okay in that when we go through and we're figuring out how to uh materialize that products table that i showed earlier in sql server we're needing to determine like okay are we using varchar or invar car like this so everyone knows we use inbarcar um to support unicode and customers around the globe um but so you know we're doing this you know in varchar thing in other places we're using other sql data types okay so that's the first thing to maybe keep in mind is behind the scenes we have to set the data type on the storage that we're using okay now um historically we have not allowed you to change the data types at all if you pick single line of text and you actually need a multi-line text you're kind of out of luck and the thing is the way dataverse actually stores those things in sql or in other storage systems it actually doesn't change okay um really it was just around allowing uh updates of metadata and so that's what i believe nathan published this blog uh you know yesterday um that's what we're allowing to do where if you accidentally pick multi-line text and you need single line of text or vice versa and if as long as we don't have to change how the data is physically stored we now allow you to um to change the data type of columns now the question and maybe to give an example of this imagine i have a column that is of type multi-line text and i have words typed into it and i want to convert it to a decimal the thing is that's not a conversion that can be done and so that's why we don't allow it uh in the experience because you'd end up with data loss now the second part of krishna's question is well i have to do this through api why is that um and the answer is this we had lots of customers that said hey i really really really need to be able to change this please help us out and so we built the capability kind of in the data back end but we're still in the process of updating our experiences so that people can do that through clicks rather than code and so this is a place where you can change this now uh if if you're familiar and comfortable with apis but um you know if not you've got your your experiences so i i hope i answered uh the question krishna thank you all right moving on to the next one this one is from vishal is there any limitation for querying millions of records in my database tables because database is scalable it is designed to hold large data or big data will it have an impact on performance i'm assuming he's relating this to building apps and flows that that query big data it's a great question vishal and so what i would say here is like from a proof point perspective we have customers with billions upon billions that's billions with a b records within a single table okay and so the thing is like the system can scale to be to be quite quite large okay now the thing is when we say querying millions of rows that kind of means different things to the different people and so one of the things that is cool about dataverse is we actually look at the way you define the structure of the data we look at the views that you have defined for uh the data and we actually and then we look at how your apps are interacting with the data and we use this to optimize uh how we define the the clustered and non-clustered indexes in which the data is stored okay and so the thing is like we optimize dataverse based on how you describe your data and how you're using your data to be as fast as possible now you can do things that will make the dataverse perform slowly the same way that you can do things that make really any data source perform slowly and so you know i don't want to say that dataverse is is perfect it's not um but the thing is for kind of like uh reasonable actions that we can take to ensure great performance we do all of those things for you okay um and so the thing is we have systems with billions of records in a single uh in a single table that still provide sub-second response times okay and and so we can scale quite well and we do all the optimizations we can from a data persistence and a data indexing perspective um to ensure that that the system runs well um so i hope i answered the question oh i think that that that was that was a very comprehensive response and there's just a lot of questions coming in and we'll try to get to that but i think reza we have time for one more yeah probably a couple more so i'm trying to skim through the questions i'm sorry guys if you missed some questions sorry but we are trying to get as many answers i'm trying to pick questions here so here's another one i like this one uh this one's from gaurav so as a developer who's new to this platform right it's first time i'm accessing this platform and you know i'm comfortable with maybe sharepoint and now i notice that there's this dataverse and ryan has kind of convinced me today that dataworks is is an extremely powerful data service and a lot more than that so i want to start learning and exploring about dataverse and i am new i don't have a license and i'm assuming he's relating to a premier license so is there anything for me to get started with as a developer great great question and so the answer is yes and you kind of have two paths okay number path number one is dataverse for teams and one of the reasons why we recommend this is you get you know essentially the more uh office license use you have the more uh database for teams environments you get um and so this is a way to not only kind of kick the tires and try it but even do it at scale if you want okay that's option number one option number two that we have is we do have our community plan which includes dataverse entitlements as well and so you can sign up for the community plan you can create a dataverse environment um and so you you then have an environment for kind of like personal development and those sorts of things so anyway like we have both of those options available like please come to kick the tires like we think this is a pretty cool thing uh you know if i didn't think it was cool i wouldn't work on it um so yeah we totally have these options for you and you know feel free to choose your own adventure based on what will work best for you perfect perfect and the the last question and you know we don't we do not want to make this a sale for daggers of course it's not we are trying to tell you what's available here so i i like i kind of like this question i'm taking it in good way the question is what is something that database is missing so let's try and do a comparison here is missing as against let's say a sharepoint or a sql right these are the common data sources that we select so what is something that it's missing when not to go for it when to go for sharepoint when to go for sql what would your recommendations be and trust me right that's the number one question almost everyone is asking totally and i think i might answer the question like a little bit more specifically even than how uh shriya asked it and how you asked it in that the biggest thing that i see folks like sometimes getting like when we think about sql server sql server can actually support a variety of workloads right it can support um transactional workloads and that's really what's in data versus wheel house um and it can support analytical workloads with its columnar data stores and so the thing is dataverse is not a data warehouse right it's it's not an ascube it is not uh it doesn't use columnar storage behind the scenes like it is designed for more transactional style workloads okay um because like think about it in power bi when you load data you load like a bajillion rows in like five seconds and then you do read only operations get through that in your dashboard the reason why it does those super high volume imports is because it's optimized to ingest a bunch of data and then do read-only stuff against it okay when we think about dataverse dataverse is designed to support interactive transactional data that's used throughout the power platform okay so that's scenario one that i'd say and don't do that go use you know an analytical data store of some sort okay scenario number two is dataverse is very very focused today on apps that get used within your corporation um or in some cases with like your partners with some what we call b2c connectivity to like your company's customers through power portals or through power virtual agents and so if someone was saying hey you know what i'm wanting to build you know uh christmas lights for uber uh or you know uber for christmas lights sorry i gotta reverse there um the thing is the people that are going to be working with that app primarily are not people that reside within my organization and think about how we talked about how much of our security is built and designed and optimized around azure active directory it doesn't make sense right because the general public has no private presence in your company's directory and and so i think that those are kind of two kind of main litmus tests uh is it for a line of business app if no then it's probably not right for dataverse and then question number two is is it an analytical app and if the answer is yes then it's probably probably go somewhere else so hopefully those are like two great examples of kind of disqualifying use cases for for dataverse brilliant i think i i really like that that analogy all right so uh i think that's that's it for live q a and uh once again uh thank you to mr ryan for taking the time out and coming and joining our show thank you so much and sharing honestly i think you've gone above and beyond what what we knew it's gonna be awesome but i think we at least me my mind is blown so really appreciate this thank you so much and uh i i hope you you come join us again i would love to thank you so much daniel and reza for uh inviting me and allowing me to join you today and thank you for everyone that tuned in for allowing me to to share a little bit with you about dataverse hopefully we can help you be successful and do more with your data i see a lot of love and interest we i think we broke almost the number 200 so so i think we have an awesome audience and uh i see a lot of love coming in for for that of us so thank you ryan once again awesome thanks really thank you cool all right so just some items to talk about really fast um what we have uh for the next session it's show number four and it is all going to be about power virtual agents uh things we're going to cover about power virtual is both the angles building standalone chat bots directly in power virtual agents will switch gears and now build it in microsoft teams and then we'll also talk about how we can leverage power automate and build flows to go and pull information from databurst and you know really get the power out of the pva chatbot a little bit more responsive over there and we will also show some hidden gems things which are not very obvious but they are existing in pva so you definitely want to keep track of that one so it's on september 24th same time over there and then finally i just want to quickly show our status right now as far as all the um you know the events that is happening specifically our hashtag so then we can't see it right now yeah it's coming up and so right now as we stand these are our top three winners um first of all just big thank you to everyone who tweets about all of this with the hashtag um in our rnd underscore show thank you so much for doing that you're picking all of them up and we're going to go ahead and keep track of it and as we get close to the end of the season we will be making that announcement of who the winner is and then you know that person will get a chance to actually have a lot of one-on-one time with us so but all said and done thank you for helping us promote um you know our channel over here and thank you for helping us just you know help share the knowledge you're helping us help you in that way so um that's basically all i had uh thanks once again for stopping by reza any last words well i think once again i'd like to thank uh ryan for joining us and thank you everyone for for coming live to our show amazing audience i think uh uh you guys keep loving us and and we will keep working hard to to bring quality content and bring amazing people like like ryan so thank you thank you bye everyone see you see you in two weeks bye

Channel: Reza Dorrani

Views: 36,454

Keywords: microsoft dataverse, what is dataverse, microsoft dataverse tutorial, what is microsoft dataverse, common data service, microsoft, power platform, power apps, how to, dynamics 365, dataverse power platform, dataverse, relational database, database relationships, how to use dataverse, dataverse security, intro to dataverse, dataverse power apps, dataverse tutorial, dataverse vs sharepoint, reza dorrani, dataverse for beginners, dataverse overview, microsoft dataverse overview

Id: VgX2BIdEdqs

Length: 72min 11sec (4331 seconds)

Published: Fri Sep 10 2021