>> ALL RIGHT, LET'S GET STARTED. SO THIS IS THE LAST SESSION OF BUILD. MY NAME IS ANKIT AS THAN GNAW. I YOU GUYS ARE HAPPY THAT IN 60 MINUTES HOPE YOU GUYS AND FOLKS ARE ENJOYING IT WILL BE OVER. ALL RIGHT YOU MADE BUILD THIS YEAR. THIS IS THE LAST IT. I THINK WE SHOULD CLAP FOR EACH TALK AT BUILD. WE SAVED THE BEST ONE OF YOU. THANK YOU. [APPLAUSE]. FOR THE LAST. THAT'S WHAT WE ARE SO WE HAVE 60 MINUTES AND YOU PAID HOPING FOR IN THIS SESSION. LET'S FULL PRICE FOR THE BUILD. SO LET'S GET START THE. BEFORE WE GO FURTHER GET EVERYTHING THAT WE CAN GET OUT I HAVE A QUESTION FOR THE AUDIENCE OF THE SESSION IN THE NEXT 60 MINUTES HERE IN TERMS OF HOW MANY FOLKS AND WE'LL GO THROUGH IT QUICKLY. IN THE AUDIENCE ARE JUST GETTING I'M A PROGRAM MANAGER ON -- IT'S STARTED WITH MACHINE LEARNING? COULD ONE OF THE LARGEST BIG DATA. AND WE HAVE A RAISE OF HANDS? AWESOME. WE HAVE CUSTOMERS USING TECHNOLOGY THANK YOU. IN THIS SESSION WE ARE LIKE KAFKA HIVE AND HBASE. SO I GOING TO TALK ABOUT HOW YOU CAN THOUGHT IT WOULD BE A GOOD IDEA ACTUALLY ENTER THE WORLD OF MACHINE FOR ME TO SHARE WHAT THESE CUSTOMERS LEARNING. I AM JOINED BY CESAR AND GO THROUGH WHEN THEY ARCHITECT THE CHRIS LAUREN. WE ARE EXCITED TO BIG APPLICATIONS AND WHAT ARE THE BE HERE TODAY. AS YOU PROBABLY ALREADY TRADE OFFS THEY MAKE. AND AGAIN, SEEN AT BUILD . NET IS A GREAT WAY NOTHING RIGHT AND WRONG. SOME THINGS TO BUILD A WIDE VARIETY OF APPLICATIONS. MIGHT WORK IFFER YOUR SCENARIO AND YOU CAN BUILD WEP APPS IN AS VDOT SOME MAY NOT. BUT I WANTED TO SHARE NET AND MOBILE APPS IN SGLAM RON WHAT WE GATHERED. SO IT'S ALL ABOUT AND -- ZAMRON WE ARE TALKING ABOUT CUSTOMER JOURNEY AND THE PAIN POINTS MAKING MACHINE GREAT FOR LEARNING AND HOW THEY MADE THE DECISIONS IN ML. NET. LET'S GO OVER QUICKLY THAT THEY MADE. TYPICALLY IF YOU WHAT WE ARE GOING TO COVER IN THIS THINK ABOUT BIG DATA LANDSCAPE. SESSION. LET'S START WITH GIVING I THINK YOU GUYS ARE FAIRLY -- YOU YOU AN OVER VIEW OF WHAT IS NEW UNDERSTAND THE PATTERN. AND IF YOU WITH ML. NET 1. 0. WE WILL SHOW THINK ABOUT THE CUSTOMER BASE THEY YOU HOW TO GET STARTED FICLY WITH WOULD BE BUILDING A LAKE WITH A ML. NET AND THE AUTO EXPERIENCE L FRAMEWORK. BUT I THINK FOR THIS THE COOLING EXPERIENCE WHICH INCLUDE TALK IT WOULD BE A GOOD REFERENCE MODEL BUILDER AND ML. NET CLI AND BECAUSE WE SEE CUSTOMERS USING THAT WE ARE GOING TO GET INTO HOW YOU ARCHITECTURE. SO THERE IS SOMETHING CAN DEPLOY ML. NET AT SCALE AND CALLED HOT PATH WHERE YOU INGEST RICH OPEN SOURCE POPULAR LIBRARIES DATA FROM DEVICES AND SENSORS AND LIKE TENSEOR FLOW AND ONYX. FOR TYPICALLY IN INSIGHT PEOPLE USE THOSE OF YOU IN THE ROOM PERHAPS KAFKA TO INGEST THAT DATA. AND THEN NEW TO MACHINE LEARNING ONE WAY ONCE THEY HAVE THAT DATA THEY NEED YOU CAN THINK ABOUT MACHINE LEARNING TO BE ABLE TO DO STREAM PROCESSING. IT IS ABOUT PROGRAMMING THE UNPROGRAMABLE. SOMETIMES IF YOU THINK ABOUT DEVICE FOR EXAMPLE IF I ASKED YOU FOLKS DATA IT SO COULD BE JSON. AND AT IN THE CROWD CAN YOU GO AHEAD AND THE MINIMUM YOU ONLY NEED ABOUT BUILD A FUNCTION WHICH TAKES AN 4 OF THEM. SO MAYBE THEY ARE STRIPPING IMAGE AND RETURNS WHETHER THIS IMAGE DOWN THE DATA THEY NEED. AND OTHER HAS A FACE IN IT OR NOT, YOU MIGHT INSTANCES THEY MIGHT BE RUNNING NOT KNOW WHERE TO START. IF I TAKE IT ON THE FLY AND MACHINE LEARNING. ANOTHER EXAMPLE WHERE I GIVE YOU AND SO THAT'S THE STREAM PROCESSING A DESCRIPTION OF A SHIRT LIKE THE AND THEY TEND TO STORE IN A SQL ONE I AM WEARING RIGHT NOW AND ASK OR IF IT'S DEVICE DATA IT'S NO SQL YOU AGAIN CAN YOU WRITE ME A FUNCTION STORE. AND A LOT OF PEOPLE USE HBASE THAT RETURNS WHAT IS THE PRICE OF OR.. . DEPENDING ON THE CHOICE TO THE SHIRT, IN THIS CASE YOU MIGHT STORE THAT. AND THAT INFORMS THEIR BE ABLE TO LOOK AT SOME KEY WORDS LAYER. AND SO FOR THE CORE PATH LIKE LONG SLEEVES OR BUSINESS SETTING ANALYTICS WHERE YOU ARE STORING AND TRY TO GUESS THE PRICE OF THE ALL THIS DATA AND NOT ONLY NEED CHURCH. B -- SHIRT. THEN IF I SAY TO RUN ANALYTICS ON A REALTIME BASISES SCALE THIS UP TO 1, 000 PRODUCTS AND YOU WANT TO RUN ANALYTICS ON IN TERMS OF ONLINE STORE SALE IT A HISTORIC BASE ALSO. SO YOU CAN MIGHT BE A BIG CHALLENGE THERE. STORE IT IN A DATA LAKE STORAGE. EVEN THOUGH YOU DON'T PERHAPS KNOW SO ON THE BATCH PROCESSING SIDE, HOW TO GET STARTED WITH WRITING TYPICALLY YOU WOULD GET DATA FROM THESE FUNCTIONS, WHAT YOU DO HAVE DIFFERENT APPLICATIONS ON PREMISES IN FRONT OF YOU IS EXAMPLES. YOU OR IN THE CLOUD. AND THEN THEY WOULD HAVE EXAMPLES OF IMAGES WITH FACES BE INGESTING DATA EVERY MINUTE, IN THEM AND YOU HAVE EXAMPLES OF HOUR, WEEK, DEPENDING ON THE USE IMAGES THAT DO NOT HAVE A FACE IN CASE AND THEY WOULD BE GIVING THAT THEM. WHAT MACHINE LEARNING IS ALL DATA. AND THEN NEED TO THINK ABOUT ABOUT IS LEARNING FROM THESE EXAMPLES THE GOVERNANCE. HOW TO THINK ABOUT AND THEN BUILDING A FUNCTION OR THE METADATA AND MANAGEMENT, SECURITY, WHAT WE CALL THE MACHINE LEARNING CONTROL. AND ADMINISTRATION. AND MODEL THAT YOU CAN THEN USE. ANOTHER THE SERVING LAYER. I THINK MANY WAY OF THINKING ABOUT THE MACHINE OF OUR CUSTOMERS BASICALLY RUN THE LEARNING MODELS YOU CAN THINK OF ANALYTICS AND PUT THE DATA IN A IT AS AN INTELLIGENT FUNCTION WHICH DATA WAREHOUSE WAREHOUSE. BUT WE'RE TAKES AN INPUT OF AN IMAGE IN THIS SEEING MORE AND MORE USE CASE WHERE CASE AND RETURNS WHETHER THIS IMAGE DATA LAKE ITSELF BEING USED IN A HAS A FACE IN IT OR NOT. SO WITH PLACE YOU WANT TO OPEN IT UP FOR ML. NET, WITH MECHLT YOU CAN ML. ENTIRE ORGANIZATION AND THEY ARE NET YOU CAN HAVE THE IMAGES LIKE ABLE TO THEN DRIVE ANALYTICS FROM THE ONE I SHOWED YOU. YOU CAN BUILD THERE. SO WE'LL TALK ABOUT THAT AZURE MODELS FWITH REGRESSION WHICH AS WELL. SO WHEN CUSTOMERS LOOK ANSWER MODELS LIKE HOW MUCH, HOW AT THIS KIND OF PROBLEM SOLVING MANY. WHILE YOU ARE USING ML. NET OR ARCHITECT, THERE ARE A TON OF WHAT IS GREAT IS YOU CAN USE C# TECHNOLOGIES IN AZURE. THERE IS AND F# SKILLS TO BUILD THESE MODELS. DATA FACTORY. AND DATA BRICKS FOR EVEN THOUGH WE ANNOUNCED ML. NET SPARK WORKLOADS. AND SO FORTH. BUT AT THE BEGINNING OF THIS YEAR IF
YOU TODAY'S FOCUS IS ON INSIGHT. THIS LOOK AT THE HISTORY OR THE ROOTS IS WHERE WE'LL SPEND OUR TIME. SO OF ML. NET THEY COME FROM MICROSOFT WHY WOULD SOMEBODY USE HDINSIGHT. RESEARCH PROJECT CALLED TLC WHICH WHY DO WE SEE CUSTOMERS USING HDINSIGHT. HAS BEEN IN THE COMPANY FOR A NUMBER I THINK NUMBER ONE THE REASON THEY OF YEARS. AND IS CUSED EXTENSIVELY DO IT IS THEY WANT TO PRESERVE EXISTING BY PRODUCTS LIKE AZURE MACHINE LEARNING, INVESTMENT. THEY MAY BE RUNNING EXPERIENCES LIKE WINDOWS HELLO OFFICE ON PREMISES AND A ONE TO ONE TRANSLATION. POWERPOINT AZURE TEAM ANALYTICS IF THEY WERE DOING THIS IN A FACTOR AND A WHOLE LOT MORE TO NAME A FEW. THE RESPONSIBILITY OF SUPPORT LIES SINCE WE ARE TARGETING DEVELOPERS ON THEM. IT'S MANAGE OFFERING SO WITH ML. NET WE HAVE DESIGNED THE YOU GET A LOT OF PIECES AND HDINSIGHT FRAMEWORK IN THAT MATTER. WITH 1. IS ABLE TO SUPPORT MULTIPLE WORKLOADS. 0 WE ARE ADDING NEW FEATURES LIKE THERE IS NO PARTICIPANT OF THE ARCHITECTURE. AUTO ML AND TOOLS LIKE AUTO BUILDER THERE IS NO -- THEY FIND ALL THE AND COMMAND LINE INTERFACE WHICH COMPONENTS AND ALL THOSE COMPONENTS WILL MAKE REALLY EASY FOR YOU TO WORK TOGETHER WITH EACH OTHER AS BUILD CUSTOM MACHINE LEARNING MODELS. THEY ARE PART OF THE SAME PLATFORM. WE HAVE ALSO FROM THE GET GO THOUGHT THEY WANT TO KEEP THE SAME CODE OF EXTENSIBILITY AS A KEY POINT BASE. SOMETIMES OUR CUSTOMERS TEND OR KEY VALUE PROP OR KEY DESIGN TO RUN IN GOOGLE AMAZON . AND BECAUSE PRINCIPLE IN ML. NET. NOT ONLY CAN WE'RE USING 100 APACHE THEY DON'T YOU LEVERAGE AND USE THE ML. NET HAVE TO CHANGE THE CODE BASE. AND TRAINERS AND ML. NET TRANSFORM THAT LEVEL SECURITY WE'LL TALK ABOUT COMES OUT OF THE BOX BUT YOU CAN THAT. AND LOW COST. AND YOU SEE ALSO BENEFIT FROM USING OPEN SOURCE HOW IT'S PRICED. IT'S VERY LOW COST. POPULAR FRAME WORKS LIKE TENSEOR BIG DATA PROVIDER IN AZURE. BUT FLOW OR ONYX USING THE SAME SET ANYWAYS, SO SOME OF YOU ARE WONDERING THAT ML. NET OFFERS. JUST TO -- "WHAT IS IT BY THE WAY". SO IS INSIGHT, JUST ONE MORE POINT ON THIS SLIDE TO TALK QUICKLY. IF YOU LOOK AT THAT I WANT TO MAKE. LIKE EVERYTHING HIVE, KAFKA, STORM AND OTHERS. BUT ELSE IN. . NET IN THESE DAYS ML. EVEN THOUGH THEY HAVE DECIDED THAT NET IS FREE IT IS CROSS PLATFORM THEY WANT TO BUILD SOMETHING ON AND IT IS OPEN SOURCE. THAT SOUNDS HDINSIGHT. THERE ARE MANY QUESTIONS GREAT. LET'S TALK ABOUT PERHAPS THEY NEED TO ANSWER. FOR EXAMPLE, WHAT ARE SOME OF THE THINGS YOU WE TALKED ABOUT THE STREAM PROCESSING CAN DO WITH ML. NET 1. 0? THERE IN THAT ARCHITECTURE. SHOULD WE ARE A NUMBER OF THINGS THAT YOU GO TO SPARK STREAMING OR THING ABOUT SEE ON THE SLIDE THAT YOU CAN ALREADY STORM. OR IF WE ARE THINKING ABOUT DO. YOU CAN BUILD SENTIMENT ANALYSIS OUR SERVING LAYER. A LAYER THAT MODELS, YOU CAN BUILD A PRODUCT WILL GIVE TO THE DATA ANALYST, SHOULD RECOMMENDER. YOU CAN BUILD PRICE WE USE PRESO. SPARK SQL. STORAGE. PREDICTION AND CLASSIFY IMAGES AND IN AZURE THERE ARE MULTIPLE TYPES BUILD A SALES FORECAST AND A WHOLE OF STORAGE SYSTEMS. PREMIUM. SO LOT MORE. ALL OF THESE SCENARIOS HOW DO WE MAKE THAT DECISION IN. YOU ARE LOOKING AT HERE ARE AVAILABLE. SHOULD WE THINK OF AZURE -- OR AIR LET ME WALK YOU THERE. WHAT YOU FLOW. OR WE HAVE BEEN USING -- ON ARE GOING TO SEE HERE IN A MINUTE PREMISE IS THAT GOOD ENOUGH. OR OR IN A SECOND IS THE . NET WEBSITE. SHOULD WE USE ETL. SO THOSE ARE IF YOU ARE LOOKING FOR ML. NET YOU THE QUESTIONS THEY NEED TO FIGURE CAN CLICK MACHINE LEARNING ON OUR OUT. SO LET'S START GOING THROUGH WEBSITE. IF YOU GO FURTHER YOU ARE THE QUESTIONS AND TALK ABOUT SOME GOING TO SEE THE VARIOUS SAMPLE OF THE PRO'S AND CON'S. FIRST LET'S WE HAVE AND CLICKING ANY ONE OF GET THE EASY ONE OUT OF THE WAY. THEM IS GOING TO TAKE YOU TO OUR SO SPARK, HIVE. THOSE ARE WHAT WE SAMPLES REPO SO YOU CAN HELP BUILD SEE IN THE SERVICE. SPARK IS GETTING THEM FROM SCRATCH. LET ME SHOW YOU QUITE A BIT OF ADOPTION. WHERE IF THIS QUICKLY. SAMPLE REPO. YOU CAN YOU THINK ABOUT P, WE HAVE A CUSTOMERS SEE WE HAVE SAMPLES FOR FRAUD DETECTION. WHO MAYBE ARE RUNNING IT ON PREMISES, YOU CAN CLASSIFY ISSUES. YOU CAN BUT NOW THEY WANT TO RUN THE SAME RECOMMEND PRODUCTS. YOU CAN DO A SYSTEM AS IS IN THE CLOUD. BUT NOBODY SALES SPIKE ANOMALY DETECTION. YOU IS BUILDING NET WORK APPLICATIONS CAN USE ONYX TO DO IMAGE CLASSIFICATION USING P. BUT IT'S SLOWER THAN SPARK. AND OBJECT DETECTION. FOR EACH OF SO IF YOU ARE RUNNING SOMETHING THESE SAMPLES FOR DETECTION HERE IN P YOU ARE PAYING MORE MONEY BECAUSE WE HAVE BOTH THE TRAINING CODE FOR THE PERFORMANCE IS SLOWER AND YOU THESE MODELS AND CONSUMPTION CODE. ARE SPENDING QUITE A BIT OF RESOURCES YOU CAN SEE IN EACH OF THE EXAMPLES THERE. SO I THINK THAT PIECE, A YOU HAVE WHERE THE DATA SITE COMES LOT OF PEOPLE WHO ARE WRITING AND FROM THE TASK IT REPRESENTS AND MAKING APPLICATIONS DECIDE TO GO THE CODE FOR BUILDING THE MODEL WITH SPARK. HIVE, PEOPLE DO SAY AND CONSUMING THE MODEL. IF YOU THEY DO USE IT FOR ETL. BUT IN REALITY ARE NEW TO MACHINE LEARNING AND IT'S MORE AROUND DATA WAREHOUSING YOU WANT TO GET STARTED CHANCES SIDE OF IT. AND STREAMING ENGINE. ARE IF YOU VISIT SAMPLES REPO THE AGAIN, IF YOU USE STORM OR SOME SCENARIO YOU WANT TO ENABLE IS PERHAPS CUSTOMERS THAT USE STORM SWEAR BY ALREADY THERE. THIS WILL BE A GREAT STORM. THEY LOVE IT BECAUSE IT'S START FOR YOU TO LOOK AT THE SAMPLES LOW LATENCY NATURE. YOU CAN PROCESS AND LEARN. NEXT PERHAPS LET'S TAKE A SINGLE EVENT AS IT COMES. YOU A SCENARIO HERE OR LIKE SENTIMENT DON'T NEED TO MICROBATCH. AND BATCH. ANALYSIS AND LET ME SHOW YOU A LIVE AND THERE ARE SYSTEMS LIKE THAT APP USES THIS. SO WHAT YOU HAVE WHERE IT'S VERY IMPORTANT FOR THOSE IN FRONT OF YOU HERE IS A BLAZER SYSTEMS TO BE ABLE TO PROCESS EVERYTHING ML. NET APP. THIS IS USING BLAZER SINGLE EVENT AS IT COMES. SO STORM AND THE MODEL THAT IS RUNNING BEHIND HAS BEEN POPULAR IN THAT SENSE. IT IS MEL. NET IT SAYS MACHINE LEARN BUT SPARK HAS A LOT OF MOMENTUM SOMETHING FUN. IT IS A POSITIVE BECAUSE SPARK IS A GENERAL PURPOSE SENTIMENT. IT IS WORKING WELL. IF FRAMEWORK AND DOING ETL WITH SPARK, I SAY MACHINE LEARNING IS NOT FUN IT MAKES NATURAL SENSE TO USE THE YOU WILL SEE THE SENTIMENT DROPS. SAME PLATFORM THAT FITS YOUR NEED. THAT IS AN EXAMPLE OF ML. NET PLAY. BUT IT'S A MICROBATCH. YOU GET THROUGHPUT I WANT TOP BUILD FROM SCRATCH. LET BECAUSE YOU ARE BATCHING MANY EVENTS ME SHOW YOU HOW YOU CAN DO THAT. TOGETHER. BUT UNDERSTAND THE TRADEOFF I AM GOING TO BRING OUT VISUAL STUDIO AROUND VERY LOW LATENCY STREAM PROCESSING HERE. YOU USE VISUAL STUD IO FOR WITH SIMILAR EVENTS. AND THE THIRD ML. NET YOU WANT THE KNEW GET PACKAGE. ONE IS A BIT MORE COMTEMPLATED IS ONCE YOU ACQUIRE THAT PACKAGE ALL INTERACTIVE QUERY TECHNOLOGY CHOICE. OF THE SOURCE CODE IS CURRENTLY AT THIS POINT IN TIME YOU HAVE ESTABLISHED UNDER THE MICROSOFT DOT ML NAME A LAKE AND IN THE LAKE NOW WE HAVE SPACE. IF YOU WANT THE CODE YOU DATA IN IT'S RAWEST FORM. IT COULD CAN DO THAT. ONCE I SET UP THE NUGGET BE TEXT, JSON. OR PARQUET OR SO PACKAGE AND NAME SPACES I CAN GO FORTH. AND YOU WANT TO EXPOSE TO AHEAD AND START CREATING MY MACHINE YOUR ORGANIZATION. SO IN SOME CASES LEARNING CONTEXT. FOR THOSE OF YOU YOU WOULD BE EXPOSING TO A SMALL IN THE ROOM FAMILIAR WITH ENTITY TEAM. SO SPARK ESQL WOULD BE A GOOD FRAMEWORK THIS IS SIMILAR TO CREATING FIT. BUT AS YOU TEND TO SORT OF A DB CONTEXT. THE NEXT THING I HAVE EXPOSE IT TO MORE AND MORE USERS TO DO IS READ IN MY DATA SET FOR QUESTIONS AROUND RESOURCE GOVERNANCE TRAINING. FOR TRAINING AND TESTING. BECOME VERY, VERY IMPORTANT. SO IF YOU NOTICE ON THE RIGHT SIDE IF YOU LOOK AT THE -- INTERACTIVE YOU WILL SEE I HAVE TWO DATA SETS CLUSTER. YOU CAN DO VERY SOPHISTICATED HERE. I HAVE THE LABELED TEST DATA THINGS. -- TALKS ABOUT QUERIES. SET. IF I EXPLORE ONE OF THESE DATA THAT BECOMES VERY IMPORTANT. RESULT SETS HERE YOU ARE GOING TO SEE THAT CACHING IS A VERY IMPORTANT AREA. THIS HAS TWO COLUMNS. THE FIRST IMAGINE YOU HAVE HUNDREDS OF USERS COLUMN IS COLUMN TEXT. IT IS THE SUBMITTING QUERIES TO THE CLUSTER ACTUAL TEXT. THE SENTIMENT COLUMN AND THEY ARE ISSUING THE SAME QUERY. IS THE SECOND COLUMN WHICH IS VALUE NOW SOME OF THEM ARE INTERESTED 1 OR 0 WHICH REPRESENTS IF THE SENTIMENT IN THE SAME RESULTS AND IF THE RESULT IS GOOD OR NOT. FOR EXAMPLE IF I IS ALREADY AVAILABLE. WHY COMPUTE SAY OVERALL I LIKE THIS PLACE A ONCE MORE. SO RESULT CACHING WHERE LOT, THE SENTIMENT ON THAT ONE IS RESULTS IS CACHED WITH UNDERLYING POSITIVE THAT'S WHAT THE VALUE ONE DATA HASN'T CHANGED. OTHER ASPECT IS SUGGESTING. SO THE WAY I READ IS MATERIALIZED VIEWS. SO WHAT SLOWS IN THIS DATA SET INTO MY ML. NET YOU DOWN IN BIG DATA. THEY JOIN. ENVIRONMENT IS BY USING AN [INDISCERNIBLE] BECAUSE YOU ARE DOING COMPUTATION. CLASS. IT HAS TWO FIELDS. THE FIRST AND THOSE TAKE RESOURCES AND TIME. FIELD IS OF TYPE TEXT IT IS CALLED AND CUSTOMERS THAT WANT RESULTS TEXT AND TYPE STRING. WHAT THIS GET FRUSTRATED. SO THERE IS TECHNIQUE IS REPRESENTING IN MAPPING IS THE WHERE YOU CAN PRECOMPUTE YOUR RESULTS FIRST COLUMN HERE IN THE DATA SET. AHEAD OF TIME BASE ON SOME TIME THE SECOND FIELD IS CALLED LABEL. SCHEDULE OR WHETHER THE SOURCE DATA THAT IS A COLUMN INDEX ONE. THAT CHANGED OR NOT. AND THE OTHER ASPECT IS REPRESENTING WHETHER MY SENTIMENT IS THE TRANSACTIONS. WE TALK ABOUT IS POSITIVE OR NEGATIVE WHICH IS BIG DATA SYSTEMS. YOU WRITE ONCE, NEITHER ONE IN THIS CASE. I AM GOING AND READ MANY TIMES, AND UPDATE TO USE THIS SENTIMENT CLASS TO READ AND DELETE NEVER. THAT IS NOT TRUE IN MY DATA. I AM GOING TO SAY DATA IN A PRACTICAL LIFE. THERE IS BADDATE VIEW HERE. I AM GOING TO PASS IN THAT. AND REGULATION LIKE GDPR WHERE MY CLASS HERE WHICH TELLS US HOW YOU NEED TO DELETE THAT CUSTOMERS TO READ THE DATA SET. AS PARAMETERS DATA YOU NEED TO UPDATE AND DELETE. I AM GOING TO PASS IT TO MY TRAINING NOW IF YOU LOOK AT PRESTO. IT'S DATA SET. THEN A PARAMETER HERE NOT HAS STRONG ON THE THE SCALE IT TELLS IT HEY I ALREADY HAD A SIDE OF IT. WHERE IT REALLY SHINES, HEADER. I AM ALSO GOING TO GO AHEAD IMAGINE A SCENARIO YOU HAVE A DATA AND READ IT QUICKLY. I AM GOING IN ORACLE. AND HIVE AND SOME PLACE TO USE MY TRAINING DATA MODEL I ELSE. AND YOU WANT TO WRITE A SINGLE AM GOING TO USE MY TEST DATA TO SELECT QUERY THAT GOES AGAINST ALL DETERMINE HOW WELL MY MODEL IS ACTUALLY THE SOURCES. PRESTO HAS CONNECTERS PERFORMING. AT THIS POINT I HAVE WITH ALL THESE DATA SOURCES AND RUN INTO MY DATA SETS. THE WAY MACHINE PEOPLE CAN DO SOPHISTICATED QUERIES LEARNING WORKS IS WHAT I HAVE READ THERE. HOW MANY OF YOU HAVE TO DEAL IN THIS DATA RIGHT NOW THE TEST WITH METASTORES? OKAY, SO WHAT IS COLUMN HERE THIS IS A PRETTY CLEAR METASTORES. IN BIG DATA THERE IS COLUMN OR TEXT IN GENERAL NEEDS DATA IN DATA LAKE. BUT THEN METADATA TO BE CONVERTED TO A PARTICULAR ABOUT THAT DATA IS STORED IN METASTORE. FORMAT THAT CERTAIN MACHINE LEARNING SO THAT'S THE METASTORE. NOW SO ALGORITHMS CAN UNDERSTAND. WHAT HIGH METASTORE IS THIS GLUE THAT I NEED TO DO HERE NEXT IS TAKE THE BINDS ALL OF THESE BIG DATA ENGINES STRIPPING COLUMN HERE AND CREATE TOGETHER. IF YOU LOOK AT SPARK IT A VECTOR WHICH IS BASICALLY TYPE UNDERSTAND -- IF YOU THINK ABOUT ESSENTIALLY. THE WAY I DO THAT WITH PRESTO IT UNDERSTAND HIVE CATALOG. ML. NET IS CREATE A PIPELINE. CREATE IF YOU LOOK AT P AND SOME OF THESE A PIPELINE WHICH TRANSFORMS MY DATA. TECHNOLOGIES. SO IT'S A VERY CRITICAL ONCE MY DATA IS IN THE RIGHT FORMAT COMPONENT. SO IF YOU LOOK AT -- I CAN ADD ML TRAINER TO IT. I AM YOU CAN BASICALLY HAVE A SINGLE GOING TO SAY I ESTIMATOR I TRANSFORMER. METASTORE AND A DATA LAKE AND ALL I AM GOING TO BUILD THIS ESTIMATOR THE ENGINES CAN COME AND UNDERSTAND IN A STEP WISE MANNER. IN THE FIRST THIS DATA. WHICH IS GREAT FROM CONCEPT STEP I AM GOING TO GO AHEAD AND PERSPECTIVE. BUT IF YOU DOUBLE CLICK SAY I AM GOING TO TRANSFORM MY TEXT ON IT AND YOU THINK THEN, EVERY FOR THIS TRANSFORMATION I AM GOING ENGINE IS TRYING TO DO THE COMMON TO USE THE TRANSFORM CALLED SYNCHRONIZE MINIMUM THING. RIGHT? AND IT MAY TEXT SOMETHING CALLED IN GRAMS. BE GOOD FOR SHARING IN SOME CASES. THIS TAKES IN THE INPUT COLUMN. BUT THE ENGINES CAN NOT OPTIMIZE THIS IS A RESULT OF THE FEATURIZEZATION FROM A PERFORMANCE PERSPECTIVE. WILL BE. I AM GOING TO PASS AS AN FROM FROM A TRANSACTION PERSPECTIVE. INPUT THE FIRST COLUMN WHICH IS ALL OF THEM NEED TO ABIDE BY A COMMON THE SENTIMENT DOT TEXT. I HAVE TAKEN MINIMUM THING THAT A COMMON METASTORE MY INPUT TEXT COLUMN AND CONVERTED CAN DO. SO WHAT IS HAPPENING WITH THAT INTO FUTUREIZED TEXT COLUMN THE METASTORE? NOW SPARK AND HIVE USING THIS. ONCE I HAVE DONE THIS ARE SORT OF A PRIMARY PARTICIPANT I CAN GO AHEAD AND ADD A TRAINER IN THE METASTORE. SO THE COMMUNITY TO THIS. I AM GOING TO SAY DOT APPEND. IS, YOU KNOW, SPARK IS GOING TO SINCE WE ARE TRYING TO ESSENTIALLY HAVE IT'S OWN METASTORE. AND HIVE PERFORM SENTIMENT ANALYSIS AND TRYING WILL HAVE IT'S OWN METASTORE. SO TO PREDICT WHETHER THE SENTIMENT NOW 2 METASTORES AND THE WAY THEY IS 0 OR 1 THAT'S A BINARY CLASSIFICATION TALK TO EACH OTHER IS INNER HOUSE. PROBLEM. I AM GOING TO SET IT AS IT'S QUITE FAST. AND NOW THE IMPLICATION A PART OF MY CON TECH. I AM GOING OF THAT. IF YOU USE TRANSACTION TO SAY TRAINERS AND NOW WHAT I CAN DATA IN HIVE. SPARK COULDN'T READ DO NOW IS I CAN CHOOSE AMONG ONE THAT DATA. IT COULD ONLY GRAB THAT OF THE MANY TRAINERS THAT ML. NET DATA. BUT NOW THAT IT'S GOING TO PROVIDE. THIS IS WHAT INTELLIGENCE SUPPORTIVE AND COMPATIBLE. ALL OF IS SHOWING ME. I AM GOING TO USE SUDDEN SPARK CAN DEAL WITH HIVE'S LOGISTIC TRANSGRESSION. YOU LABEL TRANSACTIONAL DATA FOR EXAMPLE. AND THE FEATURES COLUMN THAT I LABELED SO THESE ARE SOME OF THE BENEFITS OR TRANSFORMED. NOW I HAVE BUILT THAT COME WITH THE ARCHITECTURE: A PIPELINE THAT CAN TAKE THE DATA BUT IF YOU ARE BUILDING A DATA LAKE AND TRANSFORM IT INTO THE RIGHT SYSTEM WITH THE LATEST OPEN SOURCE FORMAT AND MACHINE LEARNING ALGORITHM. TECHNOLOGIES YOU NEED TO UNDERSTAND THE NEXT STEP IS TO TRAIN MY MODEL, THIS DIFFERENCE. AND THEN THE DATA THE WAY I DO THAT IS BY CALLING PIPELINE ORGANIZATIONAL CHOICES. THE FIT API ON THE ESTIMATOR. PASS IF YOU ARE IN AZURE YOU ARE PROBABLY IN THE TRAINING DATA THAT I JUST FAMILIAR WITH AZURE DATA FACTORY. LOADED. THIS ARE WILL RUN THE TRAIN A LOT OF OUR USES USE AZURE DATA THE DATA AND PROVIDE ME A TRAIN FACTORY. AND AIR FLOW. AND OPEN MODEL. ONCE IT IS READY I CAN PERFORM SOURCE PROJECT. IT HAS A LOT OF SOME PREDICTIONS SO THAT'S MY NEXT MOMENTUM WITH IT. IF YOU LOOK AT STEP. OUR PREDICTIONS WILL MODEL THE STARS ON IT 12, 000 STARS. SO THE TRANSFORM I TRAINED MY MODEL THERE IS A HUGE MOMENTUM WITH RESPECT IN THE TRAINING DATA BUT I WANTED TO THAT. IF YOU ARE BUILDING A MULTICLOUD TO TRANSFORM THE PREDICKS IN THE SOLUTIONS YOU NEED TO DEPLOY ON DATA. I CAN USE MY TEST DATA WHICH PREM. OOZY IT'S POPULARITY IS GOING I MADE EARLIER. ONCE I PERFORMED DOWN. BUT YOU ALREADY INVESTED IN THE PREDICKS I NEED I AM GOING TO IT YOU CAN BRING IT ALONG. BUT I VALUE THE MODEL. I AM GOING TO STORE DON'T THINK THIS IS TIME TO BRING IT IN THIS WIRE CALLED MET TRICKS. A NET NEW APPLICATION FOR A BRAND AS I AM GOING THROUGH THIS PROCESS NEW APPLICATION. RIGHT? ALL RIGHT, I AM GOING TO USE THE BINARY CLASSIFICATION SO I'LL PROBABLY TAKE A COUPLE OF BECAUSE THAT'S MY MACHINE LEARNING QUESTIONS. THERE. ALL RIGHT. NO TASK. I AM GOING TO PASS IN MY PREDICTIONS QUESTIONS. WE SAVE TIME. LET'S MOVE I HAVE GOT AND LABEL IN THE SCORE ON. OKAY, SO ONCE YOU, YOU KNOW, FIELD. SO THAT'S PRETTY MUCH ALL KIND OF DECIDE THIS HIGH LEVEL, I NEED TO DO TO BE ABLE TO BUILD WHAT TECHNOLOGIES ARE YOU GOING A SENTIMENT ANALYSIS MODEL HERE. TO PICK, THE NEXT BIG QUESTION IN I START UNLOADING MY DATA. I CONVERT FRONT OF YOU WOULD BE STORAGE. RIGHT? THAT INTO THE RIGHT FORMAT ADD A IF YOU ARE BUILDING NET NEW APPLICATION MACHINE LEARNING ALGORITHM, NEW THAT'S GREAT. BUT IF YOU ALREADY TRANSFORMATIONS AND NEW TEST DATA HAVE HUNDREDS OF TB'S SITTING IN IN THIS CASE. I WILL DECIDE A BREAK A LOOP IMPLEMENTATION ON PREMISES. POINT HERE RUN THIS IN A DEBUGGER HOW DO I MOVE THAT DATA TO AZURE. SO I CAN SEE WHAT KIND OF ACCURACY AND THEN IN AZURE YOU HAVE SO MANY WE ARE GETTING WITH THIS MODEL. OPTIONS. WHAT SHOULD I PICK AND I AM GOING TO STEP OVER HERE. YOU WHAT ARE THE TRADEOFFS. ANY ONE ARE GOING TO SEE UNDER METRICS, OF THE STORAGE OPTIONS THAT YOU YOU ARE GOING TO SEE AN ACCURACY PICK, THESE ARE GOING TO BE REMOVED. FIELD. THIS ACCURACY FIELD IS TELLING THE STORAGE IS ALWAYS REMOTE. IT'S YOU HOW WELL THIS MODEL IS PERFORMING. A FEW MILLE SECONDS. THE CACHING . 80 IS SUGGESTING THIS IS ABOUT ON CLOUD BECOMES IMPORTANT. AND 80 PERCENT ACCURATE RIGHT NOW ON WE'LL TALK ABOUT THAT AS WELL. IMAGINE THE TEST DATA WHICH IS NOT BAD GIVEN YOU HAVE 2TB'S OF DATA ON PREM AND WE JUST GOT STARTED. A CONCEPT WE YOU HAVE 45MB PER SECOND LINK AND CAME UP WITH A CONVENIENCE API CALLED YOU NEED TO TRANSFER THAT. IT WILL A PREDICTION ENGINE. WHAT IT ALLOWS TAKE 12 YEARS. SO THAT'S THE REALITY YOU TO DO IS IT ALLOWS YOU TO PREDICT OF IT. IN THE BIG DATA TRANSFERS ON A SINGLE INSTANCE OF DATA. I ARE TRICKY. YOU KNOW, ON WIRED. CREATE THE PREDICTION ENGINE. SAY AND SO AGAIN, IF YOU HAVE A SMALLER PREDICT ON SINGLE INSTANCE OF DATA. DATA SET YOU CAN USE EXPRESS ROUT I WILL SAY PREDICTION ENGINE HERE. AND THE TOOLS AVAILABLE THERE. BUT THE PREDICTION ENGINE TAKES IN AGAIN IF YOU HAVE A LARGE DATA SET TO THE INPUT CLASS SENTIMENT AND OUTPUT AZURE. THERE IS A DATA BOX. I DON'T CLASS AND TAKES IN THE INPUT AS KNOW HOW MANY OF YOU ARE FAMILIAR THE MODEL. ONCE I CREATED THE PREDICTION WITH A DATA BOX. GREAT. GOOD DEAL. ENGINE I USED IT HAD TO MAKE A SINGLE AND SO THE DATA BOX, AND WE HAVE PREDICTION ENGINE. NOW I CAN PASS BEEN WORKING WITH THE DATA BOX FOR IN A NEW SENTIMENT TO IT. I AM GOING QUITE SOME TIME. THEY WROTE AT THE TO CALL THIS NEW SENTIMENT THE SAME BOTTOM. THERE IS A LINK THAT TAKES ONE ON THE APP THAT MACHINE LEARNING YOU TO A STEP BY STEP GUIDE IN TERMS IS FUN. I WILL RESET MY BREAK POINT OF HOW THIS APPLIANCE WOULD WORK. HERE. I WILL SHOW YOU HOW THIS PREDICTION SO WE BASICALLY SHIP THESE DEVICES ACTUALLY WORKS OR HOW THE PREDICTION TO YOU, AND THEN THERE ARE A SET IS DOING. I WILL TAP OVER THIS YOU OF INSTRUCTIONS YOU NEED TO FOLLOW WILL SEE IN THE DEBUGGER HERE IS AND OFF LOAD THE DATA INTO DATA THE PREDICTION FOR THIS SENTIMENT BOX AND THEN SHIPPED BACK TO AZURE IS POSITIVE AND PROBABILITY AND AND GETS PLUGGED IN. SO IF YOU HAVE SCORE IS TELLING YOU HOW WELL THE A LOT OF DATA YOU CAN USE THAT OPTION. PREDICTION IS. THAT'S A QUICK EXAMPLE NOW ONCE YOU MOVE THE DATA, THEN, ON HOW YOU CAN GET STARTED WITH TO STANDARD BLOCKS STORAGE. PREMIUM ML. NET TO BUILD A SENTIMENT ANALYSIS MANAGE. HOW SHOULD WE THINK ABOUT MODEL. IF SOME OF THESE CONCEPTS AND WHAT SHOULD WE PICK? THE BLOB SEEM STRANGE TO YOU DON'T WORRY HAS BEEN THERE FOREVER. IT LAUNCHED WE WILL COVER THIS IN OUR DECK HERE. WITH AZURE. IT'S HAVE MATURE TECHNOLOGY. SO THE NEXT THING I WANT TO COVER IT'S BATTLED TESTED WITH GOOD AND IS -- I ALREADY COVERED THAT ONE. BAD DAYS. IT LAX FROM A FEATURE THE NEXT THING I WANT TO COVER QUICKLY PERSPECTIVE. FOR EXAMPLE. ONE SIMPLE IS WHAT ARE THE FEATURES WE ARE OPERATION, YOU KNOW, THAT HAPPENS ADDING TO ML. NET 1. 0. THE FIRST IN A BIG DATA WORLD. ATOMIC RENAME. THING I WANT TO TALK ABOUT IS AUTO HAVE YOU HEARD OF THAT TERM. SO ML. NET OR AUTO MAXIMIZING THE AI. WHEN THE SPARK EXECUTERS ARE DIFFERENT THIS IS A NEW FEATURE WE HAVE ADDED. NODES AND TRYING TO CRUNCH THE LOCAL IF YOU REMEMBER THE CODING EXAMPLE DATA, AND AS THE INTERMEDIATE RESULTS THAT I WENT OVER WHEN I WENT AHEAD COMES BACK THEY GET PUT INTO SOMETHING AND CHOOSE THE BINARY TRAINER WHAT CALLED TEMPORARY FOLDER, RIGHT? YOU MIGHT HAVE OBSERVED IS IN ML. WHEN ALL THE RESULTS ARE COMPUTED NET WE HAVE A NUMBER OF TRAINERS THE TEMPORARY FOLDER AND RENAMED THERE. WE HAVE AVERAGE PRECEPT TRON, OUT PUT FOLDER. THE RENAMING OF WE HAVE [INDISCERNIBLE] AND SO ON. THE FOLDER IS A METADATA OPERATION. SO NOW AS A PERSON WHO IS STARTING NOTHING MOVES, YOU CHANGE THE NAME NEW WITH ML THIS MIGHT BE TRICKY OF THE FOLDER. -- SO WHETHER THIS TO SEE WHICH ONE PERFORMS BEST FOR HAPPENS YOU ARE ACTUALLY A PHYSICAL THIS SCENARIO. LIKEWISE EVEN IF MOVE OF FILES. SO IF BASICALLY SLOWS YOU FIGURED OUT THE RIGHT TRAINER DOWN THE ETL PIECES. ALL OF YOU FOR YOUR EXAMPLE OR FOR YOUR SCENARIO, HAVE DONE QUITE A BIT OF WORK. -- YOU CAN FINE TUNE THESE TRAINERS TO MOVE THAT DATA. BUT IT'S NOT WITH THESE SETTINGS CALLED HYPER AS GOOD AS METADATA OPERATION. SO PARAMETERS. AGAIN IF YOU ARE STARTING THE PATTERN FOR ANALYTICS BIG DATA MACHINE LEARNING AND YOU ARE NEW WORKLOAD WE SHOULD USE -- IT WORKS THIS MAY BE CHALLENGING FOR YOU. WELL. WE'RE TRYING TO TRAN. THESE WITH AUTO ML YOU KIND OF LIKE SOLVED PREMIUM ARE IMPORTANT. SO I'LL TALK THIS PROBLEM. AUTO ML AUTOMATICALLY ABOUT THAT NOW. I TALKED ABOUT THIS BUILDS THESE MODELS WITH THE BEST IDEA OF -- YOU HAVE QUITE A BIT PERFORMING TRAINER AND SETTINGS OF BAND WITH. YOU CAN GET 10GB'S FOR YOU. IT WILL USE THE LOCAL COMPUTE PER SECOND. SO IF YOU HAVE A WORKLOAD TO FIGURE OUT THE BEST COMBINATION THAT READS AND WRITES A LOT OF DATA HERE AND PROVIDE YOU THE BEST PERFORMING WHICH IS GREAT, BECAUSE YOU CAN MODEL. YOU CAN RUN THE AUTO ML EXPERIENCE BATCH THAT DATA. WE BATCH IT BEHIND WE HAVE LOCALLY. WE SUPPORT THREE THE SCENE AND YOUR PERFORMANCE IS TASKS THERE. WE SUPPORT REGRESSION VERY GOOD BECAUSE OF THE HIGH BAND BINARY CLASSIFICATION, AND MULTI WIDTH. AND ONE OF THE CHALLENGES CLASS CLASSIFICATION. YOU CAN USE IS LOW LATENCY WORKLOAD. IN SQL THE AUTO ML EXPERIENCE WE BUILT WORKLOAD WITH HBASE OR SPARK STREAMING IN . NET IN THREE DIFFERENT WAYS. OR STORM WHERE YOU NEED TO BE ABLE YOU CAN USE THE TOOL I AM ABOUT TO WRITE VERY, VERY FAST BECAUSE TO SHOW YOU NEXT. YOU CAN USE THE OF THE LOW LATENCY WORKLOAD AND ML. NET CLI WHICH I WAS GOING TO THE STORAGE IS SITTING 20 TO 200 SHOW YOU. WE HAVE API FOR ML. NET MILLI SECONDS A WAY SO HOW ARE YOU THAT YOU CAN TRY OUT. IF YOU WANT GOING TO DO THAT. SO I'M GOING TO TO BUILD MODELS ON THE FLY. SO THE TALK ABOUT THE USE CASE HBASE. HOW NEXT THING I WANT TO GET INTO IS MANY OF YOU UNDERSTAND HOW HBASE THE TOOLING EXPERIENCE THAT I AM WORKS. A FEW PEOPLE. SO LET ME EXPLAIN PROVIDING YOU FOR ML. NET. WE ALSO NOW FOR THE FOLKS WHO HAVE NOT HAD ANNOUNCED AT BUILD THIS YEAR A NEW A CHANCE TO LOOK AT HBASE. SO HBASE TOOL CALLED THE ML. NET NEW BOD IS A NO SQL STORE. AND ALL THE DATA EL BUILDER. THIS IS A VIDEO STUDIO IS STORED IN A SORTED ORDER. SO EXTENSION. IT IS A VERY SIMPLE UI IT'S GOOD FOR HIGH THROUGHPUT WRITING TOOL THAT USES THE CUSTOM MODELS END POINT READS THAT AND WHAT THE AUTOMATICALLY USING THE ML. WHAT USE CASE IS. SO IN HBASE ALL THE IT ALSO DOES FOR YOU IT GENERATES CLUSTERS NODES ARE KNOWN AS REGION CODE FOR MODEL TRAINING AND CONSUMPTION. SERVERS. EACH REGION SERVER CAN LET ME SHOW YOU THIS TOOL IN ACTION. HAVE MULTIPLE REGIONS. IN THIS CASE SO THE WAY YOU CAN LAUNCH THIS TOOL I'M DEPICTING 3 OF THEM. AND THE IS BY RIGHT CLICKING AD AND CLICKING REGION NODES HAVE IN MEMORY. THEY MACHINE LEARNING HERE. IF YOU WANT HAVE A MEMORY. SO WHEN YOU WRITE TO FIGURE OUT WHERE YOU CAN GRAB SOMETHING DATA IS ACTUALLY WRITTEN THIS TOOL YOU CAN GO TO OUR WEBSITE TO THESE H FILES. IT IS WRITTEN QUICKLY. IT IS . NET ML. THIS IS IN MEMORIES. IT'S GREAT AND VERY THE MODEL BUILD ERR PAGE WHICH SHOWS FAST BECAUSE YOU ARE WRITING IN YOU DIFFERENT FEATURES IN MODEL MEMORY. AND THEN AFTER A CERTAIN BUILD ERR. IT ALLOWS YOU TO DOWNLOAD THRESHOLD, SAY YOU HAVE A THRESHOLD THE VISUAL EXTENSION FOR IT. I HAVE YOU -- THE PATH IS VERY FAST BECAUSE ALREADY DOWNLOADED THE EXTENSION YOU ARE COMMITTING TO YOUR STORAGE I DON'T NEED TO DO THAT AGAIN. THIS LATER. BUT WHAT HAPPENS WHEN ANY IS THE FIRST SCREEN MODEL BUILDER. NODES GO DOWN YOUR DATA WILL BE THE FIRST SCREEN WILL SHOW YOU DIFFERENT LOST. SO THE WAY HBASE PROTECTS MACHINE LEARNING USING THIS TOOL. IS BY IMPLY THING -- SO WRITE TO WE HAVE EXAMPLES FOR PRICE PREDICTION THESE MEMORY STRUCTURE IN A HIGHLY WHICH IS AN EXAMPLE OF SENTIMENT DECORATED FASHION SO IF A READ IS ANALYSIS WHICH IS AN EXAMPLE OF COMING YOU CAN -- I WILL ALSO WRITE A TASK BINARY CLASSIFICATION. YOU A LOG TO THE UNDERLYING STORAGE CAN BUILD OTHER SCENARIOS BUILDING IF THIS WERE TO GO DOWN I COULD A CUSTOM SCENARIO TEM MET HERE. RECREATE THAT DATA. SO THAT'S THAT I AM GOING -- TEMPLATE HERE. I AM PURPOSE. SO IF YOU LOOK AT THE OPERATIONS GOING TO CLICK ON ANALYSIS. THE THERE, IF YOU DID THE INSERT, UPDATE, NEXT SCREEN IS ABOUT DATA. THIS GET AND DELETE IT NEEDS TO HAPPEN ALLOWS YOU TO LOAD YOUR DATA. SINCE IN ORDER. YOU CAN NOT DO DELETE I HAVE ALREADY SHOWN HOW TO LOAD BEFORE INSERT HAPPENS. BUT WHAT A FILE LET ME SHOW YOU SQL. I AM YOU SEE WITH ANY KIND OF REMOTE GOING TO CONNECT TO AN AZURE SQL STORAGE IS THAT FOR A LONG TIME DATABASE HERE. CHOOSE SQL OF. LET YOU SEE GOOD LATENCY. BUT ALL OF ME ACTUALLY DO THAT AGAIN. IT IS SUDDEN YOU SEE 300, 700 MILLE SECONDS GOING TO COME BACK HOPEFULLY. ALL SPIKES IN ALL THE -- KIND OF -- RIGHT. SO LET ME JUST LOG IN AGAIN BLOCKED BEHIND THE BIG LATENCY EPISODE. HERE IN SQL. I AM GOING TO TRY IT SO WHAT WE DO FOR THOSE KIND OF AGAIN. >> THANK YOU WHO ALSO BUILDS WORKLOADS WE INTRODUCE THE PREMIUM THE DATABASE FOR THIS DEMO. SO YOU -- SO IN THIS CASE INSTEAD OF WRITING LOG INTO YOUR DATABASE AND NOW WE THAT TO THE BLOB STORE OR -- GENERATION CAN PULL IN ONE OF THE DATABASES. WE WRITE THAT INTO A PREMIUM MANAGED I AM GOING TO USE THE SAME DATABASE DISK ATTACHED TO THE CLUSTER. AND THAT I USED EARLIER. THIS IS CALLED SO THEN YOUR WRITES ARE VERY, VERY YUMMY FOOD. I AM GOING TO CLICK FAST. THEY'RE ASTRO NOMICLY FAST. OKAY. I CAN CHOOSE THE TABLE. IN SO WE SEE, THE GREAT PERFORMANCE THIS CASE THE SAME SENTIMENT REVIEW JUST BY MAKING THE INTELLIGENT CHANGES. TABLE. WHAT THIS IS GOING TO DO WE'RE STILL PUTTING THE DATA IN IS SHOW YOU A PREVIEW FOR WHAT YOU STORAGE. BUT THE WAY WE THOUGHT WOULD SEE ESSENTIALLY. THE NEXT ABOUT, YOU KNOW, THINKING ABOUT TASK THE TOOL ASKS ME TO PROVIDE THE PROBLEMS IN ARCHITECTURE. SO IS THE COLUMN IN WHICH YOU PREDICT. IF YOU ARE USING HBASE THERE ARE I AM GOING TO CHOOSE THE SENTIMENT 2 FLAVOURS ONE WITH THE STANDARD COLUMN AS THE COLUMN I WANT TO PREDICT. STORAGE AND THE OTHER IS -- WHERE ONCE I HAVE DONE THIS I CAN DWOE WE PUT PREMIUM MANAGED DISK SO THE TO THE NEXT PHASE WHICH IS THE TRAINING WRITES ARE VERY, VERY FAST. YOU PHASE. I AM GOING TO GO AHEAD AND WOULD SAY HOW ABOUT READS. YOU CAN TRAIN FOR ABOUT 30 SECONDS HERE. HAVE WORKLOADS WHERE YOU MAY NEED AS THIS TOOL THE MODEL BUILDER IS A LATENCY IN A SINGLE MILLE SECOND TRAINING WHAT YOU ARE GOING TO SEE DIGITALS. SO WE HAVE STARTED SUPPORTING HERE IS AT THE BOTTOM OF THE ACCURACY PREMIUM STORAGE. BUILT ON SSD. SO OF THE MODEL THAT HAS GOT SO FAR YOU ARE GOING TO GET VERY FAST LATENCIES. THE BEST ALGORITHM, THE BEST ALGORITHM QUESTIONS? >> SO FAR SO GOOD? GOOD IT HAS CHOSEN AND THE DIFFERENT DEAL. SO AGAIN, SO WE TALKED ABOUT ALGORITHMS IS TRYING OUT. THE MORE THESE TECHNIQUES LOCAL VERSUS REMOTE. TIME YOU WILL PROVIDE THE TOOL, BUT HOW SHOULD YOU THINK ABOUT YOU THE MORE TIME YOU HAVE DIFFERENT KNOW THE MORE CHEAPER STORAGE, A ALGORITHMS AND DIFFERENT MODELS REMOTE STORE AND MAKING IT FASTER? FOR YOU. IN THIS CASE MY DATA SET SO THE CACHING TECHNIQUE IS VERY, WAS SMALL BUT IF YOU HAVE DATA SETS VERY IMPORTANT AND THERE ARE MULTIPLE THAT IS MORE THAN A GIG IT MIGHT DIFFERENT WORKLOADS AND CACHING TAKE A COUPLE OF HOURS. IF YOU HAVE TECHNIQUES. AT SPARK WE DO SPARK A DATA SET ABOUT A TERABYTE THAT IO CACHE. IT'S BASED ON R CACHE MAY TAKE A COUPLE OF DAYS. I DID WHICH IS AN APACHE PROJECT. SO IF IT FOR 30 SECONDS. YOU SEE THE ACCURACY YOU ARE READING FROM SPARK IT'S WE GOT IS 86 PERCENT. IF YOU REMEMBER CACHED LOCALLY AND IF YOU ARE ISSUING FROM MY LAST EXAMPLE THAT I DID THE SAME QUERY THEN IT CAN BE VERY, WHEN I CREATED THE SENTIMENT ANALYSIS VERY FAST. IN HBASE AND PHOENIX MODEL THAT ONLY THIS 80 PERCENT APART FROM WHAT I TALKED ABOUT. ACCURACY. IT IS ONLY 30 SECONDS THERE IS BUCKET CACHING. SO AGAIN, OF TRAINING. ONCE MY TRAINING IS RECENTLY WRITTEN DATA, RECENTLY COMPLETE I CAN GO TO THE NEXT SCREEN READ DATA WOULD BE PUT ON THE L2 NOW WHICH IS THE VALUE SCREEN. THE CACHE AND YOU CAN SERVE THAT DIRECTLY VALUE SCREEN WILL ALLOW YOU TO LOOK AND THEREBY GIVING THE ILLUSION AT THE BEST MODEL PERFORMANCE. THE THAT IT'S VERY, VERY FAST. IF YOU DIFFERENT MODELS EXPLORED AND SO ARE LOOKING AT HIGH -- IN TERMS ON. IF AT THIS STEP YOU ARE UNHAPPY OF SERVING LAYER IT HAS SOMETHING WITH THE ACCURACY YOU GOT, YOU CAN CALLED INTELLIGENT CACHE WHERE ALL GO BACK AND TRAIN FOR LONGER. THAT OF THE DATA IS AGAIN CACHED INTO RESULTS IN BETTER ACCURACY. YOU THE LOCAL MEMORY. AND WHAT I SAY CAN ADD MORE DATA TO THE PROBLEM LOCAL MEMORY THERE ARE 2 PARTS TO AND SO ON. THE LAST STEP HERE THE IT. DYNAMIC RAM AND THE LOCAL SSD. MODEL BUILDER ALLOWS YOU TO DO IT SO IF YOU ARE CREATING ONE OF THE ALLOWS YOU TO GENERATE CODE AUTOMATICALLY. CLUSTERS WITH A D14. VM TYPE. IT WHEN I CLICK PROJECTS HERE IT IS HAS ABOUT 800GB OF LOCAL SSD. FREE. GOING TO CREATE TWO PROJECTS. THE NO CHARGE: I GUESS IT'S BUNDLED FIRST PROJECT THAT IT CREATES IS IN. THAT'S WHAT IT IS. SO BETWEEN THE CLASS LIBRARY. WHAT THE CLASS RAM, SSD AND REMOTE STORE, THE ENGINE LIBRARY HAS IS A MODEL ZIP FILE ITSELF TAKES CARE OF HOW TO RECYCLE WHICH IS A TRAINED MODEL THAT JUST THE BITS. SO IT'S USEFUL IN THOSE TRAINED IN THE TOOL IT HAS THE INPUT CASES. SO DO CONSIDER USING THESE AND OUTPUT CLASS. ALONG WITH THE OPTIONS WHEN YOU BUILD THESE OPTIONS. MODEL AND INPUT AND OUTPUT CLASSES SO THIS IS THE IO CACHE. YOU KNOW, WE CITE ATED ANO -- CREATED PROJECT YOU JUST ACTIVATE IT. AND THEN YOU CODE. IF I SHOW YOU THE TRAINING GET THE CACHING. AT THE BOTTOM IS CODE QUICKLY YOU CAN SEE HERE THAT THE -- ON 1TB SCALE. AGAIN IN TB THIS IS ESSENTIALLY THE SAME FEATURIZEZATION -- THE NOT EVERY QUERY RESPONSED THAT WE DID BY MANNED EARLIER. IN FAVORABLY TO CACHING. SOMETIMES THIS CASE IT IS A BETTER PERFORMING THE CACHING WILL NOT WORK. BUT ON LEARNER. IT ALSO FOUND THESE HYPER A FULL WORKLOAD IT'S 2X FASTER. PARAMETERS WHICH PROVIDED THE BEST BUT THERE ARE 20 PLUS QUERIES IN PERFORMANCE SO YOU CAN SEE HOW THE -- BENCHMARK WHERE IT'S 8 TO 10X EXPERIENCE IS HELPING YOU CREATE FASTER. SO THE QUERIES, LIKE REPEAT THE BEST PERFORMING MODEL. THE OTHER QUERIES THAT IS WHERE YOU WILL SEE FILE WILL YOU SEE IN THE PROJECT A LOT OF PIECES. SECURITY. THERE IS THE PROGRAM CS FILE. WHAT WE IS A QUESTION. [OFF MIC] >> YES, ARE SHOWING IS HOW YOU CAN NOW USE WE HAVE. BUT WE'RE NOT ALLOWED TO A TRAINED MODEL SO YOU START WITH SHARE THE NUMBERS. GOOD QUESTION A GANG CREATING THE CONTEXT AND THOUGH. [LAUGHTER] >> OKAY. ALL LOAD ML. NET MODEL AND YOU CAN START RIGHT. SO LET'S TALK ABOUT FROM MAKING PREDICKS. I AM GOING TO HIJACK A SECURITY, THIS IS IN A DATA ARCHITECT THIS CODE A LITTLE BIT HERE AND SECTION. BUT SECURITY IS TOP OF ADD MY OWN SINGLE INSTANCE OF DATA MIND WITH EVERYBODY. SO THE WAY AND SAY, NEW, YOU CHEAT A LITTLE WE THINK OF SECURITY, YOU NEED TO BIT AND SEE WHAT THE INPUT IS. THE THINK OF AUTHORIZATION AND DATA FIELD CALLED NEW SENTIMENT. I AM PERSPECTIVE. YOU CAN PUT THIS CLUSTER GOING TO GO BACK AND CALL NEW SENTIMENT. INSIDE A VERY SECURE NETWORK. AND I AM GOING TO SET THIS TO BE THE THAT INFORMATION IS OUT THERE QUITE SAME THING I TRIED EARLIER WHICH A BIT. FOR AUTHENTICATION AND AUTHORIZATION IS MACHINE LEARNING IS FUN. LET WE DEPEND ON AZURE DIRECTIVE, IT'S ME TRY THIS AGAIN. SO IF I RUN THIS APACHE. ALONG WITH THE -- WHAT YOU NOW, I SET MY BREAK POINT HERE WHAT CAN DO ON JEN 1 OR JEN 2. AND DATA YOU WILL SEES THE GENERATED CODE PROTECTION. THE DATA IS PROTECTED THAT THE MODEL CREATED FOR ME AND IN TRANSIT OR STORAGE. SO ALL THE NOW I AM ESSENTIALLY GENERATING PIECES ARE THERE. I WANT TO EXPLAIN THE CODE TO MAKE A SINGLE PREDICTION. THE DIAGRAM BECAUSE IT'S A BIT COMTEMPLATED. WE ARE GOING TO SEE THE DEBUGGER IF YOU LOOK AT THE OPEN SOURCE TECHNOLOGY AGAIN IF I GO TO THE PREDICTION THE SPARK, HIVE AND HBASE AND OTHERS. METRIC YOU WILL SEE THE PREDICTION IN OPEN SOURCE, FROM A PROTOCOL IS TRUE YOU SEE THE PROBABILITY PERSPECTIVE, ALL OF THEM WORK ON AND SCORE AS WELL. THAT IS KIND A -- SO THAT'S THE PROTOCOL. BUT OF LIKE A VERY QUICK DEMO ON WHAT IF YOU LOOK AT AZURE ACTIVE DIRECTY MODEL BUILDER CAN DO FOR YOU. MODEL STORED IN THE CLOUD. SO THESE ARE BUILDER WE CAN USE THAT IN THE TOOL. 2 DIFFERENT PROTOCOLS. AND SO THEN HOPEFULLY THIS MAKES BUILD MACHINE WHAT THAT MEANS IS THAT IT'S NOT MODELS EASY WITH ML. NET. I AM GOING AS EASY AS JUST POINTING THE CLUSTER TO SWITCH BACK TO THE DECK HERE TO AZURE DIRECTY AND THINGS WILL QUICKLY. THE NEXT THING I WANT TO WORK. THEY DON'T BECAUSE THEY'RE SHOW YOU IS A COUPLE OF OTHER TOOLS. DIFFERENT PROTOCOLS. SO THEN WHAT FOR THAT PORTION OF THE TALK I AM HAPPENS WE USE A COMPONENT AZURE GOING TO LEAVE YOU WITH CESAR WHO ACTIVE DIRECTY. CALLED AZURE ACTIVE IS GOING TO COVER THIS. >> THANK DIRECTY DO MAIN SERVICES. IT'S BASICALLY YOU. [APPLAUSE] >> COOL. THAT WAS THE DO MAIN SERVICES INSTANCE. SO AN AWESOME DEMO. WHAT WE ARE GOING WE SYNY THE USER IDENTITIES TO THE TO SEE NOW IS WHAT ANOTHER APPROACH, DO MAIN SERVICES AND THEN BASICALLY ANOTHER TOOL THAT WE CREATED USING ACTS AS A SERVER THAT CAN ISSUES ML BUT INSTEAD OF USER STUDIO THERE THE TICKETS TO OUR CLUSTER. SO IT'S MAY BE OTHER USERS THAT WANT TO A BIT OF COMPLICATED SET UP, BUT USE THE COME ONLINE INTERFACE FOR IT WORKS. SO LET'S LOOK AT A LITTLE INSTANCE ONE COULD BE THIS IS CROSS BIT OF DEMO IN TERMS OF HOW, YOU PLATFORM YOU CAN RUN THE CLI ON KNOW, ONCE YOU GO THROUGH THE SETUP THE MAC, LINUX OR WINDOWS. ANOTHER AND HOW IT SORT OF WORKS AND WHAT REASON IS BECAUSE ONCE YOU KNOW CAN YOU DO WITH IT. SO IN THIS CASE, THE PROCESS AND MAYBE YOU WANT TO I HAVE A CLUSTER. SORRY, LET ME GENERATE A MODEL EVERY DAY OR WITH SWITCH. YOU GUYS CAN SEE IT. SO NEW DATA OUT OR YOU WANT TO AUTOMATE I HAVE A CLUSTER AND WHAT YOU SEE WITH THE CLI IN A DIFFERENT PIPELINE RIGHT NOW. THIS IS IS A CLUSTER OR WHATEVER, THEN IT IS VERY USEFUL MANAGER. IT DOES A FEW THINGS. SUCH TO HAVE ALSO A CLI. LET'S SEE A AS MONITORING CONFIGURATION. SERVICE DEMO ABOUT IT. SO HERE, I HAVE WINDOWS. MANAGEMENT AND SO FORTH. AND ALSO IN THIS CASE I AM GOING TO RUN IT PROVIDES CERTAIN TOOLS. A FEW THINGS. WITH POWER SHOW. YOU CAN RUN THE SO FOR EXAMPLE, ONE OF THE THINGS SAME DEMO OP A MAC OR COME ONLINE IS PROVIDES AND HIGH VIEW. YOU CAN AS WELL OR LINUX. THE WAY YOU GET WRITE HIGH QUERIES IN THE DATA LAKE. THE CLI IS THIS IS A GLOBAL TOOL. AND NOW I TALKED ABOUT THE AUTHORIZATION YOU INSTALL IT AS A GLOBAL TOOL COMPONENT WHICH IS APACHE RANGER. AS YOU CAN SEE HERE WITH . NET TOOL THIS IS IT. AND IT HAS A LOT OF INSTUALLED OR ML. NET WHICH IS THE DIFFERENT PLUG INS FOR A LOT OF NEW GET PACKAGE YOU AUTOMATICALLY DIFFERENT THINGS. SO WE'LL LOOK INSTALLED. I GET THIS I DON'T NEED AT HIGH PLUG-IN. AND WE'LL DEFINE TO DO IT NOW. THE OTHER THING I THE HIGH ACCESS POLICIES HERE. AND WANT TO SHOW YOU IS ALSO HOGWARTS I'LL CLICK ON THIS ONE. SO ONE OF WHERE YOU CAN WRITE ML. NET, OUTER THE POLICIES I ACTUAL HAVE A TABLE TRAIN AND THEN TASK. THEN WE HAVE CALLED HIGH SAMPLE TABLE A DEFAULT ALSO TAP OUT OF COMPLETION YOU CAN TABLE THAT COMES WITH THE CLUSTER PRESS TAP AND SEE A DIFFERENT ML AND SO ON THE HIGH SAMPLE TABLE TASK WHICH IS BINARY CLASSIFICATION WHAT I WANT TO BE ABLE TO DO, I MULTI CLASS SPECIFICATION AND REGRESSION. WANT TO MASK SOME DATA, BECAUSE IN UP COMING VERSIONS WE WILL HAVE I DON'T WANT TO A CERTAIN USER TO THE REST OF THE ML TASKS WE HAVE KIND OF SEE THIS DATA. SO I COULD IN ML. NET. LET ME RUN ONE SAMPLE SHOW YOU LIKE WHAT I HAVE DONE HERE. USING THE SAME SIMILAR DATA SET AND SO I NAMED IT LIKE SPARK POLICY. WE WERE USING. BASICALLY YOU CAN HERE IS YOUR TABLE. AND FOR THIS SEE ML. NET OUTER TRAIN BINARY CLASSIFICATION COLUMN, WITHIN THAT TABLE CALLED THEN THE NAME OF THE CODE OR THE CLIENT ID, FOR THIS USER -- AND FOLDER IS GOING TO BE THE SENTIMENT THIS IS THE DOMAIN USER, THIS IS MODEL. THE NAME OF THE DATA SET THE USER IN THE AZURE ACTIVE DIRECTY. IS LABELED TSV. IT IS TEXT FILE I WANT TO BE ABLE TO REDACT THAT WITH TABS BETWEEN COLUMNS. FINALLY DATA. AS SIMPLE AS THAT. SO NOW I NEED TO PROVIDE THE LABEL COLUMN COMES HERE AND RUNS THE QUERY HERE. NAME WHICH IS WHAT IS THE COLUMN AND AS YOU CAN SEE I'M LOGGED IN I WANT TO PREDICT? I WANT TO USE AS ASHISH. AND IT BRINGS THE DATA A TARGET. RIGHT? FINALLY THE TIME AND YOU SEE THE CLIENT ID IS REDACTED. THAT I AM GOING TO BE LOOKING FOR NOW THERE IS AN OTHER USER HERE, BETTER MODELS. IN THIS CASE JUST AND HER NAME IS ALICE. AS YOU CAN 15 SECONDS BECAUSE IT IS A QUICK SEE HERE: LET ME MAKE IT A LITTLE DEMO. BUT WHEN YOU ARE WORKING WITH BIGGER. AND SO THIS IS ALICE. AND LARGE INTERSETS YOU MIGHT NEED MANY AS YOU CAN SEE IN THE POLICY I DID MINUTES OR HOURS. YOU CAN SEE IT NOT MENTION ALICE. SHE HAD ALL THE SWEEPING, IT IS LOOKING FOR BETTER ACCESS AND NOW ALICE IS RUNNING ALGORITHMS AND YOU CAN SEE THE BEST THE SAME QUERY, SELECT STAR FROM ACCURACY SO FAR WAS 87 AND FINALLY THE SAMPLE TABLE AND NOW YOU CAN YOU GET A SUMMARY WITH FIVE BEST SEE SHE CAN SEE THE ACTUAL DATA. ALGORITHMS, THE BEST ONE IN THIS THIS DATA IS NOT DATA LAKE. THIS CASE IS LB AT GS LOGISTIC REGRESSION. IS RAW TEXT FILES SITTING IN THE YOU CAN GET MORE INFORMATION ABOUT GENERATION TOOL. AND NOW YOU ARE THESE METRICS IN THE URL WE HAVE ABLE TO DO COLUMN LEVEL SECURITY HERE. MOST IMPORTANTLY WE ARE GENERATING IN TERMS OF WHICH USER CAN SEE WHICH THE CODE AND THE MODEL, THE MODEL DATA FROM THE FILES. SO THAT'S VERY THAT WAS CREATED WHEN TRAINING AND POWERFUL. SO THIS IS HIVE OPERATIONS THE CODE FOR TRAINING IT. SOMETHING AND OF COURSE YOU EXPECT US -- YOU THAT MAYBE WAS NOT CLEAR WHEN DOING KNOW, HIVE TO INSPECT ALL THESE THE DEMO PREVIOUSLY, ONCE YOU HAVE PIECES. EACH UNONE OF YOU -- YOU TRAINED A MODEL THEN YOU CAN SAVE WANT TO ACCESS THE UNDERLYING STORAGE IT AS A [INDISCERNIBLE] FILE. AND DIRECTLY. HOW DOES THAT WORK. HOW YOU WILL LOAD THAT FROM YOUR END DO THE IDENTITIES GET PROPAGATED USER APPLICATION, LIKE A WOOD APPLICATION. FOR UNDERLYING STORE. SO LET'S LOOK THE ML MODEL DOT ZIP FILE AND THEN AT THAT. SO I WANT TO LOOK AT AZURE THE CODE. THE CODE IS SIMILAR. IT STORAGE EXPLORER. SO HERE IS AZURE WAS GENERATED HERE. YOU CAN SEE STORAGE EXPLORER AND WITHIN THAT THAT NOW WE HAVE THIS FOLDER, WHICH I HAVE A FOLDER CALLED "TEST "AS IS NEW. IT IS STARTING WITH THE YOU CAN SEE THERE. AND IF I LOOK TSV. THIS HAS BEEN GENERATED. WE AT MANAGE ACCESS, AND AS YOU CAN HAVE THE SAME PROJECTS FOR THE CLASS SEE -- ON MICROSOFT. COM HAS ACCESS LIBRARY AND CODE FOR TRAINING AND ON THIS FOLDER. BUT YOU DON'T SEE SCORING EXACTLY THE SAME WITH VISUAL ALICE IN THERE. SHE HAS NO ACCESS STUDIO. IT IS THE SAME BECAUSE VISUAL ON THE FOLDER. SO NOW, THIS USER, STUDIO WAS DOING THAT ON TOP OF LET'S SAY IN THIS CASE ASHISH, SEARCHES THE CLI. WE ARE CONSISTENT AND REALITY IN THE CLUSTER NODE. AND I'M SEARCHING IS THE SAME THING. LET ME GO BACK TO A CLUSTER NODE AND FROM THERE TO THE NEXT SECTION. IT IS ABOUT I'LL DO THE COMMAND. SO NOW I'M SCALING AND GOING TO PRODUCTION. ABFS AND HERE IS THE CONTAINER AND SO FAR WE HAVE SEEN HOW YOU CAN I WANT TO BROWSE THE CONTENT OF TRAIN YOUR MODEL, HOW YOU CAN TEST THE FOLDER AND SO I HIT ENTER. AND IT, VALIDATE IT, BUT WHAT ABOUT IT SAYS, 2 RECORDS FOUND. YOU SAW PUTTING THIS IN YOUR REAL OBLIGATIONS THAT ASHISH HAD ACCESS TO THE UNDERLYING IN . NET ALLEGATIONS OR HTL NET. STORAGE. WE WERE ABLE TO PASS THAT NET FRAMEWORK AND. NET CORE. IT IS IT GOT RESPECTED. SO SAY ALICE IS CRUSH PLATFORM YOU CAN RUN IT HAD NO PERMISSIONS THERE AND WANTS ON WINDOWS LINUX OR MAC. WHEN MOVING TO BROWSE A TEST FOLDER AND HIT TO PRODUCTION, IT IS NOT JUST ABOUT ENTER AND SHE SO SAY REQUEST DENIED PREPARING YOUR DATA AND BUILDING BECAUSE SHE DOESN'T HAVE PERMISSION. OR TRAINING IT IS ALSO ABOUT RUNNING SO WHAT I'M SAYING IS YOU COULD THAT MODEL IN YOUR APPLICATION. SET THE PERMISSIONS AT JUST THE YOU MIGHT HAVE QUESTIONS LIKE THIS DATA LAKE ITSELF. AND YOU'RE USER LIKE HOW CAN I OPTIMIZE THAT FOR IDENTITIES WOULD FLOW FROM CLUSTER RUNNING AN ASP NET APPLICATION OR AND THEY WOULD BE RESPECTED. THIS MULTI THREADING OR HOW CAN I INCLUDE IS USEFUL FOR SCENARIOS YOU ARE THE CREATION OF THAT MODEL THAT MOUNTING MULTIPLE CLUSTER AGAINST WE WERE SEEN WITH ANKIT IN MY CI/CD DATA LAKE. LET PEOPLE DO WHATEVER PIPELINES. HOW CAN I AUTOMATE THAT. ENGINE THEY WOULD LIKE IN THE ORGANIZATION. LET ME DO ONE DEMO ABOUT HOW YOU SO THAT'S THE GOOD PART ABOUT IT. CAN USE A MODEL INTO ASV NETWORK OKAY. SO NOW WE JUST SAW 2 THINGS APPLICATION. SO 8. SO HERE YOU HAVE AND MAY HAVE CONFUSED SOME OF YOU. A SOLUTION WHERE I HAVE AN API THAT THERE ARE NOW 2 AUTHORIZING ENGINS WE WILL TAKE A LACK IN A MINUTE. IN THE SCHEME. APACHE RANGER AND WE HAVE TWO PROJECTS THAT THOSE THE OTHER ONE. WHICH ONE WOULD APPLY ARE PRECISELY GENERATED BY THE CLI AND WHEN. SO HERE IS TABLE FOR YOU. OR VISUAL STUDIO. IT IS A MODEL SO WHEN YOU THINK ABOUT APPLICATION PROJECT THAT HAS A ZIP FILE AND LEVEL ACTIONS. SUCH AS YOU ARE DOING DATA CLASSES THAT THEY WERE SHOWING. A HIGH QUERY TO GET THE DATA FROM LATER WHEN WE ARE DOING UNIT TESTING. HIVE OR SPARK. IN THOSE CASES, THE NOW I AM GOING TO FOCUS ON HOW CAN HIGH PLUGGIN OR IN THE CASE -- THEY I USE THIS MODEL ZIP FILE. THERE BECOME THE AUTHORIZER. SO THAT'S ARE BASICALLY THREE CLASSES THAT THE PIECE. BUT LET'S SAY YOU ARE YOU NEED TO USE WHEN RUNNING A MODEL. CREATING A HIGH TABLE, AND IN HIVE ONE IS THE I TRANSFORM ERROR MODEL, TABLE YOU WRITE THE SCHEMA, BUT SO THOSE ARE THREADS SAVED. YOU YOU ALSO TELL A LOCATION IN TERMS CAN PUT A SINGLE TONE OR [INDISCERNIBLE] OF WHERE THE DATA IS. SO YOU ARE IT WOULD BE BETTER YOU CAN REUSE POINTING TO A LOCATION WITHIN THE IT FROM DIFFERENT THREADS IN YOUR STORAGE. IN THAT CASE YOU HAVE TO APPLICATION. IT IS THE PREDICTION HAVE A PERMISSION IN BOTH PLACES. ENGINE WE TALKED ABOUT FOR DOING YOU NEED IT IN RANGER OR UNDERLYING SINGLE PREDICKS. THAT IS NOT A THREAT STORE. AND IF YOU THINK ABOUT A SAVED. YOU NEED TO USE IT IN A SPECIAL SPARK DATA API WHERE YOU ARE DOING WAY WHEN YOU ARE RUNNING A MULTI -- YOU ARE READING A CSV FILE IN THREAT APPLICATION. THAT'S WHAT A STORAGE LOCATION AND IF YOU DON'T I WANT TO SHOW YOU IN THE API. YOU HAVE PERMISSION YOU WILL GET ACCESS CAN SEE THAT I HAVE COPY THE ZIP DENIED. SO IN ALL OF THE SCENARIOS FILE FOR THE MODEL. I HAVE THE DATA WHERE YOU ARE DIRECTLY REFERENCING CLASSES FOR READING AND USING THE TO THE STORAGE, AT THE MINIMUM YOU MODEL. WHAT WE DID IS BECAUSE DURING MUST HAVE PERMISSION AT THE STORAGE THIS IN A SCALABLE WAY, YOU USE LEVEL AND ALSO HAVE PERMISSION AT THE [INDISCERNIBLE] FOR PREDICTION. THE RANGER LEVEL WITH DIFFERENT I WROTE A POST ABOUT THAT AND HOW PLUGGINS IN THERE. ALL RIGHT, ANY YOU CAN DO THAT. A FEW MONTHS AGO QUESTIONS? GOOD DEAL. RESILIENTSY. IN THE ASVI TEAM WITH RYAN NOVAK WE DON'T TALK MUCH ABOUT THESE PIECES LET'S CREATE A . NET, AS VDOT NET BUT I THINK WE SHOULD. A LOT OF EXTENSION PACKAGE SO YOU CAN USE THINGS CAN GO WRONG. RIGHT? . SO IT THE SAME WAY YOU CAN USE FOR LET'S SAY YOU ARE -- YOU HAVE A INSTANCE SIGNAL R OR [INDISCERNIBLE] SYSTEM WHERE , YOU KNOW, YOU BUILD FRAMEWORK. THEN YOU CAN -- IT IS A LOT OF CLUSTER EVERY SINGLE DAY. GOING TO BE SCALABLE. IT IS SUPER YOU PROCESS THINGS WHERE YOU SCALE EASY. YOU GO TO THE START UP CLASS UP AND SCALE DOWN. IF YOU LOOK AT AND THE CONFIGURE SERVICES. YOU THE CLUSTER, IT IS MADE UP OF A JUST NEED TO REGISTER THE PULL WITH BUNCH OF AZURE SERVICES. VM, STORAGE, THE SAME DATA CLASSES WE WERE MENTIONING YOU NAME IT WE USE IT. AND ON TOP WHEN CREATING A MODEL. LOADING FROM OF THAT THERE IS MULTIPLE OPEN SOURCE A FILE WHICH IS THE PATH TO THE FRAMEWORKS WE WIRE UP TO BUILD THIS STEEP FILE. WE REGISTERED THAT AS CLUSTER. THINGS CAN GO WRONG. IF YOU WOULD DO A SIMILAR THING QUICKLY. ANY ONE OF THE COMPONENTS -- IF USE INJECTION. GET THE OBJECT. PREDICTION YOU HAVE A SYSTEM WITH SO MANY COMPONENTS, AND PULL AND COLD PREDICT WITH THE THE RELIABILITY IS THE MAXIMUM OF DATA THAT CAME FROM HTTP. FOR INSTANCE THE MOST UNRELIABLE COMPONENT. THAT'S YOU CAN SEE THAT WE ARE SENDING WHAT IT COMES DOWN TO. SO ACCOUNT THIS DATA. ML. NET IS AWESOME. IT FOR THE FAILURES, RIGHT, YOU ARE IS SAYING SENTIMENT THE PROBABILITY CREATING CLUSTERS. ONCE IN A WHILE FOR THAT TEXT. LET'S MOVE ON. THE THERE WILL BE A CLUSTER THAT FAILS DEMO I WANTED TO SHOW YOU WITH THE TO COME UP. IMPLEMENT AND RETRY API. ANOTHER THING IS ABOUT CICD LOGIC. SAME WITH THE MANUAL SCALE AND DevOps. WE WERE RUNNING THIS UP AND SCALE DOWN. WHEN YOU SCALE IN PRODUCTION YOU WANTED TO BE ABLE DOWN, AND SOMETIMES THIS HAPPENS TO HAVE CONSISTENT CODE WITH THE QUITE A BIT. SO LET'S SAY, YOU HAVE MODEL WITH PARTICULAR DATA AND FOR LOCAL DAT DATA. AND NOW YOU SHRINK THAT YOU NEED TO ENGAGE THE CREATION THE CLUSTER ALL THE WAY BACK TO OF THE MODEL IN YOUR PIPELINE. I 3 NODE OR 1 NODE CLUSTER, RIGHT? AM GOING TO DO A DEMO ABOUT THAT. AND WE DO DECOMMISSIONING. SO THE I WILL TRY TO DO IT QUICKLY. BASICALLY NODE WILL BE DECOMMISSIONED. THE I AM GOING TO START FROM A TROUBLE ONE YOU SCALED DOWN AND THE LOCAL CASE WHICH IS I HAVE A FEW AZURE DATA GETS REPLICATED ON THE REMAINING APPLICATION WEB APPS IN LINUX, WINDOWS NODES. IF IT'S DRASTICALLY SCALED WHERE I WANT TO EMPLOY API. ONE DOWN THE NODE MAY NOT HAVE SPACE OF MY PEERS THEY CHANGED THE DATA AND THE CLUSTER CAN GET INTO SOMETHING SET AND SWITCHED THAT TO GitHub A NAME NODE GOING INTO A SAFE MODE. AND DEPLOYED THAT DIRECTLY FROM IT DOESN'T UNDERSTAND. WHERE IS VISUAL STUDIO INTO AZURE. THE WEB MY DATA. I'M PANICKING, AND SO THERE API IS WORKING. THIS IS YOUR URL, ARE A FEW MANUAL STEPS I LISTED DEPLOY THE APP SERVICE IS WORKING THERE TO BRING CLUSTER OUT OF THE WRONG. YOU WILL SEE THE ML NET IS NAME NODE GETTING INTO THE SAFE AWESOME BUT THE POSITIVE SEPTEMBER MODE. OTHER THING THAT WE -- IF SENTIMENT IS FALSE. IF I SAY THE YOU ARE DOING A SCALE UP AND SCALE FOOD IS HORRIBLE THE POSITIVE SENTIMENT DOWN TO SAVE COST. WE ANNOUNCED IS TRUE WHICH IS WRONG. I GO AHEAD AUTOO SCALE. SO YOU COULD LEAVE AND I CAN GO TO MY BUILD PIPELINES YOUR SCALING OF YOUR CLUSTER TO AND LOOKS LIKE WHEN THE RUN DATA US AND BASED ON A LOAD OR SCHEDULE. SET WAS PUBLISHED I AM GETTING ERRORS WE CAN SCALE DOWN THE CLUSTER. IT'S HERE I CAN SEE I HAVE QUITE A FEW A GRACEFUL SCALE DOWN. WE WOULDN'T TESTS THAT DIDN'T PASS. AT LEAST DRASTICALLY FROM 100 TO -- NODE. I CAN FIX THIS. WHAT I WILL DO I WE DO IT IN A GRACEFUL FASHION. WILL EXPLAIN IN THE CODE I AM GOING SO THE WAY WE DO IT IS SET UP. AND TO TRIGGER A NEW BUILD BY PROVIDING SAY WHETHER I WANT A LOAD BASE OR THE GOOD DATA SET. SO I AM JUST TIME BASED. AND THEN IF YOU SAY COPYING THIS DATA SET, RENAMING TIME BASE THEN WE USE SCHEDULE A IT. THEN I AM GOING TO PUSH IT INTO SPECIFIC TIME WHEN YOU WANT CLUSTER GET WHICH IS WHERE I HAVE THE DATA TO BE SCALED DOWN AND THEN MONITOR. SET. DEMO, PUSH, GOOD DATA SET. THIS IS A MUCH CLEANER OPTION BECAUSE NOW IT IS GOING TO BE IN GitHub, WE DO IT VERY THOUGHTFULLY AND PURPOSEFULLY. AND SINCE I HAVE CONFIGURED CONTINUOUS AND YOU WILL BE IN LESS TROUBLE. INTEGRATION AND DEPLOYMENT IT IS SO NOW YOUR CLUSTER IS UP IS RUNNING GOING TO TRIGGER THIS BUILD. IF AND DEALT WITH THE SCALE UP AND I GO BACK TO MY BUILD, YOU SHOULD SCALE DOWN FAILURES. SO WITH THE BE HERE. YOU SEE HERE IS THE STARTING. WAY YOU BUILD CLUSTERS -- SO MANY THIS IS GOING TO TAKE A FEW LIKE DIFFERENT COMPONENTS AND ANY COMPONENT A COUPLE OF MINUTES. SO WHILE THIS CAN GO DOWN AT ANY POINT IN TIME. IS WORKING, FIRST OF ALL I WANT THERE IS A BUILT IN RESIS SDIL -- TO SHOW YOU THE CODE ABOUT THE TESTS IF A WORKING NODE GOES DOWN. IT WHICH IS INTERESTING. IN THIS CASE WILL PROBABLY TAKE MORE TIME TO THE UNIT TEST I HAVE HERE ARE TESTING COMPLETE BECAUSE THE NODE WENT DOWN THE MODEL. ONE TEST YOU CAN DO YOU BUT THE JOB WON'T BE IMPACT BY THAT. CAN RUN THIS TEST HERE. ONE TEST AND IN A DISTRIBUTED SYSTEM THINGS WOULD BE ABOUT SIMPLY TESTING THAT DO GO DOWN. YOU KNOW, THINGS ARE THE STATEMENT, THE NEGATIVE STATEMENT MEANT TO GO WRONG AND A SYSTEM IS WITH A WRONG -- SORRY. WITH A BAD BUILT IN A WAY YOUR SYSTEM WON'T SENTENCE OR NEGATIVE SENTENCE IS BE IMPACTED. HOWEVER CATASTROPHIC DOING IS RIGHT. LIKE THIS ONE. THIS FAILURE COULD HAPPEN. A STORAGE MOVIE IS BORING IT IS GOING TO PREDICT OR NETWORK OUTAGE CAN HAPPEN. SO FALSE THE OTHER ONE WILL BE TRUE. PLAN FOR DISASTER OPTION. THERE OTHER TESTS EVEN MORE INTERESTING IS A BUNCH OF WAYS YOU COULD DO I AM LOADING THE MODEL AND I AM DISASTER RECOVERY. I HAD SOME RESOURCES GETTING THE METRICS LIKE ANKIT SHOWED THERE WE WALK YOU THROUGH A LAB AND I AM SAYING HEY, IF THE ACUITY HOW TO SET IT UP. SO THE RESOURCES IS NOT HIGHER THAN W80 PERCENT IT ARE DOWN THERE. FOR HBASE THEY RELY IS NOT PASSING THE TEST. BREAK THE ON HBASE REPLICATION IMPORT AND BUILD. EVEN FURTHER YOU CAN ALSO EXPORT. SO THIS IS A TOPIC IN ITSELF. GET A LOT OF LIKE HUNDREDS OF RECORDS I COULD SPEND AN HOUR JUST TALKING AND JUST TEST ALL OF THOSE HERE. ABOUT -- BUT I RECOMMEND GOING THROUGH THIS IS WHAT WE ARE DOING PRECISELY THIS MATERIAL AND MAKING YOURSELF IN THE BUILD. I WANT TO EDIT THEN FAMILIAR WITH THAT. LAST IS MONITORING. THE BUILD JUST TO SHOW YOU A FEW BEFORE I TALK ABOUT MONITORING, STEPS OF THE BUILD. YOU CAN ALSO ANY QUESTIONS? >> I HAVE A QUESTION DO IT WITH THE NEW YAML I WOULD ABOUT -- AND FOLLOW-UP TO THAT, DO IT IN YAML INSTEAD OF THE TASK IF WE ALREADY HAVE HDINSIGHT SET HERE. IT IS THE LEMAHIEU WITH -- UP NOW. .. . >> SO THE QUESTION YAML BUT BETTER TO SHOW THE DIFFERENT TASKS. WITH THE BUILD YOU NEED TO BUILD THE APPLICATION FOR TRAINING, TRAIN THE MODEL, THEN WE BUILD A UNIT TEST PROJECT, RUN THE UNITS TEST, COPY THE FILE THAT WAS GENERATED WITH THE APPLICATION BUILDING THE MODEL INTO THE WEB API. WE BUILD THE WEB API AND FINALLY WE PUBLISH IT PIPELINE ARTIFACT. IF YOU CAN GO AND SEE IF IT WAS FINISHED. IT IS FINISHING, AND THEN RIGHT NOW IT IS FINISHING, AND NOW IN A FEW SECONDS YOU WILL SEE THAT I AM GOING TO SEE THE TEST. AND THEN IT IS STARTING AND TRIGGERING ALSO A RELEASE. IT IS TRIGGERED BECAUSE OF THE ARTIFACT. YOU CAN SEE THIS RELEASE THAT IS NOW PUBLISHING THE WEB API INTO MY STAPLING ENVIRONMENT WHICH IS TWO DIFFERENT AZURE APP SERVICE ONE IN LINUX AND ONE IN WINDOWS. ASK ME IF I WANT TO GO TO PRODUCTION BUT IN THIS CASE I HAVE TO HAVE A MANUAL APPROVAL. IT IS GOING TO FINISH IN A MINUTE. WHILE IT IS FINISHING I WANT TO SHOW YOU THE PIPELINE FOR THE RELEASE. YOU CAN SEE THAT I AM DEPLOYING TO A WEB API FOR LINUX AND WINDOWS. FINALLY, LET'S SEE IF IT FINISHED. OKAY. IT IS PUBLISHED ALREADY IN QA PRODUCTION IS ATTENDING APPROVAL. THEN I REFRESH. YOU CAN SEE IT RUNNING OUT OF THE API. I AM GOING TO RUSH THIS ONE AS WELL. NOW THE SENTIMENT OF ML. NET IS TRUE RUNNING IN AZURE. THIS IS FALSE. THAT'S IT. THANK YOU. [APPLAUSE] >> WE HAVE COVERED A LOT OF TEXT AND NUMERIC DATA TYPES YOU ARE ALREADY PRETTY FAMILIAR WITH HOW TO WORK WITH. HOPEFULLY YOU HAVE SEEN HOW WE CAN ADD INTELLIGENCE TO YOUR APPLICATIONS. NOW I AM GOING TO SHOW YOU HOW WE CAN USE OUR PREVIEW FEATURES OF ML. NET TO INCORPORATE PRETRAINED DEEP LEARNING PMODELS INTO YOUR APPLICATIONS TO BE ABLE TO WORK WITH OTHER DATA TYPES, LIKE IMAGES AND SPEECH, AUDIO AND MORE. WE CAN CURRENTLY SUPPORT TAKING IN PRETRAINED TENSER FLOW MODELS AND ONYX MODELS. ONYX FOR THOSE OF YOU THAT ARE NOT AWARE IS AN OPEN SOURCE INITIATIVE THAT WE PARTNERED WITH FOLKS LIKE AMAZON, INVIDIA, FACEBOOK AND MANY HARDWARE PROVIDERS LIKE INVIDEA INTEL AND OTHERS TO BE CROSS PLATFORM INDUSTRY STANDARD OF TRAIN MACHINE LEARNING MODELS. ANY MACHINE LEARNING MODELS OR PYTORCH, ET CETERA, BE CONVERTED TO ONYX. I AM GOING TO SHOW YOU HOW TO TAKE PRETRAINED TENSER FLOW MODELS AND ON MIX MODELS INTO YOUR APPLICATIONS. MANY OF YOU HAVE PROBABLY SEEN EXAMPLES OF USING DEEP LEARNING BEFORE WHERE WE USE A CLASSIC PRETRAINED MODEL THAT'S AVAILABLE ON THE WEB TO SIMPLY DOWNLOAD. THIS ONE IS CALLED YOLO YOU ONLY LOOK ONCE. IT IS AN OBJECT DETECTION EXAMPLE. I AM SHOWING HERE IN ASP. NET APP AND SELECT DIFFERENT IMAGES AND IT IDENTIFIES AS I AM BOUNDING BOXES IDENTIFY WHAT'S IN THE BOUNDING BOXES PROBABILITY AND CONFIDENCE LEVEL THAT YES THIS REALLY IS A SHEEP FOR EXAMPLE. NOW I AM GOING TO SHOW YOU HOW TO ACTUALLY BUILD THIS HERE IN VISUAL STUDIO. NOW JUST LIKE ANKIT AND CESAR SHOWED YOU BEFORE WE WILL START WITH AN ML CONTEXT. EVEN THOUGH WE HAVE PRETANED MACHINE LEARNING MODELS IN THIS CASE THE ONYX YOLO MODEL, YOU ARE OFTEN TIMES GOING TO DO PREPROCESSING OR TRANSFORMATION OF YOUR DATA BEFORE FEEDING IT THROUGH THAT MODEL. SO ML. NET NOT ONLY HAS TRAINING CUSTOM MACHINE LEARNING MODELS BUT THE REQUIRED PREPROCESSING STEPS NECESSARY TO USE THESE DEEP LEARNING MODELS. YOU CAN SEE FIRST LOAD THE IMAGE LIKE WE LOADED TEXT BEFORE, AND TO FIT THE MODEL INTO -- FIT THE DATA INTO WHAT THE EXPECTED NODE IS OF THE DEEP LEARNING GRAPH WE ARE GOING TO RESIZE THE IMAGE EXTRACT THE PIXELS AND APPLY THE ONYX MODEL AN AFTERWARDS WE USE THE SAME PREDICTION MODEL WE SAW BEFORE WHICH SIMPLY TAKES IN AN IMAGE DATA AND RETURNS RESULT. THIS PREDICTION ENGINE FULLY ABSTRACT ALL OF THE OTHER PREPROCESSING STEPS. YOU CAN THEN SHARE THIS PUBLISH THIS AS A NEW GIT PACKAGE AND WHATNOT YOU CAN USE IT IN MANY OTHER TYPES OF APPS LIKE AZURE FUNCTIONS ET CETERA. I WILL SHOW YOU ALSO HOW TO TRAIN YOUR OWN CUSTOM VISION MODEL. I WILL TOGGLE BACK OVER. I CLOSED THIS. LET'S LOAD THIS GUY BACK UP. SO I AM GOING TO SHOW YOU HOW TO DO SOMETHING CALLED CREATING A MODEL ON SAMPLE WHICH USES A PRETRAINED TENSER FLOW MODEL AS A STEP IN THE PIPELINE PROCESS FOR ITEMS YOU WOULD FIND IN A GROCERY STORE. THE TENSORFLOW MODEL WAS IN EVERYDAY ITEMS LIKE BIKES AND CARS AND PEOPLE AND STUFF LIKE THAT. WE USED A TECHNIQUE CALLED TRANSFER LEARNING. IT TAKES ALL OF THE KNOWLEDGE THAT YOU HAVE LEARNED IN ONE CONTEXT WHERE SOMEBODY ELSE HAS TRAINED A MODEL FOR MAYBE HOURS OR DAYS ON LOTS AND LOTS AND LOTS OF DATA. USING THAT IN YOUR ML. NET PIPELINE AND TRAIN ANOTHER MODEL THAT IS SPECIFIC TO YOUR TASK. IN THIS CASE I HAVE USED A BUNCH OF DIFFERENT DATA FROM THE GROCERY STORE TO TRAIN A MODEL THAT SOMETIMES IS PRETTY DARNED GOOD AND SOMETIMES LIKE THIS ONE, WELL, I DON'T KNOW IF IT IS SODA OR NOT, BECAUSE SOME OF THE CANS OF SODA SOME OF THE TYPES OF SODA LOOK A LOT LIKE BOTTLES OF JUICE SO THE MODEL MIGHT GET A LITTLE CONFUSED BUT IT IS PRETTY DARN ACCURATE. THIS ONE IS DEFINITELY JUICE. THIS ONE IS COFFEE, CAKE, ET CETERA. TO TRAIN THIS MACHINE LEARNING MODEL, WHAT WE WILL DO HERE IS WE WILL LOAD FROM AN ENUMERABLE. JUST LIKE WHEN WE LOADED FROM SQL SERVER BEFORE YOU CAN CONNECT TO PRETTY MUCH ANY KIND OF DATA SOURCE YOU WOULD LIKE TO USING THE LOAD FROM ENUMERABLE. IN THIS CASE I AM GOING TO ENUMERATE OVER ALL OF THE IMAGES THAT WE HAVE IN THESE DIFFERENT FOLDERS. SO YOU CAN SEE I HAVE ONE FOLDER PER CLASS, SO CAKE, FLOUR, CANDY, CEREAL, ET CETERA, ET CETERA. I HAVE A BUNCH OF TRAINING IMAGES IN HERE. NOW IT IS THE END OF THE DAY I HAVE BEEN HERE A WHILE I AM THINKING ABOUT COFFEE. THIS IS THE INPUT FOR TRAINING OUR NEW MODEL. IT IS GOING TO LOAD THESE UP. IT IS GOING TO ITERATE OVER THESE. IT IS AGAIN GOING TO LOAD THE IMAGES, RESIZE THEM, EXTRACT THE PIXELS AND HERE IS WHERE WE WILL LOAD THE TENSEORFLOW MODEL. THEN WE WILL TRAIN ADDITIONAL MULTI CLASS CLASSIFIER ON TOP OF THAT USING THAT OUTPUT OF THE TENSORFLOW MODEL AS THE INPUT TO THIS CLASSIFICATION LAYER. THEN FINALLY WE WILL OUTPUT THE ZIP FILE JUST LIKE YOU SAW BEFORE. WHEN WE RUN THIS, I HAVE LOADED IT AHEAD OF TIME JUST FOR SPEED PURPOSES. WE AGAIN ITERATE OVER ALL OF THE DIFFERENT CLASSES, PRINTOUT HOW MANY SAMPLES EACH ONE OF THESE CLASSES. IF THE MODEL IS NOT PREDICTING WELL ON ONE OF THOSE CLASSES, JUST LIKE CESAR SHOWED YOU EARLIER, USUALLY YOU WILL NEED TO DEBUG YOUR DATA, NOT DEBUG YOUR CODE. YOU MIGHT ADD ADDITIONAL PICTURES INTO ONE OF THOSE FOLDERS TO GET MORE SAMPLES OF ONE OF THOSE CLASSES. THEN WE WILL LOAD UP THE TENSORFLOW MODEL, TRAIN THE CLASSIFIER AND THEN OUTPUT THE ZIP FILE. JUST LIKE ANKIT SHOWED YOU EARLIER THEY PRINTOUT THE CLASSIFICATION METRICS, THE QUALITY SCORE IF YOU WILL EASILY AND THEN RUN THROUGH SAMPLES TO GET THE SCORE IN THE COMMAND LINE HERE OR INSIDE END UNIT TEST HARNESS THAT YOU SAW BEFORE AS WELL. SO WE ITERATE OVER A BUNCH OF THESE IMAGES IN THE TEST FOLDER INSTEAD OF THE TRAIN FOLDER. YOU WANT TO MAKE SURE THE TESTING OF IMAGES WERE MODELLED IN THE CUR TRAINING OTHERWISE YOU MIGHT FIT THE MODEL WHICH WOULD BE BAD. I CAN PUBLISH THIS USING AZURE DevOps AND PROCEED DOWN THAT PATH. I WANT TO GO INTO A LITTLE BIT OF DETAIL COVERING ONE OTHER TOPIC. MODEL EXPLAIN ABILITY. INTERPRETING THESE MACHINE LEARNING MODELS CAN BE PRETTY TRICKY. A LOT OF YOU ARE PRETTY DARNED FAMILIAR WITH HOW TO TEST AN APPLICATION. TESTING A MACHINE LEARNING MODEL CAN BE TRICKY. EXPLAINING HOW IT WORKS TO SOMEBODY ELSE IN A WAY YOU SOUND COMPETENT IS EVEN TRICKIER. THERE IS PRETTY STANDARD TECHNIQUES IN THE INDUSTRY THAT ENABLE YOU TO DO THIS SO YOU CAN EXPLAIN AND DEBUG YOUR MODELS, BUT MORE IMPORTANTLY MANY OF YOU I KNOW WORK IN INDUSTRIES WHERE THERE'S REGULATORY REQUIREMENTS WHETHER IT IS FINANCIAL OR HEALTH CARE OR WHATNOT. OFTEN TIMES THEY WON'T LET YOU SHIP SOMETHING TO PRODUCTION UNLESS YOU CAN PRINTOUT SOMETHING EXPLAINING HOW IT WORKS AND WHY YOU BELIEVE IT IS GOOD. ADDITIONALLY HERE IN MICROSOFT WE SPEND A LOT OF TIME AND ENERGY MAKING SURE THE DATA SETS WE USE WITH OUR MACHINE LEARNING MODELS ARE NOT FAIR AND NOT BIASED SO YOU CAN USE TOOLS LIKE THIS TO EXPLORE THE DATA AND UNDERSTAND WHY THE MODEL IS PREDICTING THE WAY THAT IT DOES. YOU CAN DETERMINE WHETHER YOU ARE USING CHARACTERISTICS OF THE DATA THAT IT SHOULDN'T WHETHER IT IS AGE, GENDER OR MANY THINGS THAT THE MODEL PROBABLY SHOULDN'T TAKE INTO ACCOUNT AT INAPPROPRIATE TIMES. BY RUNNING THROUGH EITHER BOTH KIND OF A GLOBAL SET OF DATA OR ON A SPECIFIC PREDICTION THERE IS DIFFERENT TECHNIQUES FOR ML. NET TO ENABLE WHAT'S HAPPENING. HERE TO SEE AN EXAMPLE HEALTH CARE IF YOU PREDICT QUALITY SCORE FOR HOW HEALTHY AN INDIVIDUAL IS, YOU ARE 82 PERCENT HEALTHY. THEY ARE PROBABLY GOING TO LOOK AT ME LIKE BUDDY, WHAT'S WRONG? YOU WOULD WANT TO RETURN A KIND OF EXPLANATION ALSO. HERE ARE THE CHARACTERISTICS YOU COULD IMPROVE. THIS IS WHY IT PREDICTED THAT YOU ARE NOT 100 PERCENT HEALTHY. RIGHT? WHEN DEBUGGING YOUR MODEL, YOU MIGHT LOOK AT ALL OF THE DATA IN THAT DATA SET TO UNDERSTAND THE DISTRIBUTION OF THE FEATURES AND THEIR RELATIVE IMPORTANCE IN TRAINING THAT MODEL. I AM GOING TO SHOW YOU EXACTLY HOW YOU CAN DO THAT WITH ML. NET. SO I HAVE BUILT AN EXTREMELY FANCY WIN FORM APPLICATION JUST FOR FUN. I HADN'T DONE THAT IN A WHILE. BUT YOU CAN SEE USING THE NEW YORK CITY TAXI FARE DATA SET. IT IS A VERY SIMPLE DATA SET TO UNDERSTAND. MOST FOLKS HAVE TAKEN A TAXI TRIP BEFORE. THERE IS DATA IN THIS DATA SET THAT IS LIKE THE LENGTH OF THE TRIP LIKE DISTANCE AS WELL AS TIME AND HOW PEOPLE PAID WHETHER IT IS CASH OR CREDIT CARD OR OTHER FEATURES AS WELL. ON A PER TRIP BASIS WE CAN LOAD UP THE TEST DATA SET ANDITIER RATE OVER THE DIFFERENT FEATURES IN THIS PARTICULAR MODEL TO UNDERSTAND WHY IT PREDICTED THE FARE IS GOING TO BE $10. THE TRIP DISTANCE WAS SUPER IMPORTANT IN THIS ONE RELATIVE TO THE OTHER ONE. FOR SOME OF THESE THE DIFFERENT FEATURES LIKE TRIP TIME AND TRIP DIFFERENCE ARE RELATIVELY BOTH IMPORTANT. SOMETIMES YOU WOULD SAY WELL ON THIS PARTICULAR TRIP, THE TRIP TIME WAS A LITTLE BIT MORE THAN THE TRIP DISTANCE. I CAN ONLY IMAGINE SOMEBODY HAD THE CAB DRIVER WAIT A WHILE TO GO AND GRAB THE BAGS AND SOMETHING LIKE THAT. THEY CHARGED A WAITING FEE. DIDN'T GO AS WELL AS FAR MAYBE IT WAS TRAFFIC. I DON'T KNOW. IN EACH INDIVIDUAL PREDICTION YOU CAN GET AN UNDERSTANDING OF WHY IT PERFORMED THE WAY THAT IT DID. TO DO THAT WE MENTION THE DATA SET HERE AND JUST LIKE WE HAVE DONE BEFORE WE WILL LOAD UP THE DATA. WE WILL DO SOME TRANSFORMATION LIKE ONE HOT ENCODING AND NORMALIZE THE DATA A BIT THEN WHEN WE TRAIN THE MODEL WE CAN ADD ONE FINAL THING HERE TO CALCULATE THE FUTURE CONTRIBUTION. CALCULATING THE FUTURE CONTRIBUTION IS WHAT WILL RETURN THE EXPLANATION OF WHY THE MODEL PERFORMED THE WAY THAT IT DID. ADDITIONALLY WHEN WE TRAIN THIS MODEL IT NOT ONLY WILL INCLUDE THAT PARTICULAR STEP, BUT WE CAN ALSO RUN THROUGH -- WE CAN LOAD UP THE TEST DATA SET. WE CAN SEE HOW THIS IS TRANSFORMING THE DATA. I TALKED ABOUT ONE HOT ENCODING AND NORMALIZATION. THIS IS WHAT THE COMPUTER ACTUALLY SEES THERE, THESE INDIVIDUAL 1'S AND 0'S WHICH IS A LOT OF FUN. WE CAN PLOT A CHART THAT SHOWS THE KIND OF RELATIONSHIP BETWEEN HOW THE ML. NET PREDICTED VERSES HOW WELL IT ACTUALLY PERFORMED. MY CHART IS NOT COMING UP FOR SOME REASON. THAT'S FINE. YOU HAVE SEEN PLENTY OF CHARTS ALREADY. HOPEFULLY THAT GIVES YOU AN UNDERSTANDING OF HOW MODEL EXPLAIN ABILITY CAN HELP YOU STEP THROUGH DEBUG YOUR CODE, YOUR MODEL, AND UNDERSTAND YOUR DATA SO THAT YOU CAN EXPLAIN TO OTHERS WHY THEY SHOULD TRUST IT. WHERE DID WE GO IN OUR POWERPOINT SLIDE HERE. ALL RIGHT. BEAUTIFUL. SO I WANT TO THANK YOU ALL. THE JOURNEY FOR ML. NET FROM 0. 1 LAST YEAR AT BUILD TO 1. 0 TODAY HAS BEEN PRETTY FANTASTIC. THERE HAVE BEEN A WHOLE BUNCH OF DOWN LOADS, A WHOLE BUMP OF COMMITS FROM A WHOLE BUNCH OF PEOPLE IN THE COMMUNITY. IT HAS BEEN REALLY FANTASTIC TO WORK WITH YOU ALL ON CREATING A MACHINE LEARNING MODEL FRAMEWORK FOR . NET DEVELOPERS. THERE ARE A NUMBER OF CUSTOMERS USING IT IN PRODUCTION TODAY GETTING GREAT RESULTS. MADE IT SUPER EASY TO INCORPORATE MACHINE LEARNING INTO THEIR . NET APPLICATIONS IN AZURE LIKE ANY . NET APPLICATION. AUTOMATED MACHINE LEARNING LEARNING CAPABILITIES HOPEFULLY ARE A GREAT WAY TO GET YOU STARTED LIKE THEY HAVE SOME OF OUR OTHER CUSTOMERS. I KNOW LEARNING THIS MACHINE LEARNING STUFF CAN BE PRETTY TRICKY. BUT THE TOOLING SHOULD MHELP MAKE IT EASIER. WE ARE GOING TO CONTINUE TO SHIP REGULAR RELEASES. GOING TO CONTINUE TO IMPROVE OUR CAPABILITIES WE JUST TALKED ABOUT SOME OF THEM LIKE THE MODEL BUILDER AND AUTOMATED MACHINE LEARNING STUFF IS IN PREVIEW RIGHT NOW. WE ARE GOING TO IMPROVE IT AND GET IT TO GA. WE WANT YOUR FEEDBACK AS QUICKLY AS POSSIBLE. GO OUT AND TRY IT TODAY. WE WILL MAKE IT EASIER TO TRAIN IT OUT ON AZURE AND MAKE SUPPORT FOR THE NEW TYPE OF TASKS IN MACHINE LEARNING. WE WANT TO HEAR YOUR SCENARIOS TO FIGURE OUT EXACTLY WHAT WE SHOULD INCLUDE IN THE NEXT SET OF RELEASES. GET STARTED TODAY GO TO . NET/MR. GO OUT TO THE DOCKS WE WILL HELP YOU FIGURE OUT HOW TO ACHIEVE YOUR GOALS WITH MACHINE LEARNING. HAPPY CODING. REALLY APPRECIATE YA'LL STAYING HERE SO LATE IN THE DAY. I WANT TO THANK ANKIT AND CESAR FOR THEIR TIME AS WELL. IT HAS BEEN FANTASTIC WORKING WITH YOU ALL AND LOOKING FORWARD TO CONTINUING. [APPLAUSE]
