Idea Hour: Enhancing Digital Accessibility with Cephable’s Adaptive Voice Controls

[Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] hey everyone how's it going uh welcome to another idea hour with seable I'm Alex from the seable team and in the next sort of hour hourish depending on you know how things go we're going to be talking about basically all things voice controls uh with seable so things like using adaptive voice controls different settings for voice controls how to set up profiles for using voice controls how to dictate what you're saying to type what you're saying as you say it and the different ways you can actually do that too and we're going to be using voice controls in a whole bunch of different ways so we're going to do some uh things like navigation uh using it across different apps we're going to be filling out some forms uh we're going to be uh playing some games in the browser we're going to be playing some other games I'm thinking of playing Minecraft even have it loaded up down here ready to go but we can try all sorts of different stuff um and as a reminder for these idea hours please join in chat we want to know uh what you're up to and how how it's going and by the way hi Anthony hi Toby uh and Julia and other folks in the chat too um bring your questions what can I help answer uh for you about using voice controls to seable and if chat doesn't work well for you I'm also in the seable Discord in the lounge voice Channel too where you can jump in on voice and ask questions uh I will note that I'm deafened over there intentionally using the deafened setting so that I'll go switch over when I hear someone join um but just know that also your audio is going to be on the whole stream so other folks that are uh listening in will be able to hear you as well but if you want to just be able to talk through something that's always an option as well but bring your questions bring your ideas uh this should be a pretty fun one so well let's uh let's jump in and get started I've got your chat over there I see you all hello hello uh I've got a whole bunch of different apps pre-loaded so we can do stuff uh and of course I've got seable so let me switch over to this guy so now you can actually see seable and I wanted to sort of start from from the very basics of of everything voice controls and how it all works so we're going to start just like navigating around the seable app um so if you want to follow along on seable on your machine feel free to do so but then we're going to jump into tons of fun different use cases I also want to note I'm going to be building a lot of the control profiles from scratch so I can show you how to do all that as well but I am planning on sharing those profiles in the Discord after the Stream So if you're watching and you're like wait how did Alex do that thing like I wanted to play Wordle the way that Alex did or I wanted to use that navigation stuff you can either build it alongside while I'm going through some of these things or you can shortcut right after the idea hour and get everything ready to roll right out of the box by just using the share links so without further Ado let's talk about voice controls so seable does voice in a few different interesting ways um first and foremost it's going to use your microphone that you're using on your computer uh so for example right now I have a microphone in front of me that's this one hello hello uh that I have just placed in front of me and because that's the microphone that I have set up for Windows for my default that's what I would be using for voice controls too um and so if I want to use voice controls I need to set up at least one voice control in a control profile and then there's a gigantic start microphone button here or if you're especially a keyboard heavy user it is the first uh tab when you're on seable so like when you load seable it's like tab that's your microphone uh so either one of these buttons do the same thing they pull up your microphone and start processing your voice controls I also want to note a couple things about how we do voice controls uh and actually I'll start the microphone so you can see some of it live here as well so when you click Start Voice start microphone you'll see the microphone start to process your speech uh in my case I have it set to down here and it'll show the speech as you're using it so you can see it processing what I'm saying you can see it's pretty accurate it's showing the text of what I'm saying and I can also do things like like pause voice controls so I don't have anything firing if I don't need to see all the the rumbling and bumbling of the text as I'm saying it uh then I can pause that too also hi everyone hey Alan how's it going so with the microphone running seable is actually doing all of that speech recognition to understand what you're saying entirely offline so first and foremost although you're going to use SEO and you're going to be connected to the internet and you're turning on your microphone and you might have your microphone running all day don't worry it's never leaving your own computer that you're running it on it's never being streamed to anywhere not even to seph's platform at all uh it's never being recorded anywhere nothing like that the audio that you're using is used to process your speech and then it's gone never to be seen Again by anyone or anything including when you're using it in other apps those other apps also cannot get your audio or hear your audio in the case of using it on the desktop like this um so know that you are you are safe and secure and private when you're using voice controls but beyond on the privacy and security of all that it also means that you can use your voice controls even if your computer is not connected to the internet maybe you have like a really spotty internet connection or maybe you're traveling somewhere and you need to use your voice controls for an app and you just don't have internet um and and that if that's the case to totally fine you can you can just use stable voice controls click Start microphone and you're entirely offline uh and good to go there's a couple things where you do need an internet connection the things like if you want want to use voice controls from your phone to your computer then both of them need to be connected to the internet so they know how to talk to each other but more on the sort of mobile app side of things in a little bit let's start by diving a little bit further into voice controls in a control profile and also the different voice settings so what if I go to create a new control profile I can of course choose to use an existing template that seable has for like different games or maybe just write down here to writing and editing because that's a common thing that we use voice for any of these existing profiles that have voice controls in them you will see denoted with the label that says it as voice controls I as of right now I'm pretty sure every single uh pre-built template for a control profile that we have has some level of voice control so not really something to worry about but if you ever need to double check that you can let's go ahead and jump into one of these I'm going to use the uh keyboard navigation Basics so if I go click preview here should load up my profile and we can see all of the different controls within it so for example we have things like enter which has a bunch of different inputs it has two different voice controls I can say enter or I can say select so for any different control that you're creating you can actually have multiple voice control set up for it um you can also use a voice control along with another input so this one also uses open your mouth if you're using the camera controls and also has a button in the mobile app that you could tap and in this case it's going to hit the enter key so I can say enter or I can say select which is pretty cool I can also build on this and add other inputs as well so let's go ahead and look at what adding a as I already have keyboard navigation Basics here a new voice control from scratch looks like so this keyboard navigation one doesn't do anything Mouse it's all Tab and shift tab you can see all the different inputs it's got some Arrow key stuff and we're going to use this in a little bit to do some types of navigation with voice only um what I can also do is add a new control so these ones are already existing because they're part of the template but let's say I wanted to add one for click because maybe I need to click something so I can go to add a control you see the very top thing here to add as a voice control it is the most common input used across seable hands down almost everybody at least tries voice controls or has at least one voice control that they use even folks with very limited speech uh can use at least a single voice control I'll talk more about that when we get into microphone settings as well but to add a voice control I don't have to do any sort of training or recording in order to figure out what I want to say I can just type the word that I want to use but what I can also do is if I'm a voice controls user and I can't type on a keyboard you can also use seph's voice controls to dictate in what you want to type out I'm not going to do that now just to you know not over complicate things but wanted to call it out anything where you're typing you can use seable voice controls to dictate into that text field you would just say type and the word you want to type so let's say for for example for click I wanted to use click I'm going to type it in on a physical keyboard here you could use a virtual keyboard uh you could also say type click with seable voice controls or turn on dictation mode with this button again we'll get into that a little bit later so now with my voice control here where I want to say click I'm going to choose what happens next and so in creating a control seable does all sorts of different things as an output it can press keys it can press buttons it can release keys and buttons it can toggle them meaning if it's already being pressed it will release it if it isn't being pressed it will pull it will push it down you can type phrases out you can move the mouse a certain direction you could use contextual TalkBack to have s will speak for you you can also stop all outputs for example saying stop in order to stop things from happening or you could set up multiple steps and create all sorts of complicated macros in this case all I want to do is a simple click so I'm going to choose again the most common option that we see which is pressing a key or button and then I'm going to select in this drop down that I want to press the left click so I want to tap the left click button and I can hit finish so now with that we see a new control in our list that has say click it also still has the other commands and everything else now I mentioned before you can add multiple different voice commands multiple different voice inputs to a single control um you can do that by going into a given control like this one for a say click and tap the edit button and then you can add another input so I can click add an input I can select voice control and maybe I want to say something like tap as another input to say so now I could say click or I could say tap both of those are going to do the exact same thing they're going to click my mouse wherever it currently is it's pretty cool so I'm going to save this I'm going to go back to the home screen with these changes and just talk about one more thing before we jump into actually using some of these different controls one of the important things that we do differently with voice controls at stable is we have what we call adaptive voice controls meaning that seable is going to try to adapt its speech to your speech to be as accurate as it can be and we give you a few different options to be able to control that the way that it does this is it looks at the different commands that are in your current profile that you have for your controls and tries to understand where it's being inaccurate or where it thinks it might be close to what you're trying to say let's say for example I have a hard time pronouncing the word click it's a pretty hard word to say actually two hard K sounds plus an l sound uh in the middle doing the K to l sound is also pretty challenging in terms like tongue movement just objectively a hard word to say for how short it is um but if I'm saying click over and over again and it's not recognizing it as click eventually it does start to adapt and say well we're getting some it's looks like what I'm understanding is close to click and you're saying it over and over again so maybe I'll just do the click and if you don't change the way that you're talking then I can start to assume that you were trying to say click and I can adapt how I'm recognizing your speech to that and so this will happen as you continue to change profiles and add new controls it'll continue to try to adapt to you over time so the more that you use it the better it gets now some of the questions I get about this all the time is will it make the speech recognition more accurate for me when trying to dictate anything in that case the answer is unfortunately no right now so if you're using full dictation and you want to say all the things and you want it to be super accurate it's really only adjusting the commands for controls and not the commands for dictation right now that's something that we're always looking to improve in terms of just our overall accuracy with full dictation but just note that the the Adaptive side of it really mostly applies to using it with inputs as of today right that could change in the near future um the other thing we get asked all the time is is seable storing all of my audio then if it's doing this retraining and again the answer is no we don't store any of that stuff we don't even store the words that it's being processed after you say um so you don't really have to sort of worry about those things see a question in chat from Anthony can you chain commands meaning if you say click click without pausing it runs the click action twice or you say up and right and it runs up and right action simultaneously oh my gosh great question okay the answer is yes to the first part and a caveat to the second part when you're using voice controls um and it starts to recognize your speech it will start the commands based off the first thing that it hears but it won't just pick up commands in in the middle of your speech in order to not have false positives now this is something we're looking to possibly make an option for so definitely want your feedback but for example if you said click click it would click twice in a row but if you said up and right it would actually only do the up command and not the right command because there's a break in that chain with the word and um but it if you said right up left down then it would do those completely in order and you can say those pretty quickly but what you'll notice is for example if I unpop my voice controls even though this control profile has all sorts of different inputs including things like uh voice commands for saying up and down you can see it's seeing that I'm saying up and down down here in the speech but it's not actually doing the thing but if I pause up stop then it starts to run it because it's actually waiting for me to be intentional about the voice control I want to use this is by Design with seable because we don't want basically you to have to worry about your speech when you're talking to others or if you're talking in apps or if you're playing games and you're talking in voice chat or if you're like me streaming right now and trying to use voice controls having to just like fire stuff all the time we did some testing in the earlier days of seph's voice controls and found that when we did turn that on meaning when we just fire voice controls whenever we hear them then we created a lot of weird behaviors that people didn't expect and they didn't really understand what was going on and so instead people found that when you paused and then said the voice command that was when they were sort of that pause right beforehand created the intention of wanting to use the voice command Toby your question uh is there a time frame for someone to stick with it you recommend to new users for retraining um I would just say honestly keep using it like it like the more you use it the better it gets and we're constantly working to improve how we do that adaptability uh the only caveat to that I would say is if you're working with someone or you are someone with very impaired spee very limited speech or limited vocab in your speech then I would actually recommend trying out an option which is my Shameless psych way to show you microphone settings and this option here called optimistic voice controls now with this setting turned on there's a whole big write up here but I'll I'll explain it I think in in possibly easier terms terms um is with optimistic voice controls on seable no longer tries to understand all of the words that exist and instead it only cares about the words that are in your current control profile so with this on right now I'm actually going to hit save here it only now will recognize commands that are in my current profile and it will optimistically hence the name predict what I'm trying to say based off of that to the closest word based on the phonetic sound of it meaning that for a profile like our Mouse Basics where I have like click and double click again click being a hard word to say you no longer have to annunciate the full word you can make just one you know k sound or just the L Sound and that will basically fire away uh pretty quickly so if you're working with someone or you are someone who's having a hard time with overall accuracy of speech because maybe you have limited speech or limited vocab uh or your speech impairment makes it hard for the microphone to understand you or even if you're just in a scenario where like you have a bad microphone and it's not working well for you I recommend at least trying out optimistic mode and if you don't like it uh you can basically just turn it off again and and get back to the sort of regular uh adapt the other thing that's cool to know about um the optimistic mode is again because it only cares about the controls that are in your profile it means in the case of Anthony your question if if you say up and right it actually will work back toback because it no longer even cares about that word and it's just up right like those are the words it knows and so it's going to fire those so just to show you an example of what this could mean when trying to basically use these different uh commands with optimistic mode I'll rurn on voice controls and I'll just make parts of the sound and you'll see it start to optimistically predict them uh way faster by the way seable voice controls are already real time and really quick but it's it's predicting it before I even finish the sound which is pretty cool but can also be a little bit hard to get used to but also if you're using it for like really fast inputs for things like gaming it can be really really useful because you can just say parts of the words and and talk really fast so let me resume voice controls and show you what I mean down stop so I didn't have to say stop I just said STA and it basically triggered it now I can also note that when you have optimistic mode turned on your dictation mode is not going to work or it will try to but it's only going to type the words that are in your current profile that's sort of the trade-off um Anthony your question can you go into a separate training mode where you read a paragraph and it improves the recognition no we intentionally went a different route with how we do the Adaptive training because what we found is that having to like go through the training process like other speech recognition where you have to say tons and tons of phrases and usually requires like a few hours of audio to get real accuracy is such a tedious process that being said it's not off the table for the future it's just how we look at adapting voice controls now and trying to find a balance of like what's easy to set up and use and uh what can we actually you know use in the future to to really work on accuracy so we didn't want to make it super hard to get into some level of adaptiveness we also didn't want you to have to worry about it right like in the end you shouldn't have to think about adapting your voice controls seable should do that thinking for you that's sort of our goal love the questions by the way definitely keep them coming um one other note about voice controls is you can use voice controls across devices meaning that if you know in this example I'm using uh voice controls on my desktop with my microphone that I'm using to talk to you all as well I can also use voice controls from my phone let me show you what that looks like real quick too so if I pull up the seable app and then I'm going to mirror my phone screen let me just pull up my little mirroring app called visor so what you're going to see over here on the right this is actually the seable mobile app on my phone um and you can see I have a Mac and a PC connected I'm on my PC right now and so if I go to my PC I can see that I can start camera and I can also start microphone here so when I tap start microphone here that will actually be using voice controls for my phone it's going to use my phone's microphone if I have you know uh Bluetooth headphones in it's going to use that microphone to then use controls on my computer so let's say for example I wanted to scroll down here in the seable app I'm going to go ahead and tap start microphone and you can see it start to listen to me down stop so this speech is running entirely on my phone I can leave I can turn on optimistic mode from there I can do all those same things um without actually having to use my microphone on my computer now getting ahead of a couple things here um and some other sort of tips and tricks that I have before we get into actually trying some of this stuff out one of the things that um we hear a lot is hey I have like a very specific microphone setup like how do I make that work with seable um and I wanted to note a couple things one seable uses whatever your quote default communication devices this is typically the same microphone that's used uh when you're on like a zoom call or a Microsoft teams call or something like that and that's what it defaults to use for your voice controls we do have on the road map the ability to change which microphone you're using but I also wanted to show you how you can do that in the meantime so as an example if I'm using my microphone here my microphone that I'm using actually has a physical mute button so for example if I press this mute button you all won't be able to hear me but I'll keep talking and seable won't be able to hear me either so I'm going to click I click to unmute my mic and now it hears me but when I do that muting I didn't pause here right I didn't click pause and so a lot of times people will sort of be confused if I'm hitting a physical mute button on my microphone and all of a sudden my voice control sto working well it's because your microphone's muted and seable can't hear you the same way that you all couldn't hear me uh when I was muting my my physical microphone the other thing you can do on both Windows and Mac although I'm on windows and I'll show you here to choose which microphone is being used is you can actually go to your Windows settings and then into your uh audio so if you go to your sound settings here you can probably make this a little bigger maybe maybe not uh you can go into let's see where is it if I scroll down I can go into volume mixer here and then let's say for example I wanted to switch which microphone is being used for seable I can go to this input device here and choose from the drop down an intentional one so I'm using the yeti Nano right now if I always wanted to use that or if I didn't and I wanted to switch to my headphones I could choose that from the dropdown um Mac has a similar setting it's a little bit harder to get to but just wanted to note like if you need to change the type of device you're using because maybe you have multiple microphones in your setup you can choose to do that here or of course you can again use the mobile app and offset all of it and just use your your mobile device which also has its own settings for for what microphone to use all sorts of fun stuff when you get in these more complex setups for you know what what microphones we want to use but amazing any other questions in chat Anthony I see adding the ability to correct words that it mishar you're not the only one that's brought that up we're we're taking all this sort of feedback and stuff and these ideas uh and helping them sort of inform the road map on on voice improvements that are coming in the next few months so I'll talk a little bit later about some of that uh the sble cleanup ignore background noise like a ventilator Sound Processing spe spe uh great question in general yes but also sometimes your microphone will do that anyway or your operating system will uh as a matter of fact I'll show you again in Windows settings like the the microphone that I'm using here this Yeti one it does a lot of that automatically um so if I go into sound again and I go into my where is it my devices if I click on my Yeti Nano here you'll see this device effect and so Logitech blue voice 2.0 effect this is something that's actually built into the Windows operating system with support for this microphone that's built by Logitech who built my microphone to do things like focus on The Voice Audio Only um so we do have like some very basic like noise cancellation but we also want to hear the scenarios where you're trying voice controls and you're in those scenarios and it's still happening another similar thing that we hear sometimes is folks that are using voice controls in a really loud environment like maybe you're in an office and you have 10 different people sitting around you who are talking on the phone all the time as part of their job well is their speech going to be picked up and if so how do we fix that so in that case Yes actually there is still some challenges we see with folks using voice controls where there's a lot of people talking and it's trying to identify who the primary user is but we are constantly working to uh you know try to improve those use case scenarios when it comes to using voice controls a bit more great questions though and great feedback and things that we need to continue to improve uh our efforts on when it comes to voice controls okay let's get into some use cases so we've already I've already shown you some sort of spoilers of some of these things because we've just been doing it in the seable app um but a lot of people when they think of voice controls just think of dictation or some other people think of like just keyboard shortcuts and stuff like that but there's some really cool things you can do with voice controls uh like navigating around and as a matter of fact our pre-built profile for keyboard navigation basics kind of has a lot of this covered so for example if I go to view this profile we have things like holding the up and down arrow keys and stop which you heard me doing already so instead of just having a voice control shortcut that like does a keyboard shortcut equivalent we're able to hold the down arrow key whenever I say down and also release the up arrow keys so you're not trying to do both in case up is being held until you say stop or until you say up and then when we stop we release all those arrow keys um but that can make it really easy to navigate what would otherwise be pretty challenging voice only apps like actually scrolling through a web page it's usually pretty hard uh also doing things like Tab and shift tab in order to go to the next thing or back to the previous thing so I can say next or tab or I can say back previous or shift tab on top of using these other inputs to tab navigate around plus enter and space so we're going to use a couple of these just out of the box pre-build template you don't have to configure any of this from scratch uh to navigate a couple web pages so I'm thinking we can pull up a new site we could do some scrolling we can do some searching all using voice only uh instead of having to sort of keyboard navigate around so let me go ahead and pull up one from my other monitor over to here so for example uh I'm over here on the BBC now and I want to start my microphone controls uh so that I can do things like search for articles pick one and navigate around so let's go ahead and do that I'm going to switch back to seable and click Start microphone now another thing you'll notice is that the mic microphone controls stay on top of my current window they didn't go behind the browser like the regular seable window did they'll stay on top as long as you want them to and you can move that window around and change the default position and everything else too but now that I have it set up let's do a couple things to navigate search scroll through an article tab tab back back back back back back back back back back back oh can you say start listening or stop listening that actually is going to come in our next update Anthony we have an whole new update coming for what we call Global controls that includes a whole bunch of things including opening an app by name so being able to say like open Chrome course let me get this ad out of the way uh being able to say like start voice controls stop voice controls um all those things are are going to be coming as part of like defaults and you'll still be able to change those to so even turning off and on dictation are things that are going to be coming up uh in the next major update basically tab tab enter is there a dark mode in the works for SE yes there's a dark mode in the new mobile app there is a dark mode on the road map for the desktop app I trust me it's one of my top personal priorities is I am a dark mode fan uh the mobile app dark mode looks beautiful we just have to bring some of that stuff to the desktop app too um okay now that we're in our search box now we don't want to like Tab and have to like you know use a keyboard to type these things in we can continue to use our voice only using that type keyword there's two different ways we can dictate stuff we can use that type keyword or there is this button down here looks like a little keyboard that we can click to turn on dictation mode when you click that it'll turn off voice control so it won't be able to say Tab and enter it will uh just always type what I'm saying until I turn it off but in this case because we're using a mix of voice controls and I want to dictate something into a search this is a good time to use that in dictation which means that I can do things like type octopus news enter so I'm able to do that like typing of what I'm saying in the moment without having to like go click something or say another voice command to switch to dictation mode and then switch it off I'm just like look I got to type something short because I'm still navigating around I just want to go and type that specific thing you just say type and whatever the thing is and you're off and running but just like other voice controls you want to pause to make sure it's not listening like it is right now and trying to type all the stuff you're saying uh you're going to want to basically pause and then say type you know in this case octopus new tab tab tab BBC is an interesting one because I they don't have the what you would normally see a lot of times is a skip to content uh button that pops up in the header uh when you're navigating around instead now I don't know what's active another one of those things around just general web accessibility that's a challenge but let's say I don't need to tap down to a specific thing I just want to scroll to be able to see what's available down stop up stop former Lifeboat becomes glamping Yellow Submarine pretty funny um so we can basically do these types of keyboard navigation we can tab we can shift tab we can type in the moment uh and we can scroll stop scrolling all those things you know just using voice only but reading information like the search results or even like the content of one of these uh articles which just scrolling as I'm reading is one thing um I also might need to do things like actually fill out a form and those can be a little bit complicated so let's talk about that type of use case where things are a little bit different I'm going to actually close voice controls bring over a Google form that we made to shamelessly talk about a seapod circle an imaginary book club although if we get enough interest maybe it'll be a real one uh where I need to do things like fill out my name and email and these options which I may have accidentally pre-selected um but maybe I want to change and things like that oh my gosh there's all sorts of spoilers no one looked at all my favorite things in my book suggestions when I was testing this out um yeah I also want to camp in the LSM ring uh how does it no to stop typing when you're doing that uh when you pause your speech after saying type the thing then it stops uh so if you just say like type octopus news pause then it it basically trickers it um you can if you're using the full dictation mode it will type whatever you're saying after that pause and then it'll restart typing as you continue talking so it doesn't in sort of these chunks instead of like every letter at a time that way it gets as accurate as possible so let's go ahead and fill out this form but there's there's a couple caveats here right like I need to be able to dictate my name pretty easy at least for me especially I have an easy name just Alex most speech recognition will probably be able to figure it out but there's going to be scenario is where I need to type things out letter by letter and this is where um time St noted for book descriptions I'm going to use the same thing okay I am passionate about the books I do like um but I need to type out my email which you know typing out an email by just speaking it can be a little tricky with just dictation so I want to type it you know one thing at a time uh so I'm going to show you how to do that too let's go make a new control profile and I'm going to create it from scratch so one of the things that I want to do is include voice commands for keys and mouse clicks I'm going to talk about this switch in a little bit uh then I'm going to go ahead and click create from scratch so I have a new profile with nothing in it but I also need to be able to tab navigate so the same thing that we had set up in the keyboard navigation uh I'm going to go ahead and basically set up that quickly from scratch so I can go in here and I want to have Tab and I want that to press the tab key uh and I also want to be able to press enter by saying enter and I want to press a key I want to press enter you see how quickly you can sort of get at at building out these different controls uh now I want to say back and I need that to hit two buttons so I need it to press the left shift which I can scroll down to uh left shift and then another key where it also has to hit tab so I'm hitting shift and tab at the same time now what's cool about the default voice commands is that I actually don't need two of these Tab and enter the reason that I don't need these is because if we go to the advanced profiles option and into keys and buttons when that switch is turned on every key on your keyboard and your mouse clicks come with the default voice command so you don't have to enter them this is really really useful for typing things out one letter at a time uh and it's just tapping them right so for uh clicking there's a default click for even if we scroll down to Tab and enter tab there's already default tab so I actually don't need to recreate the wheel if I'm going to use the defaults uh you can also add other inputs here so if you want to change your defaults from tab to like next or something like that you can do that um but what that means is over here in the advanced section for enter and tab I can delete these I don't need them the only one I need is the one that's doing a more complexing where I'm hitting two keys at once I'm going to save that as my back so let me rename this I already have an existing profile called new profile shocker idea hour uh typing and we'll click save and finish so now I can go and switch to that profile I can start my microphone and we can fill out the form type Alex Tab and now that I'm over here in the uh email section we're going to want to type this one letter at a time so I can do things like a l e x oh gosh I forget the voice control for this classic blunder so because I forget the voice control that's the default for the little app sign uh I can scroll down all the way to that little character and see what oh literally called at sign not just at at sign c e p H oops h a b l e e dot period c o o m there you go go typed it out one letter at a time that ought to do it uh now I can do things like tab tab tab space I prefer fiction and I also prefer to meet virtually because that's where we all are uh could we record where we would like our Mouse to go like the center of the screen and have a voice command for that um that is a great question so we are working on a whole bunch of new mouse controls also you can see it trying to pick stuff up I'm going to pause voice controls because it's not going to type um uh we what you can do right now for Mouse movement with a voice control is move relative to where your mouse currently is actually I'll just jump in real quick to show you how you do that in the controls let me Zoom this in a little bit again if I wanted to say like Mouse up for example as my voice input and I want to move the mouse you actually choose the number of pixels up down left or right or combinations of those like up and left or down and right that you want to move but it is relative to where your mouse currently is um we are looking at making a whole bunch of new updates to better Mouse controls in general uh including our next version of the Apple have much better Mouse movement with camera controls so think of like head Mouse controls uh so we we are making a bunch of changes basically to how we do Mouse stuff to make that a little bit easier but right now if you're a current stable User it's all relative to where your mouse is so if you wanted to move it up 20 pi pixels you could do that and in this case You' say mouse up Mouse up Mouse up Mouse up and you can repeat those if you need to um or if you're a camera controls user and you have the ability to move your head around then even a small amount I would recommend just checking out the uh head controls uh for Mouse movement down here okay back to our form uh I'm going to now type some things like my book suggestions type I'm a big fan of The Lord of the Rings tab type my favorite snacks are burritos not what I wanted backspace backspace backspace backspace backspace delete delete delete delete delete delete delete delete delete delete delete delete delete delete there's your uh answer to your original question Anthony delete delete delete delete delete delete delete delete delete delete so we can try that again uh of course if you're using a a profile that you're not just building the bare minimums I usually recommend starting with the writing and editing profile which has things like select all delete all built in uh so you don't have to do that like tedious delete every single character while you're typing type I really like burritos there we go that's much better um there's a couple other things while we're here talking about the I really like burritos use case I'll say um when it comes to typing and there's a difference between how uh windows currently handles it with seable and how Mac currently handles it with seable and for those who have seen both uh you might have noticed that there's a difference in the way that it actually types out so for example when I typed it here on Windows it was just like boom here's I really like burritos all of a sudden and on Mac it kind of types in a really fast one letter at a time um basically on Mac ma OS we're actually hitting every single key to type it out but on Windows we're actually pasting the full text all at once this is because typing virtually with the virtual input system on the Windows operating system can be really slow or unpredictable and so this is just more or less a work around that we have but that means that the thing that I just typed I really like burritos in this case is actually in my clipboard which means I can paste it over and over again on windows so now I really really like burritos um but that's actually not the case when you're using it on Mac I just want to sort of call out that difference uh can you use capital letters yes if you use punctuation in your dictation like say type I'm a big fan of Lord of the Rings exclamation point it will automatically sentence case what you're saying so worth noting there because I didn't hear it didn't assume that I wanted it to be sentence cased that's sort of a fun little caveat U but also in the um next version of the app we have a bunch of fixes around like autoc capitalization stuff too um that I don't have in this version I should have probably loaded up the current test version although that's a little bit spicy um to figure out but yes there there are ways to do capital letters you can also turn on caps lock with a voice command too uh type one letter at a time and then turn it off which pretty cool okay we were able to fill out a form we did it with a mix of typing things short stuff longer stuff deleting stuff one letter at a time uh we were tab navigating all those things done with voice controls and in a profile that literally only has one thing in it which is going back that's all all the power of the default keys and buttons um so sort of note there pretty fun little setup little hidden thing because it can also be a little bit dangerous if you don't know that you have all these default keys and buttons and you're like hey why did it press the a key when I said that the letter A I didn't set that up in a control that's what sort of uh turned on there one quick thing while I'm here before we get on to playing some games with seable is these dictation phrases now I've been saying type as the thing to start typing you can actually change that so for example I can add more phrases or I can delete type or if I don't want that ability with this current profile to have type something you can actually remove that and then you have no in moment dictation you would only use full dictation mode but let's say for example I wanted to say type or I'm playing a game and I want to say something like chat so I can say chat I like burritos or chat seable is a cute little octopus and that would act the same way as when I was saying type that's all in the advanced settings of profile files like an autocomplete or guessing the current SL next word onscreen keyboard Toby yes so we have explored some of those things but from our experience folks using voice controls for input usually like to just have it dictate whatever you're typing out uh as you're saying it um which is why we sort of went that route however we have been exploring some other options around like other types of keyboards for nonvoice users for using things like your face or virtual buttons to type certain things and have sort of int auto complete Toby's a great use case for that because they're using I as an input uh in that case or are you asking are you talking about Toby in chat in which case I'm talking about Toby the i gaze devices pretty funny either way uh let's let's jump on to our next demo I'm thinking we should play a game because I've been sitting here just talking to you all and answering questions but I want to play can you use head movement to wait am I hearing myself why am I hearing myself never mind okay um can you use head movement to initiate typing you can use head movement for keyboard navigation or for Mouse movement with the windows virtual keyboard we have a few people that do that so you can turn on head Mouse and then use head Mouse controls to type uh yeah Toby and Chad my bad uh well anyway I gavee fun fact for you too um so has anyone in chat played Wordle before and for those who do play Wordle if if you already did today's Wordle don't spoil it but maybe give us uh you know a couple hints here now for those who have never played whle basically the idea is we have to guess what the word is and it's a felter word and we have this vertical number of guesses we have six guesses to figure out what it is um uh editing and selecting the last word and deleting yes so we have just real quick a profile for that that is our writing and editing profile well here's the Mac version anyway but there's a version here for Windows which has all of those sorts of things me Zoom this in again um so it's got like saving writing up down selecting the word to the right selecting it to the left jumping across words all of these uh and like regular Arrow navigation all this stuff is built into this profile um so if you're going to use like heavier writing and editing things I'd recommend just starting from this profile template because you get all of that like word-based navigation stuff and as a matter of fact we can uh choose to do that for this profile um but let's go ahead and make a new Wordle profile so I'm going to create from scratch I'm going to again include those default keys and buttons because I don't want to have to add one for every letter so we have our defaults here and I also want to be able to clear something out so I need to be able to delete five keys after I say clear or something like that so let's add a new control I'm going to say clear as what I want to be able to say and I need it to press the delete key basically five times so I'm going to choose this going to say press the backspace button then I'm gonna add another step and press the backspace again and then add another step press the backspace again oops not that press the backspace you could also choose to hold the backspace and I think that would work fine too and then like release after a certain amount of time um but this way it's sort of pretty clear like I'm hitting backspace five times cool so I'm going to say this is idea hour Wordle that way I can share it with you all later so let's go ahead and start the microphone again and I'm going to need someone to tell me in chat what you think the the best starter word is um but I'm going to start with just one of my go-tos w oops w e a r why enter oh my gosh there's actually a w okay for those new to Wordle this means there is a w and an r in the word but that they're not in this position because they're kind of this yellow color so the W is not the first letter and the r is not the fourth letter um arise is a good one but see now cord because of the delay of this chat I already have used the W and now I feel like I need to start with the W uh stamp is a good one because we can start to sus out some of the other ones but now we need to see for other vowels I might do stomp uh toi because that's sort of like similar thing but different uh different starter or different vowel I mean s t o m p enter well there's the O crown oh Crown might be it c r o o w n enter sorry not Crown but there is a row in the middle oh Arrow could be well Arrow it couldn't be Arrow because we have the a covered already um but we have the r the O and the W in the right spot uh oh and there's a p so there's got to be a p somewhere maybe prowl by the way for Mac users you can also just type these out but this is another one of those things where because uh seable on Windows does copying and pasting on Windows we have to do it a letter at a time another sort of fun twist to how those things work all right let's try this one let me get closer to the mic so it can pick me up p r r o w l enter guys we're so good unbelievable okay now well you can't really play again because it's sort of like a one one time deal so uh let's play some Minecraft anyone opposed to playing some Minecraft Let's uh let's let's let's build a new Minecraft profile so just like how we've been building out these other profiles I'm going to add a new profile and we can choose the type of profile that might match uh match Minecraft the best which is probably this 3D roll playing game so if I go add this profile I'm going to call it idea hour Minecraft you can see it comes with a whole bunch of stuff for voice control so it has voice controls for some of the numbers it's got Sprint uh which is holding shift and W interact which is e although I think we might need to change this Crouch attack um but let's edit this because I also need to add mine uh as a voice command but maybe let's do mine should hold click and then saying stop should stop that way you can like mine through a whole bunch of stuff can you add repeat a step text field where you type five and it repeats that step five times including instead of duplicating the step manually uh great question uh I've been having this idea for a little bit the the the one said it's like a mix of that and also like do this other control as well where we've been worried about like some infinite Loops but yeah we don't have uh repeat Stu you can duplicate it thanks Cordia I see you in chat too um okay let's add a new control for mine and they call it a mine speaking of Lord of the Rings so I want to say mine and I want that to hold although we have a bug here ignore this uh that'll be fixed in the next version Anthony I know you also saw this before too you sent me screenshots hold for an amount of time and hold until release are inverted so really I want this to be held until it's released and now I can say mine to hold left click um but let's see what else do I want to do for holding down W for oh we already have go perfect uh yeah so basically we can run around and mine some stuff up so I think we're in a good spot to try some of these things out then we'll get into some more complicated macros I think too so we can do some really fun stuff so I'm going to launch Minecraft and hopefully it doesn't have to run some gigantic update and give me grief and if it does then we're just going to have to hang out and chat I guess I updated it right before this stream and it already has another update to run so oh Minecraft you sweet beautiful game do we have any Minecraft Ste theves in the chat can I get a for all my uh villagers in the chat yeah I know I I removed all my mods by the way because I didn't want it to be brutal there we go there's Minecraft single player new world with creative mode okay please load now here's some some fun things to know when you're trying to use seable voice controls in games oh boy my frame rate is brutal um when you're trying to use voice controls in games sometimes you have to change some of your game settings to make things work um I also do think I I it is installing shaders in the background incredible um main reason being that the uh voice controls like popup window needs to be visible on top in order for voice controls to run it need like the window has to exist on top other otherwise it can slow down and eventually turn off the operating system turns it off um the if we run into that scenario actually let's see if we run into it because then I can sort of explain it so I'm going to click Start microphone and then I'm going to click back yeah see I click Start microphone and my microphone controls are not here um this happens a lot with other forms of at for folks that have ever used like the virtual keyboard on top or Anthony I know you've run into this with overjoy in the past two where the game is full screen and does not allow any overlay windows on top like our voice controls window uh then you have to go into options and change the video settings from full screen to uh where is it should be like windowed or borderless windowed except I'm forgetting how to do it in Minecraft full screen on so we want to turn it off but then we can make it bigger so it's still full screen and there our voice controls are magically on top now another thing to note though is like yeah the voice controls are here and I can see them and everything's working yeah full screen games are annoying indeed is maybe I don't want to have the voice controls visible but I still want to use them you also can do that but it does still need to be in some games by the way some some games full screen Works totally fine other games it doesn't let's say for example you can see my speech running now on top and that's great but I can't see my uh current toolbar so let's talk about a couple ways we can change that real quick you can move the voice window so you can click and drag it and move it wherever right like I can move it right to the middle but now that's also and it's right in the way um or I can move it to a corner by dragging it and things like that or even if I have multiple monitors I can move it off to a different monitor um but I might also just one not be able to use Mouse to drag and drop stuff as easily and if that's the case you can actually go to your general app settings and choose the position for your microphone window so for example the default right now is bottom center I think in future versions it's top left let's say for example I wanted to move it to top Center I can do that I can click save and now it's all the the way up there or if I wanted to move it to the bottom left I can go to General app settings go to my microphone input and go to bottom left and then hit save and it goes down here uh so you can do that you can move it around either by dragging the window or by using some of those placement things in the general app settings but you can also hide the thing so it's not in the way to do that you can go to your system tray so this little guy down here and then you'll see a little seable icon if you're on Mac this is in your very top toolbar you'll see a little seable icon if you right click it you'll see this it's I know it's really small because it's in the system tray but there's this little show/hide voice controls and it hides it it's still running which is cool so we can test that it's still running if I go back to the game go go and I'm going it's running I am I am moving uh without touching the keyboard stop and I can let me pause bring the window back up in the same spot so I can go to this little system tray and show hide voice controls again and it's still here so um let me no not that one I wanted to switch to the new Minecraft not my old Minecraft one Minecraft it's probably all the way at the bottom there it is so now we can resume it it's it's going to be running in that same default spot bottom left because that remembers the the setting that I changed changed uh and then I can jump back into the game and we can go mine so remember we had mind set to hold click which means that I don't have to click anything I could mine straight down which you should never do in Minecraft but let's go ahead and do it so I'm going to go ahead and just keep mining down until I hit some rocks and then I can't actually do anything anyway oh yeah I'm in Creative now so I'm just in Forever even though I'm uh you know I'm not telling it to hold to do anything specifically stop because I said stop it is stop mining but there's a lot of other things in a game like Minecraft we can do it's not just moving and clicking we have to be able to build we have to be able to fly around we there's console terminal things that we want to be able to do and so for example one of the things you can do in the console is say uh time set night and it'll change the time in your game or you can say time set day and apologies I know this desktop audio is a little loud we can turn that music down um and now we can actually go and add a new control to our profile so let me turn off voice controls I'm going to edit our Minecraft profile and I want to add a new one a new voice control where I can say daytime for example or uh turn it today let's say so I'm going to add a new voice control and say turn it today and we can add a sequence of events where first I need to press the slash key because that's how you get to the commands window and then I want to type those voice the the commands there that we just had so instead of pressing every single key and button I can actually just type the phrase out so I can say time set day and finish now it's worth noting some of these types of more complicated typing and pasting things that can happen some games have a challenge with so be sure to try it in each of the different games make sure that typing works this also means that you can do things like type in chat uh when you're talking to people you can set up pre-built messages to type in chat with a simple voice command which can be pretty cool so if I hit save and finish and start microphone and also I see the question are all those profiles that show uh default or the ones you built personally uh this big list is a whole bunch of personal ones but the pre-built ones are all there if you just go to click Add profile there's a whole bunch of pre-built templates here different types of game and stuff so I'm customizing on top of that this Minecraft one for example is just the 3D role playing game template okay we click microphone I know we're running out of time here and I want to at least try to get this one thing to work turn it today turn today what did I even call it another classic voice control uh thing that I find myself doing Turn it today it's going to think is today click save and finish turn it today enter and it's set it's a daytime now I had to say that enter that's kind of a Miss on my part so we can go back and edit it so that we add another step where we don't just press the slash type the thing uh we're going to add another step where we just press the enter key just to really round it all out I'm also going to move it out of the way so you can see what's going on so if we do it again set time or time set night turn it today turn it today all of that stuff happens super fast so it hit the slash type the thing hit enter all like right back to back so another one of the sort of scenarios we're adding these macros you can really take some of these otherwise complicated things you have to do a lot of keyboard stuff or a lot of stuff and make it a whole lot easier uh so having voice controls to do like admin control things in Minecraft plus movement plus mining you can also do things like pre-build structures by using Mouse movement and clicking and inventory management more automatically um so we're coming up to the end of our time here in our idea hour of all things voice controls just to sort of summarize all the fun stuff we did we talked about how seable adapts voice controls to your speech overtime using the controls in your profile we talked about how you can turn on optimistic mode to limit the speech to only the vocab of those controls really useful for situations where you need really fast voice controls or if you working with or are an individual with uh impaired speech to be able to have any the different sounds you can make turn into controls we uh talked about dictation how to type using the type command or to turn on the dictation mode the big button here uh we talked about different ways to use voice control so scrolling down tab navigating around on web pages filling out forms by using a mix of typ things in one key at a time we talked about the default commands you can use in a profile uh we played wordl and got it in four tries which is pretty good uh and also showed how we can do things like play games like automatically moving by saying go uh typing things into an admin console holding click and Mining or maybe that would be attacking in another game using a bunch of pre-built profile templates and really everything in between so uh I hope you enjoyed this idea hour and appreciate everyone in chat uh Toby Anthony let's see who else has really been helping here that's not also a seable person uh ctm therapist amazing questions um please keep the questions coming in Discord if you have other questions about voice feedback ideas we constantly want to hear them we want voice to be the best that it can be we want your voice to be used for the best that it can be um and that we we can sort of build that together not just with us guessing um so appreciate all the time thanks for watching thanks for joining uh we'll see you in Discord for more follow-ups and questions and ideas and let us know how are you using voice controls what do you want to see out of voice controls next and uh we'll see you in the next month or month or so for our next idea hour we got tons of ideas but also we love to hear from you what is the next thing you want to have covered um you know we we talked about a lot of different things here so we want to hear your ideas what can we help sort of explain or show what things were you curious you know how to apply seable towards so that you know we can really sort of get the most out of it but thank you all so much for the time and I'll see you Elsewhere on the internet

Channel: Cephable

Views: 87

Keywords:

Id: EWBSKfvXZlY

Length: 70min 55sec (4255 seconds)

Published: Wed May 01 2024