Apple Watch : Scientific Sleep Test

until recently the apple watch was one of the few smartwatches that did not natively track your sleep stages but with the beta release of watchos 9 that has finally changed however are the sleep results you get from the apple watch actually any good well that's what we're gonna test in this video we will analyze 18 nights of sleep data from the apple watch and see how this compares to the sleep stages measured by an eeg device that can actually measure your brain waves and was specifically designed for sleep stage tracking we will also check how the performance of the apple watch compares to 37 other smartwatches i've tested hello everyone for those of you that are new to the channel my name is rob and i'm a postdoctoral scientist specializing in biological data analysis now when apple first announced their new sleep stage tracking functionality they did not provide much detail as to how it actually works but let's briefly summarize what they did tell us and what this means now the best way of measuring your sleep stages what scientists also refer to as the gold standard is called polysomnography this basically means measuring the electrical signals coming from the brain while you're sleeping and in addition also measuring the movements of your eyes and some muscles this way scientists can get an accurate picture of what sleep stages you went through i've done this close to 100 times myself but i have to say it's not the most comfortable way of sleeping so to make sleep tracking more comfortable and readily available apple trained a computer algorithm that takes in the heart rate and movement data from the apple watch and uses that to predict your sleep stages to create this algorithm they collected both polysomnography data and heart rate and movement data with the apple watch on what they at least say is the largest and most diverse study population for a wearable that algorithm then uses the sleep stages as measured with polysomnography and tries to predict the same sleep stages with just the heart rate and movement data from the apple watch now if apple has enough data and good machine learning tools they could have potentially created a very good sleep stage tracking algorithm making sleep tracking available to the general public so let's see if the apple watch is any good at sleep stage tracking so how did i actually test this well i evaluated the sleep stage tracking of the apple watch for 18 nights by comparing it to the dream 2 eeg headband a device that can actually measure my brain waves and was specifically designed for sleep stage tracking now as i mentioned before the best reference device would be a polysomnography device but this would be too cumbersome for me to use for 18 nights luckily in a scientific paper the dream 2 eg headband was shown to be pretty reliable at sleep stage tracking as i discussed in this video linked up here the eeg device should therefore give us an impression if the apple watch is good enough for sleep stage tracking or if it's really bad in this case for the testing i use the apple watch 7 but i think both older and newer apple watches will give more or less the same results since the heart rate and movement measurements are roughly equally reliable between generations as we saw in previous videos measuring heart rate and movement is especially easy during the night when you're generally relatively still with a low heart rate now i should mention that i'm using the beta version of watch os 9 but i suspect nothing will change with regards to the sleep stage tracking in the official release since the algorithm was already trained so let's see how the sleep tracking of the apple watch compares to an eeg device here i show an overview of the sleep test results on top are the sleep stages as recorded by the eeg device and on the left are the sleep stages as recorded by the apple watch i wear both the eeg device and the apple watch to bed at the same time for 18 nights and we will see how close the predictions of the apple watch are to those of the eeg device now each column here sums to 100 meaning that we can see what percentage of each of the sleep stages according to the dream 2 was predicted as a sleep stage by the apple watch now if they perfectly agree all values on the diagonal should be 100 percent first of all we see that about 89 of what was deep sleep according to the eeg device was also deep sleep according to the apple watch now this is actually very good as most watches show much poorer agreement with the eeg device if deep sleep according to the eeg device was predicted differently by the apple watch it was predicted as light sleep at about 11 percent and we can see that even better based on the individual knights on top here we have the sleep stages according to the dream 2 eeg headband with the clock time along the horizontal axis and the sleep stages on the vertical axis on the bottom we have a similar plot but now for the apple watch i've highlighted all the eeg recorded deep sleep in purple here for this knight we can indeed see that basically all of the deep sleep recorded by the eeg device was also recorded as deep sleep by the apple watch so this is looking really good the apple watch is predicted a tiny bit of extra deep sleep right here and this second night shows much more of the same results the apple watch predicted much of the same deep sleep segments as the eeg headband but just detected a little bit extra deep sleep as well still this is looking really good i also wanted to look at a night where i had poor sleep quality to see how the apple watch deals with that when i don't have a normal sleep pattern and that is displayed right here as you can see i was awake for about 2 hours in the middle of the night interestingly though the apple watch correctly detected this and was basically also spot on when it comes to the deep sleeper detected so it's good to see that it also seems to deal well or at least good enough with disturbed sleep the only thing that it does generally tend to do is detect a bit more deep sleep than the eeg device as you can also see right here the same deep sleep was detected more or less but some extra deep sleep was also detected though not a lot of it still though the apple watch shows some of the best deep sleep tracking agreement i've seen for any device so far and the same is true for light sleep detection about 84 percent of what the eeg device detected as light sleep was also detected as light sleep by the apple watch this is again quite good if it was predicted differently it was most often predicted as being deep sleep but sometimes also as rem sleep or awake time ram sleep agreement is also quite good at about 67 percent this means that about two-thirds of what the eeg device predicted as being ram sleep was also predicted as rem sleep by the apple watch now before showing you the rest of the results i want to briefly explain something a 67 rem sleep agreement might not sound like much but it's actually pretty good rem sleep has historically in my testing being one of the most difficult sleep stages for a smartwatch to track the easiest way of measuring rem sleep is by measuring the movement of the eyes rem stands for rapid eye movement meaning the eyes move quite quickly during the sleep stage mostly side to side this important characteristic of ram sleep is quite easy to measure for an eeg device or polysomnography device but impossible to directly measure with a smartwatch therefore most smartwatches actually struggle to accurately detect ram sleep using just movement and heart rate but the algorithm of the apple watch seems to do this quite well now getting back to the results if the apple watch did predict something different than the eeg device when it comes to ram sleep it was mostly predicted as light sleep and we can see that clearly based on the individual knights as well now this is a similar plot to before but now with the ram sleep as measured by the eeg device marked in red as you can see there's a good agreement between the apple watch and the eeg device the apple watch basically detected the same rem sleep segments as the eeg device and we can actually see that for many nights like this one here for instance again this same rem sleep segments were detected by both the apple watch and the eeg headband the main inconsistency i could find is that the apple watch quite often misses my first short ram segment in the beginning of my night as you can see in this night right here for example where the eeg device did detect some rem sleep but the apple watch did not and we can see the same thing in this example night right here again in the beginning of the night we see that the apple watch misses one ram sleep segment but it does show a good agreement otherwise throughout the night so this type of inconsistency happened a few times but generally the apple watch was more or less spot on or at least good enough to detect all rem sleep segments this also means that for most nights we can see the sleep cycles based on just the data from the apple watch now you go through roughly four to six sleep cycles each night each one starting with light and deep sleep marked in blue and each one ending in rem sleep marked in red as you can see i likely had one two three four complete cycles this night and a fifth partial one the apple watch detects the same sleep cycles as the eeg device so we would be able to see all one two three four of them and even the fifth partial one and we can see the same thing for most of the knights generally it will detect my sleep cycles quite well as you can also see in this knight right here where i likely had one two three four five complete sleep cycles detecting the moments i was awake was also done quite accurately by the apple watch roughly seventy percent of what the eeg device detected as awake time was also detected as awake time by the apple watch if it was predicted differently it was mostly predicted as being light sleep now this actually makes a lot of sense as light sleep is the closest sleep stage to being awake looking at the individual knights we can actually see that quite well this is again a similar plot to before with the sleep stages according to the eeg device on top and those of the apple watch on the bottom in green i mark my awake moments according to the eeg device as you can see there is a quite good agreement between the eeg device and the apple watch for this knight the apple watch mostly agrees well on both the shorter and the longer awake moments though it did detect a tiny bit of extra awake time right here and also right here and we see quite a good agreement for many nights as we also see for this one right here for instance it mostly detected my awake moment though it missed some here near the end of the night so it's definitely not a perfect agreement as we can also see for this night right here where the apple watch detects a bunch of extra awake time right here here and also here however overall i would say it's doing quite well now this is all looking quite good however it's important to know how this performance compares to that of other watches so let's see how these numbers compare to 37 other watches i tested over the last years however first a quick side note if you're interested in the latest updates on the wearables i'm testing i'm planning to start back up with my newsletter and posting more off the cuff things on my instagram and my youtube shorts channel so if you're interested in any of those those are linked below of course you would make me really happy and it would help my efforts if you subscribe to this youtube channel but of course that's totally up to you now enough self-promotion let's look at the performance of the apple watch compared to many other watches this graph shows an overview of the agreement of different watches with the eeg device along the horizontal axis we have the average agreement over the four individual sleep stages and on the vertical axis we have the agreement of the worst sleep stage for each watch the better the agreement the more to the top right the device is and as you can see the best agreeing devices i tested before include different fitbits whoop straps and the widthing sleep analyzer if i now want to plot the apple watch in the same plot i actually have to make the axis a bit bigger so let's do that so that plot is displayed right here again right here are the best performing watches from before so the fit bits whoop straps and the widthing sleep analyzer however the apple watch which is marked in red actually shows a better agreement with the eeg device than any of these other watches i tested the overall average agreement is just a little bit better so that's the horizontal axis here where we see that the apple watch is just a little bit further along than for instance the fitbit inspired too but the main thing that the apple watch did really well is that it performs quite well and really consistent over all four sleep stages which means it's higher up on the vertical axis so this is actually looking really good for the apple watch now apple mentioned one more thing when they revealed their new sleep tracking algorithm they mentioned that the apple watch can also detect when you might have woken up and this is also directly related to when the apple watch thinks you fell asleep for the night and when you woke up in the morning so can the apple watch actually detect this and in this graph i show how many minutes the apple watch differed from the eeg device in detecting when i fell asleep and when i woke up on the vertical axis we have the 18 nights over which i tested the apple watch and the blue dots right here indicate the difference in minutes when it comes to falling asleep and the yellow ones when it comes to waking up so the closer to zero these numbers are the better the agreement with the eeg device now the biggest disagreement was minus 50 minutes when it comes to falling asleep meaning that the apple watch detected me as falling asleep 50 minutes before i actually did i think i was actually just laying very still for a bit so it recorded me as being asleep and the biggest difference in waking up was minus 48 minutes meaning it detected me as waking up 48 minutes before i actually did however overall the differences are not that big mostly being within 10 minutes or so of the eeg device so this is not that bad so overall i would say that this is looking super promising for the apple watch however before i get to my final conclusions i want to mention a few important limitations of the testing that i showed in this video first of all i just tested the watch on me a caucasian male that is still relatively young or at least i like to think so though apple says that they trained their algorithm on a diverse population a lot of tech is still being developed by and for white men so there might be an over-representation of data based on this subpopulation meaning that the algorithm might actually work better on meat and on some other people second i used a reference eeg device with fewer electrodes than you would typically use with polysomnography so though i do think this gives us a good impression of the performance of the apple watch i will try to test it with polysonography in the future third i use the beta version of watchos 9. though i suspect nothing will change with regards to the sleep tracking in the official release i cannot be sure now the next thing i want to talk about is not really a limitation of my testing but i want to discuss one of the major features that is still lacking from the apple watch when it comes to sleep tracking the thing i'm still missing is automatic interpretation of the sleep results and sleep coaching based on these results i think it would be amazing if based on your sleep stages and other sleep metrics the apple watch would give you advice on your sleep schedule exercise routine and other things related to sleep because it can be difficult to interpret these data for people that are not familiar with them now the whoop strap and also the aura ring do this to some degree already however how to interpret sleep data deserves a separate video now the final limitation of my testing concerns my personal preferences i actually read in some youtube comments and even on reddit that there's some impression that i'm an apple fanboy someone even commented that they thought i had a girlfriend or fiance that works at apple well i have to say i like some apple products just because they work well for instance i use macbooks because they're unix based and it helps me with my work however on this channel i always aim to share the raw results as i get them and if there's something wrong with any product whether from apple samsung or huawei i will try to tell you and for those of you that care i don't have a girlfriend that works at apple or a girlfriend at all for that matter i'm single and i'm ready to mingle let's get back to the serious stuff we were talking about the sleep tracking performance of the apple watch now out of all the devices i've tested the apple watch has shown the best performance at least based on the metrics i used to evaluate it there are some devices that are very close in performance like fitbit and woop but the money and effort that apple has probably been able to spend on getting a good reference set of data seems to have paid off we will have to see how it performs on people besides myself but my first impression is a really positive one this also means at least based on my testing the apple watch is currently the best performing device in two of the most important health and sports tracking categories heart rate tracking and sleep stage tracking heart rate tracking is something that apple has been more or less the leader in for a couple of years now and you can find videos on that right up here if you want to buy an apple watch another device or anything at all on amazon for that matter even something as small as toilet paper and at the same time support the channel there are different affiliate links in description below they do not cost you any extra and some even provide a discount in the meantime i hope this video provided some value and you learned something more about the different smart watches thank you so much for watching and catch you in the next video [Music] you

Channel: The Quantified Scientist

Views: 392,662

Keywords: apple watch, apple watch sleep test, apple watch deep sleep, apple watch rem sleep, apple watch 8, apple watch sport, apple watch 8 sleep test, apple watch 8 sleep, apple watch 8 review, apple watch 7 review, iwatch 7, iwatch 8, apple watch sleep tracking, apple watch series 7, apple watch sleep accuracy, apple watch best, best sleep tracking, best smartwatch sleep tracking, best smartwatch 2022, best android smartwatch 2022, apple vs android, the quantified scientist

Id: LPqtfC70QTU

Length: 18min 20sec (1100 seconds)

Published: Mon Sep 05 2022

Reddit Comments

[u/cst516]

Sounds like Apple taking a long time has paid off for them.

[u/katsumiblisk]

Surprised at such a poor showing from Garmin and I thought Samsung would be closer to the Apple position. I didn't watch the video so I don't know any factors that might have influenced these tests.

[u/meerdans]

For those that want to skip the video and see the graph: https://imgur.com/gn2NBv2