CocoaHeadsNL Online Meetup, 17 February 2021

[Music] [Music] so [Music] [Music] [Music] so [Music] [Music] [Music] [Music] [Music] oh [Music] [Music] hello [Music] [Music] [Music] [Music] [Music] foreign [Music] [Music] family [Music] [Music] [Music] [Music] [Applause] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] my [Music] my so [Music] [Music] so [Music] [Music] [Music] so [Music] [Music] do do [Music] [Music] [Music] [Music] here [Music] here [Music] [Music] oh [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] thank you thank you [Music] hey [Music] thank you [Music] my [Music] hi and welcome to the february meetup of coca head chanel we are a monthly meetup of developers we are part of the international cocades meetup organization we talk about anything apple cocoa objective c and swift related technologies these are our sponsors they make it possible for us to execute these meetups every month if you want to have some swag branded with coca heads and all logos please have a look at our shop cocohuts nl shop you can also win a jetbrains product coupon code join us on slack in meetups and enter the raffle if you're not on our slack yet please use this invite link to join us as well the agenda for today is ryan mcleod ui kit engineers hate him how a black box challenge is made the creator ryan will take us through creating a black box challenge combining animations accessibility audio and perhaps one or more things please have a look at our meetup page for announcements and our next meetup in march did you download our apple tv app yet did you subscribe to our youtube channels did you have a look at our jobs and most of all did you sign up for a newsletter see you next month again this me hey everyone uh i'm here to talk about black box and how a challenge is made this uh talk is titled ui engineers hate him ui kit engineers shout out to uh pim commons for that title it's a great title um funny enough came up with it before the talk and then realized uh the challenge i'm about to talk about doesn't really use a lot of ui kit and i know a few people that work on ui kit and i don't think they hate me so maybe it's just a click bait so before we get started if you haven't already i'll give you a minute to grab blackbox it's free some in-app purchases and i apologize in advance for completely spoiling one of the challenges but there's 70 something more for you to solve so hopefully that's scanned so hi everyone i am ryan mcleod i am warpling on everything uh feel free to dm me on twitter find me on clubhouse whatever is new send me something so i live in amsterdam for it's been two and a half years now in the netherlands uh before that i lived in california and a lot of duchies asked me why i would uh move to the netherlands i think there's a lot of good reasons there's you have great bikes sort of work-life balance uh amazing infrastructure but there's there's one reason that almost no one talks about that i think is is really important and that's the like super native ios apps um that are made here i didn't even realize uh ns was a sponsor i promise but you have this amazing ns app which is sort of the transit app for uh getting all over the netherlands whether it's by boat or tram or train and it's just a really nice app for seeing where everything is going where your transfers are uh what the platform is it's really well made versus uh in the us i'm very close to our amtrak app with our nice credit card point offer and some of these very spacious spacious views another app that i really like here is a post-ino which is sort of like for checking your mail it shows me like all of the mail scanned before it's delivered i can like generate postcode or stamps that i can write it has widgets it supports dark mode uh usps is on its way some really lovely web views and some some misaligned icons feel bad i feel bad ripping on usps but there's other apps like uh ticket swap it's really really nice uh mark plotz dutch craigslist pretty i mean a lot of you probably don't like it if anyone in the netherlands is maybe thinking that but compared to this which is the uf alternative for it it's uh you're doing doing pretty well so really really nice community here and i think a lot of these people behind these apps are involved in cocoa heads so thank you for letting me join you so i am here to talk about black box which is probably what i'm best known for and black box is a very different kind of puzzle game where you generally don't touch the screen it's uh comprised of a lot of different puzzles that can vary on how they work based on like simple device movements to celestial movement and it's really hard to say a lot more without spoiling it but if you know you know so blackbox was awarded an apple design award in 2017 for excellence in design with an emphasis on accessibility that i'll get into later it's been really wild five years coming up on i've gotten to meet all sorts of people around the world people have gone some dangerous places for black box uh including everest i don't think black box sent them there but i like to think it was partially inspired there's black box merch all around the world there's a lot of really cool fan arts we have some black box dogs and i think just generally black box has improved um a lot of people's lives at least i like to think that and uh yeah there's there's also a black box tattoo out in the world which very flattered after mixed feelings about so it's a it's been a big journey it's kind of how it started versus uh where we are now black box is something went from 40 something challenges to around 80. it's in about 22 languages from dutch to georgian it's about to come out on ipad next week supports dynamic type it's fully accessible supports switch control has a ton of sound has new haptics list goes on there's merch you can provide an app with cryptocurrency and apple pay point is like the app has gotten very very complicated so i would like to talk about just one tiny part of this that i've wanted to talk about for a long time which is one of my favorite challenges and this one i internally call dark scan and it's part of a series of challenges that use the camera but unlike similar challenges it doesn't use any qr codes so you can see if you're wondering where it is on your grid or if you've just started it's this little turquoise one down in the corner here so to talk about all this i'm going to break it up into four sections we'll try to keep it to half an hour but we're going to talk about the level design the visual design and implementation the audio design and accessibility so starting with the level design i think there's a common misconception that i am out of level ideas probably because i haven't uh put out a new challenge in a long time it's a good reason for that but the the truth is i have too many ideas there's always been a lot of ideas i i also have a wonderful discord of thousands of crazy people that submit fantastic ideas all the time um the problem isn't ideas it's it's more figuring out how to implement them well and yeah how to really execute on them uh i promise when i say i have plenty of ideas there are it's a it's not a limited resource so i thought i'd talk about kind of where an idea comes from really briefly i think this challenge of dark scan is fairly obvious in hindsight and the idea is that you find or make a black square in your environments and you scan it it kind of just comes from the name of the app black box but i think the inspiration came from some other places too this was around summer 2017 uh i just started making some merch so there's this idea of like having to scan a logo on the merch or something like that ml kit had just come out thought maybe we could train a machine learning model type thing seemed really gimmicky probably too hard or air prone i remember a friend had one of those like uh pin artboard things where you like can push your hand into and see the impression uh i think that was a big influence i honestly don't remember that much there's plenty of other sources uh just the other day i was looking at this sort of washing machine barrel fire pit and that kind of has some cool inspiration into some of these pixel effects point is inspiration kind of comes from everywhere so as far as taking the idea and turning it into a plan it's interesting because if you have this sort of idea of you have a black square and you want to make an app that's it as intuitive as possible to find it and scan it you could just provide those as instructions make a camera view make it kind of turn green when the thing is scanned but it wouldn't really be any fun you would just scan it and it would be done and a big part of creating a puzzle is sort of creating a gap in that understanding so one way i like to think about it is like this if you have you want to design the most intuitive interface possible and in black box i always try to do that without the use of any language so something generally visual and you want to be able to take this interface and then sort of break it so if you imagine you have this fort here in hernia you have this sort of lovely path into its center where the answer is if you took that interface then and sort of put a moat around it and split it up so it became less intuitive and then you have this gap now which you can let the player connect and that sort of is the aha moment that makes black box so satisfying and fun to play a lot of different puzzles take different strategies with this sort of stuff so if you imagine like a jigsaw puzzle a jigsaw puzzle like everyone knows how to solve it it's mostly just uh time and difficulty i guess versus like some of these desk toys that are like rope and metal uh it's not clear how to solve it and you might kind of have a strategy but you just mess with it for a while and then eventually i find these frustrating because i think they just kind of like come apart in your hands and then often you don't know how you did it that's not super satisfying and the approach i like to take with black box is a little bit more like uh baby rattles so this is sort of proprioception and how we learn proprioception so the idea with these if you're not familiar is like they're kind of crinkly or they have bells or make sounds and you put them on a baby's wrist or their legs and as their like mushy brain is just trying stuff and limbs are flailing they can sort of associate more sensory input with the thing that they're doing and eventually we learn how the controls work and hopefully we can like solve problems in the world so i like to think about black boxes uh interfaces sort of like that where it's providing this sort of rattle where you are able to learn how your world is affecting the challenge and it guides you to figuring out that solution and then actually carrying out the solution is not the difficult part sometimes it is but mostly it's figuring it out and then just executing it so once there's a good idea the question is kind of how to build it next and this involves like looking through sdks and apis usually to see if the problem is uh feasible and for anyone that doesn't know that's just kind of the tools available to anyone making an iphone app to do things with the sensors so when i was evaluating this idea to see if it was feasible i first looked at opencv open commuter vision it's open source project for detecting things and camera feeds i'd done some work with it before and knew that scanning objects with hard edges was easy but it was kind of a bulky big framework to use for this score image detector is apple's sort of built-in thing which i knew worked really well with still images but was like quite jittery with um live video and such and then the last but not least is like straight up pixel buffer so that's just means like the actual color values of the grid of pixels making up the image and from that you can do anything with enough math so once sort of there was a understanding that this challenge was going to be possible to implement in the way i wanted the next thing was to figure out the visuals of it and one of the things that's nice about working by myself not with a giant team is this kind of can happen at the same time as the level design phase and these can work really nicely in tandem so i kind of began exploring them at the same time like i mentioned uh this sort of pin art thing had been in my mind and i had made some other challenges that had these pixelated views so that seemed like a really obvious idea but not quite right wasn't satisfied with it i started browsing uh with my favorite algorithm pinterest it's always scary because you just scroll and it seems to get to know you better and better and i found something that kind of stuck out to me which was this this grid thing which you might think maybe this is someone's art or something but it turns out it was just this uh downloadable pattern pack so shout out to david zydd so i got kind of fixated on this and normally i would jump to paper first but since since this idea was just black squares i just hopped directly into sketch from there i sort of drew out some squares threw that on my phone with the sketch mirror sort of looked at how it felt and sort of arrived at these two visual elements so from here i had this idea of these grid of black squares that would get larger and smaller based on the luminosity of like the pixels behind them and this this reticle like on a camera so the idea was that this grid would kind of represent the camera feed and what was happening and guide you towards what in the world was affecting this challenge and the reticles could grow to kind of complete themselves into a square when you were close to solving the answer and this works really well because if you've ever seen like an apple watch with the rings humans really hate like incomplete shapes especially when you have this nice like motion uh easing on this so it kind of it very quickly comes close together and then the last 10 is kind of 50 of the work and one way to think of this is kind of like the the how and the what of it one is what input and how the sensor works and the other is what is your goal here and a fun side effect of this design was i don't remember how intentional this was but just the idea that like you're looking at a bunch of black squares and like you're trying to find a black square it has a nice nice metaness to it where the answer is staring you in the face the whole time so after thinking about it a lot further doing a lot of whiteboard and paper math and apparently posting artsy photos of it on instagram i just found all these it seemed the most natural to use just the straight pixel buffers to drive it and that would keep the visual really tight to the actual sensor input and from there the idea of building it was basically to use av capture session which generally provides the kind of camera feed that you would show filtered images on that can come in different sizes on different devices so i would do kind of an aspect fill on that and crop the image from there it's as simple as applying a bunch of core image filters so i kind of uh change the image to be monochrome boosted the gamma of it sort of wash out the really light elements and then resize it down and what you end up with is this really nice 2d array of luminosity values now from here this can be pumped straight to this uh sprite kit scene of sk-shaped nodes and i think i must have for the ui kit part of this talk i must have tried it with core animation layers when i first built it and seen a lot of performance issues because i tend to avoid scene kit at all costs unless i need to do a bunch of physics but it works really well in this case because it's very straightforward so for the other visual i'm able to take that array of luminosity values and do a bunch of math and get this nice progress value that ranges from 0 to 1 for how close to completion the player is and that can drive this reticle to kind of open and close and also drive a bunch of the audio and other things so the math just to clarify i first had this sort of thing where it was just looking at an average luminosity luminosity of the section with squares supposed to be scanned very quickly discovered that if you run this and your phone is laying on the table and the entire screen is black then you just solve the challenge so not very fun very easily solved by also checking this outer ring making sure it's above a certain luminosity and then checking the contrast between those two and coming up with that value there's a little bit more to this but that's the gist of it so moving through these quickly um all of the audio in blackbox is done by my sound engineer gus callahan and all of it is powered by audio kit audio kit is this insane open source audio library for ios that powers a lot of the software that djs you know and love use and it also means every single challenge in black box can be powered by its own digital synthesizer so if you imagine a setup of having a keyboard and filters and wires running between all these different things it basically allows you to build any of those completely digitally in software this is a large part about of uh what makes black box accessibility accessible to blind players these kind of alternative sonic interfaces that i'll talk about so the audio in this challenge if you're not familiar this is a little clip of what it sounds like you can hear it kind of a change you hear this kind of repeating sound come in [Music] so that whole thing is made out of four like static sounds basically which is pretty insane and how that's built up having a trying to advance this slide here i think i'm having a okay there we go so i'll have to do this through you guys um you have this first player here that if we go we'll uh play it sound so it does this repeating sound [Music] and then you have this uh other player that is looping so we call this one the sort of donk player um so this one gets repeated it just plays the entire time and then we also have these other ones if we uh there we go so this one is kind of the ambient player that plays the entire time and then we have another one that this one gets uh mixed in it's very similar sorry trying to figure out some [Music] slide control issues here so this all gets fed into a mixer which if you think of it's kind of like a a usb hub of sorts so all this audio comes in and all of it eventually needs to go out um to whatever your output is so that could be the speakers or airpods or whatever is connected so what's interesting here is the opening player can be its output can be directly wired into the mixer and that just means when it gets played it immediately gets heard but some of these other ones we want to do more advanced things with so uh something like this donk player is just going to continue to make this sound but we can feed it into a vector speed which will we can adjust how quickly the sound is played back and then that can be wired into a bit crush bit crusher which sort of does like an 8-bit type effect and lowers the sort of resolution of the sound kind of adds a nice chiptune effect um i think this is some of the genius in gus's sound design where this kind of matches in parallel with the visuals where you have this very pixelated view and this is like literally pixelating the sound so that gets connected directly to the mixer and then the next thing we have to deal with are these these two players so there's this dry wet mixer that they both get fed into dry wet like confused me for a really long time a better way to think about is just like hot cold so you can kind of mix these two together however you want and that gets fed into a verispeed so we can adjust the speed along with the donk player it gets passed through a delay and eventually also gets bit crushed and fed into the mixer and then you have this entire thing playing the whole time and as everything else is happening we can uh adjust all the parameters on these so as the player gets closer to solving the challenge the speed can be increased the volumes can be changed the mix can be adjusted and that's how you end up with that sound which i'm going to play again so you can kind of listen to and see if you can hear some of the the repeating players come in and sort of change as it gets closer to the answer my do [Music] cool so maybe now you can understand a little bit of the craziness and some of the sound design in black box there oops i don't have control it's like having some technical one second it's a very crazy setup here it's uh similar complexity of dealing with video cool okay i think we're back here so last but definitely not least i wanted to talk about the accessibility in black box um when i know i laid out the talk like this i'm still not quite sure this is the right way to do this but really accessibility is a consideration at all these steps the entire time i think it's a little easier to talk about it at the end like this but it's something that i think about when designing levels when thinking of the visuals and especially the audio and voice control and stuff like that and accessibility is about making our apps just able to be used by everyone depending on if that state is sort of permanent or temporary and if i couldn't make this challenge accessible if i couldn't have thought of a way to do it i wouldn't have made it so accessibility i think we tend those that know it in the ios world tend to think of it as voiceover and labeling our elements and such but it's a lot more than that it's the overall design and in blackbox it's also supporting things like switch control and full dynamic type and respecting when users request reduced motion and reduce transparency and things like that in blackbox it's it's very interesting so some of these challenges are the game originally was completely visual and when some players blind players approached me and asked if the game was accessible i had to start thinking if this was something that actually was a blocker or if this was something that i could solve and i think part of the determination in making black box fully accessible was kind of a proof that if this game can be then your apps definitely can also this is you might be wondering how this works though so um one of the first challenges in blackbox is this red challenge here and this involves moving the device around and seeing this different this um these visuals move around to sort of tell the orientation and if you're blind there's this sonic interface and even if you're not this also helps so hopefully you can hear this here this uh sort of guides you where the sounds get louder and sound more pleasing as you get close to the edges and close to your answers this challenge has no voiceover in it whatsoever and it's completely solvable i still remember the the first time i watched a blind dester solve this challenge and he got through it with ease and it really blew my mind to think about i've watched so many hundreds of people solve this challenge um and they have all solved it through seeing this visual and this person had a completely different interface to understanding the whole thing and was able to solve it and have the same satisfaction other challenges uh rely on voiceover a little bit more here you can hear some of them voiceover on so a lot of these elements are labeled large slider 78 on by swiping around navigating um players can listen to small switch what the state of these different items so this challenge poses a really interesting problem um because it is completely visual and it's also very finicky it wants this pixel level precision of this black box being exactly in this reticle and if you are visually impaired this is probably going to be pretty difficult to line up correctly but when i was in that phase of thinking through the apis and what i could use i had the idea to use core image detector as sort of an alternative interface and it's typically a bit of a no-no to like create a different way of doing something versus making the primary way accessible but i think in this case it's okay because the challenge is a little bit uh it asks for a little bit more as far as like that square being really straight and such but it's more forgiving and that the square can be anywhere on screen and so the way i thought to do that was with the core image detector here you'll see i sort of made a little metal view for seeing the camera view and overlaying what the core image detector sees and these are these polygons are squares that it thinks are or thinks are potential squares maybe and with this sort of debug test here i was able to see um that this would be a plausible way to make this work with some additional work so if this is normally what the challenge visual looks like to everyone i have sort of a debug mode i can turn on that makes it look like this and with ci detector you get these these polygons but you just get points corners so in this case i get these corner positions when it thinks there's a potential square and from there i can generate edge lengths i can get corner angles i can look for sort of the average color of that shape and i can generate a whole bunch of other things including orthogonality which is just how parallel that shape is and the squareness of the rectangle from there we can basically do a different kind of math and come up with a new progress based on how square the rectangle is and its darkness and such and that can drive the visuals but it can also drive the the audio from the audio audiograph that i talked about and it can provide some additional voice over feedback so now the game can kind of announce these things as the player is moving the phone around they're hearing sound change a little bit and when it changes just enough and something is just plausible enough it might announce one of these and then with that and the sonic interface players can kind of be guided towards a solution and here i'm going to play uh what that sounds like challenge type 47 clue black square's on a grid sound like a lot large semi-dark colored square detected square is not parallel with camera small semi-dark colored square detected square is not parallel with camera small semi-dark colored square detected small small semi-dark colored square detected square is not parallel with camera some threshold adjustment there but so that's one challenge um there are many many more that are just as complex or a lot simpler i hope that that gave you some insight into how blockbox functions and maybe gave you some ideas for your apps and serves as a nice example of how you can make something like this possible and make it completely accessible so if anyone has questions or comments i think i will be around answering some of those now ah finding my questions here cool so i think i'm reading these here so let's see what we had unit tests um i wish i could claim that there are tests i initially wrote some tests but being an independent developer there are there are no tests there's a lot of beta testers and a lot of thinking carefully but next project will definitely start with tess there's a question on youtube of did apple feature this app yes uh quite a few times that's been one of the things that helps me do this independently are those features so it's been um game of the day and featured in a few other puzzle categories and stuff like that along with the design award was a nice feature let's see what else we have here we have uh did he create some kind of nice setup to easily jump into a certain level during development yeah there there's a lot of um i have a whole bunch of hidden things in the debug builds that allow me to uh like remove transitions to videotape things for promo materials that challenge specifically i use um swift tweaks and fb tweaks and those i can like jump in and adjust a lot of different thresholds and parameters which during testing is a lot faster than recompiling the app over and over someone asked if the ipad app is built with swift ui um no it took me probably over two years to get everything ready for uh for ipad but i haven't touched swift ui too much yet to be honest i found some of the animation stuff that i do is a little difficult but when i have some down time i would like to start messing with it more the whole app was also originally written in an objective c because i started on it six years ago or so and swift was brand new but now about half of it and most of what i work in day to day is swift awesome um it's a debug mode accessible in the app store version i'm not quite sure what you mean but there's a uh like the excessive or the i have developer menus in the app i think i think actually a lot of them are localized and they're definitely accessible because they use all the same ui that the rest of the game uses um there's someone eugene writes that their uh 12 year old daughter started to learn swift playgrounds from blackbox which is really really cool i use playgrounds to test stuff um in blackbox so that's awesome and then if uh did you start to work on it full time straight away uh basically it i thought it might not be possible but i was i was living pretty cheaply in california um definitely thought there would be this like giant amount of becoming a millionaire gold rush type thing if it succeeded and then very quickly realize that you have kind of this that graph of like getting featured by apple and then drops off and then learning like oh this is actually just gonna be like a lot of work but um did see the path forward immediately and was able to do that had a few times of like applying to jobs and stuff and thinking it wasn't going to work anymore but managed to pull it through and how long did it take to build the first 40 levels i think that's really interesting because the the first 40-something levels i built in a year with the rest of the game and since then my development has slowed considerably having supporting all the sort of accessibility stuff having 22 localized languages supporting more different kinds of hardware uh doing all the sounds all these things have made it uh take a lot longer to add challenges so i think 40 in the first year working on it and then about 40 in the next four years so things have slowed down yeah there's a lot of questions uh from korean have you used any machine learning models um i haven't i've always been curious to but i always found it a little bit uh found them a little finicky and difficult and i think i've just kind of been intimidated i've been wanting to mess with the lobe ai which i think makes that a lot more accessible but it might might be better as a different game not sure then um when you started building it did you look into other games that are somewhat similar per liu maybe uh perlu rings of bell i there was i'm trying to remember what there were some things that inspired it one was like uh the to-do list clear they had some like secret themes that unlocked if you like open the app really late at night and then there was this inception app for the movie as a promotion and that has some really weird things in it like if you opened it while it was raining uh it unlocked something if you opened it in africa it unlocked something and i thought that was really cool along with like all these other apps from the beginning of the app store that there was like pocket warmer that just like turned on all the sensors there was like a bulldozer they used the vibration motor to like push the phone across the table it was like there was a air blower for like blowing out candles like people were doing really cool stuff and then all of a sudden everyone was just doing like lowest common denominator like fits under a glass screen works on the web works on android does everything not taking advantage of any interesting sensors and i really wanted to do that um there was one app uh progress 100 that i discovered like halfway through uh making blackbox that it wasn't inspired by but is kind of very similar and cool uh is the app going to use the lidar sensor in the future this has been asked quite a bit i tend to avoid hardware that i figure most people don't have um once it sort of reaches a majority so maybe in a few years it would be really cool to do but it also might be really cool just to make a new ar something which api do you think apple would need to add at the next wwdc oh man i would love access to the temperature sensors there is an actu well there is a temperature sensor uh it only gives you four temperature dates so i've kind of avoided making a challenge with that because i'm afraid so i want to put their phone in an oven since people have done other weird things um how can you live cheaply in california good question coming from here it's uh i think when you live with three other people out of college in a small town it's it's not not totally out of the question especially with how uh tech and being an ios developer tends to pay you can always do subcontracting um do you have challenges that are available exclusively in one language no that i think that would be a little bit limiting um i think that you mean like spoken languages but no there's one challenge that kind of involves that but i like to keep it open to everyone and then i think we have one more here uh what's what is the level that was most hard to debug oh i have to think about that uh let's see this one is still i think could use some calibration um oh man there's another one that uses qr codes and they have to be you have to scan other people's qr codes and they they generate nonces and they expire and stuff i was trying to do it without a server that one was kind of hard to get working as as i wanted it to with all the physics but i think it looks pretty good now so i think i think that's it if anyone else has questions uh feel free to find me on on twitter or anywhere else i'm warpling should be up there somewhere so thank you for this opportunity and thanks for sticking around [Music] [Music] [Music] do [Music] uh [Music] [Music] oh [Music] [Music] so [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] you

Channel: CocoaHeadsNL

Views: 514

Keywords: CocoaHeadsNL, CocoaHeads, iOS

Id: ZZm9J2Xi6Rs

Length: 78min 45sec (4725 seconds)

Published: Wed Feb 17 2021