The REAL Story On Why Space Cadet Pinball Was Removed (ft. Windows on Itanium)

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments
Captions
If you used Windows through  the 90s or early 2000s,   then these should be familiar sights and sounds.   They are, of course, the welcoming chimes of  Space Cadet Pinball, a game that shipped as part   of the base system for over 11 years, before being  removed with the release of Windows Vista in 2007. The closest anyone got to an explanation  as to why this beloved game was removed   was when Microsoft insider, Raymond Chen, wrote a  blog post about it that set the record straight.   Specifically, he states that the game was removed  due compatibility issues with 64-bit processors,   and that he spent many hours trying to fix  the game before giving it up as a lost cause. On the surface, this might make sense, but what  if I were to tell you that this wasn’t the truth? In fact, what if I were to tell you that   I have multiple versions of a  64-bit Pinball game that work? Well, get ready, because this is your host  NCommander, and today, we’re going to chart   the final days of Space Cadet Pinball, and trace  the chain of events that lead to its removal.   Our journey will take us from  the classic Windows XP desktop,   to Intel’s failed Itanium architecture,  the Longhorn project, and so much more. So I recommend you all get comfortable, and  let’s wind this back to where I started,   which was with a video posted  by Michael MJD a few weeks ago. If you’re not familiar with his channel, Michael  MJD did a video demoing a pre-release version of   Windows XP x64. During his exploration,  he showed Pinball running just fine,   which immediately got my attention, and  prompted me to start my own investigation. The easiest thing to do was to see if Pinball  had persisted to the final release of Windows   XP x64. After completing the installation process,  I can say yes it did, and it’s even listed in the   Games folder. I played a few test games, and  it seems to be just fine. Or well, mostly fine. If we look closely, we can see there’s some  graphical corruption. Several lights are   misaligned, the left flipper is clipped, while at  the same the right flipper has gained an outline.   At the same time, the ball plunger  appears to be a bit camera shy.   That being said, the game appears to  play just fine. Still, this means there   was indeed a 64-bit version of Pinball, and  while not perfect, it’s obviously playable. Just to confirm, I did actually load this  version of Pinball into the reverse engineering   tool Ghidra, and confirmed that yes, it is  a native 64-bit version through and through.   Since this is a good time to bring this up,  Microsoft actually shipped what are known as   symbol files in their debug kits for almost  every version of Pinball I examined, which   drastically aided my investigations. It’s also why  I have function names both here, and in WinDBG. Still, with a native 64-bit Pinball in hand, it  would seem that Raymond Chen’s post is wrong.   Or, well, so it would seem Let’s actually read the relevant post together:  In it, Raymond writes, “One of the things I did   in Windows XP was port several millions of  lines of code from 32-bit to 64-bit Windows   so that we could ship Windows XP 64-bit Edition.  The 64-bit version of Pinball had a pretty nasty   bug where the ball would simply pass through  other objects like a ghost. In particular, when   you started the game, the ball would be delivered  to the launcher, and then it would slowly fall   towards the bottom of the screen, through the  plunger, and out the bottom of the table.” Raymond then goes on to say that he and a  coworker tried to debug the game to no avail,   and that he believed the breakage was due to an  obscure floating point rounding issue. Writing a   patch was complicated by the fact that Pinball  was originally created by an outside company,   and that it’s source code was incredibly messy. In the end, they made the executive decision  to cut the game instead of trying to fix it. However, I played quite games on the 64-bit version of Pinball, and I didn't encounter any sort of collision or clipping issue That means, on the surface, it would seem that  reality, and Raymond’s blog post disagrees   with each other. However, this discrepancy could  potentially be cleared if we take a closer look.   Specifically, the operative key words  here are “Windows XP 64-bit Edition''.   In a move that would cause numerous headaches,   Microsoft released multiple 64-bit versions of Windows XP with frustratingly similar names. What we’ve been looking at was officially  called, “Windows XP Professional x64 Edition”.   However, in practice, a better name might have  been “Windows XP for AMD64 Compatible Processors”,   and it was the basis of the second and  ultimately successful attempt to bring   64-bit computing to the masses. Today, all  modern PC processors, even those from Intel,   are AMD64 compatible, and are frequently  referred to by the term x86_64. I normally wouldn’t need to make this distinction,  but to unravel this mystery, we need to dig into   Intel’s original 64-bit platform. That is to  say, we need to step into the world of the Intel   Itanium architecture. Itanium was originally  intended to be a replacement for Intel’s aging   x86-based line of processors, but failed to  have much impact. Honestly, the best thing   I can say about it is that it provides EPIC  examples of both how not to design a processor,   and how an entire industry can shoot itself in  the foot, but those are stories for another day. What’s important to know here is that Microsoft  did have a relatively short lived port of Windows   XP to Itanium, known as “Windows XP 64-Bit  Edition”, and this is what Raymond Chen is   likely referring to. When I started this  investigation, one of the first things   I wanted to check is if Pinball was present on any  version of Windows for Itanium. As it turned out,   when I checked the Windows XP 64-Bit Edition CD, there was a surprise waiting for me.   Hidden in the ia64 folder is this file called  pinball.ex_, which is a compressed installer file.   Decompressing it on Windows 10 shows that it is  indeed Space Cadet Pinball, and an examination of   the file shows that it’s indeed a native Itanium  version. As one might say, the mystery deepens. However, testing this binary is slightly  problematic, as Itanium binaries can’t run   on normal x86 machines, nor is there  any emulator available. That however,   isn’t a show stopper for me. As it so happens, I  actually own Itanium hardware, specifically, an HP   zx6000, which has admitly has seen better days.  Still, this system works, or well, it did. I had to do some maintenance off camera to  get it running again. While it was in storage,   it seems one of the memory sticks had failed,  and the second SCSI HDD had also checked out. I   also had to replace the CR2302 battery.  Still, with some prodding it kicked to life   with an EFI boot screen. Now came the “fun”  part, installing Windows XP 64-Bit Edition. My first attempt didn’t quite go to plan however.  With the Windows XP 64-Bit Edition CD in,   the setup process would launch, run for a bit,  and then crash. This would then be followed by   system error lights, and POST code beeps.  It should be noted that while I’ve had this system for a  long time, I primarily ran Linux in those years.   While I have installed Server 2003 and 2008  before, XP was a new experience for me. Well,   as it turned out, I had missed something  in my research up to this point. While Windows XP 64-Bit Edition was designed  for Itanium hardware, it wasn’t designed for   MY Itanium hardware. Instead, it was meant  for the first generation “Merced” systems,   and is incompatible with my second generation  McKinley system. What I needed was the annoyingly   difficult to Google “Windows XP 2003 Edition”,  and yes, that’s the actual product name.   It’s even printed on the certificate  of authenticity slip for this system. With the XP 2003 CD inserted, things  start looking up! This time, instead   of crashing the system, we’re now greeted  with a somewhat squished TXTSETUP screen.   I had a few more hiccups, but I’m going to  gloss over most of the installation process,   and save it for a dedicated video. Still,  in the end, I was successful! After   the installation finished, we’re  greeted with a tiny login screen,   a stark reminder that early versions of Windows  didn’t have great support for high DPI monitors. It’s at this point I’m going to transition using  Remote Desktop to record footage from this point   out. It’s not ideal, but it’s going to make things  a lot easier to show and demonstrate. Even on the   login screen we can already see quite a few  differences as compared to “stock” Windows XP,   such as the revised branding. That’s not counting  things such as the Luna theme being unavailable,   and the lack of fast user switching.  Infact, the entire system feels unfinished,   and it honestly behaves more like  a beta build than anything else.   I have to wonder if this product only exists  due to some sort of contract obligation. That being said, if we go into  the Start Menu, we can see that   Pinball is nowhere to be seen. It’s not even  listed in the Windows Components wizard.   A closer examination of the setup files on the CD  shows that Pinball was removed from the manifest,   and thus would never be installed. That means the  existence of pinball.exe on the XP installation   disk appears to be an accident, and thus  this is almost certainly a broken binary. However, I don’t need to take that on faith.  By looking at the installation manifest from   other versions of Windows, I can determine all  files necessary to install Pinball by hand,   and it was trivial to extract these  for testing. With a little effort,   Now, assuming that Raymond Chen’s blog post  was accurate, we should have a mildly amusing   glitch. Unfortunately, when I try to run it, it  simply crashes to the desktop. I actually ended   up installing WinDBG and try to isolate the cause  of crash, and from what I can tell, it attempts to   load several of the game resources, but then  ends up crashing with a memory access error. I had trouble finding the specific cause of  the crash, partially due to the fact that   Itanium assembly code is notoriously difficult to  read. I did see that not long before the crash,   the game tries to load a bunch of floating  point numbers relating to collision detection,   but I’m not certain if that’s related. However,  it would fit in with what Raymond Chen wrote. It should be noted that I also managed to crash  the debugger multiple times while doing this,   which should give you an idea of how  much pain and suffering I went through.   Thankfully, I did eventually dig up a newer  version of WinDBG which was much more stable,   but it’s never a good sign when  you need to debug your debugger. It was while I was in this debugger purgatory  that I had something of a revelation:   since there were multiple  versions of XP for Itanium,   it was possible that there were  multiple versions of Pinball.   Sure enough, there is a compressed Pinball  binary on the XP 2003 CD we just installed from Compared side by side, we can see differences  in the file size and modification dates,   but the biggest version here is that  this version actually starts up! That means we should be able to see Raymond  bug, right? I want you all to put in the   comments what you think is going to happen  when I launch the ball. As a reminder,   according to the blog post, the game’s  collision detector should go completely bananas. Well, that should be long enough. Let’s try it. Bet you all weren’t expecting it.  Infact, I wasn’t expecting it!   I am not exaggerating when I said I spent days  trying to figure out how this was possible. It’s pretty clear though that  something isn’t adding up.   First off, I want to make this clear, this  is a native Itanium version of Pinball,   and not the 32-bit version in disguise. We can  clearly see that with the binary loaded in WinDBG.   Furthermore, just like the version we saw on  AMD64, there are some minor graphical glitches. As far as bugs go, I also found I can’t launch  the ball while playing over Remote Desktop.   I think this is a problem with how rdesktop  is handling the space bar though, and not   a problem with Pinball per say. Even taking that   into account I can play normally by  using the “Launch Ball” menu item. Regardless, I now have multiple 64-bit  versions of Pinball, and while not perfectly,   they’re clearly serviceable. This would seem  to indicate Raymond Chen was entirely wrong on   why Pinball was removed. However, as strange  as it may seem given the available evidence,   I actually believe his account. I just  think there’s more to it than that,   especially because I can prove part of his story. Let’s take a look at a version of  Pinball I modified slightly with Ghdira.   Keep your eye on the ball. Did you catch that? I’ll  play it again in slow motion. The ball clipped through the paddle and drained.  I caused this by changing the FPU precision mode   from 53 bits to 64 bits. The increase in  precision introduced the type of rounding   issues that Raymond had speculated about,  and broke collision detection on the paddles.   Furthermore, this FPU mode would  be similar, if not identical to how   floating point numbers would behave on both  Itanium, and on the x64 builds of Windows. I think though before we go on, this  merits a bit more in-depth explanation. If you’re not familiar with  what a floating point number is,   then the short version its how a computer  handles numbers that have a decimal point.   The slightly longer version is that a floating  point number is an approximation that is useful   for handling very large and very small  numbers. Now, I could bore you with math,   but let’s go with a visual demonstration,  and for that we need Minecraft Beta 1.7.3. Now, you might be wondering why we're here ... Well, in this version of the game, if you go far enough away from the center of the world, reality kinda breaks ... In Minecraft, the position of the player  and various in-world entities are handled   through floating point numbers. For instance,  at 1 million blocks, we can see that the hit   box is off center from where it should be,  and movement is no longer smooth and fluid. If we skip ahead to 2.1 million blocks out, we  can see that the precision issues have become   noticeably worse. For instance, the hitbox  on the furnace is now considerably offset,   and flames from the torch are no longer aligned.   This is an example of how floating point  values can break down given extreme values. It should be noted that one of the most famous  glitches relating to distance in Minecraft,   the Far Lands, is not a precision issue.   They’re instead caused by what  is known as an integer overflow. However, since we’re now 12 million blocks away  from spawn, everything has become considerably   more broken. At this distance, the hit box  appears to be drunk, and well, you can see   the fire effects for yourself. Even the F3 screen  is having trouble with reporting our position. The “too long, didn’t watch” verison of all this  can be summed up as “decimal places are magic”. Now, to give you an idea of  how different precision modes   change things, let’s migrate this world  to the current version of Minecraft.   We can see the furnace, torch, and  Far Land chunks are still here,   but things are considerably less broken. Our  hitboxes and particle effects are exactly where   they should be. This is because in the intervening  versions, Minecraft changed its code to use   double precision floats, which can handle  much larger values, hence the lack of jitter. Conversely, our Pinball problem could be seen  as writing code which expects the glitchy mess   present in Beta 1.7.3, but is instead getting  the nice smooth behavior from Minecraft   1.17. I do hope this explains what is,  unfortunately, a non-trivial topic to grasp. However, the question is now, how did Pinball  get fixed? My first thought was that this was   potentially an accident. If Pinball was fixed due  to changes in either Microsoft’s compiler or the C   runtime environment, it might  be working by happenstance. Furthermore, if the XP x64 team didn’t realize  that Pinball was supposed to be broken, it   might explain why it was left enabled. However,  as I continued to look deeper into it, it quickly   became apparent that there were real code  changes, and this isn't some sort of accident. Over time, the pieces of a much larger puzzle  began to slot into place, and I ended up tracing   the history of Pinball from its NT days, all  the way through post-reset Windows Longhorn.   All my research led to one inescapable conclusion:   Pinball was intentionally  fixed for 64-bit platforms. I do honestly suspect that Raymond Chen had  assumed that after Pinball got cut from Windows on   Itanium, that was the end of it. We already know  that isn’t the case, but, it does beg the question   I’ve been building up through all the video: just  what exactly happened to Space Cadet Pinball? We know that Pinball was removed from the original  XP for the Itanium release, and at the time,   Pinball didn’t work at all. Meanwhile, a working,   albeit still disabled version shipped in  XP 2003. That might seem rather strange   unless one was to take a look at what else  Microsoft was doing in this time period. You see, in parallel to their work on Itanium,  Microsoft was also working on porting Windows   to AMD’s new x64 platform. While this port  wouldn’t ship until 2005, the earliest known   betas date to 2003, and it’s easy to assume  that work was being done well before that point. It appears Microsoft had intended to release  the AMD64 port with the next version of Windows,   which was known as Longhorn. For various reasons,  this future didn’t quite come to pass, however,   it can give us insight into  what Microsoft was planning. Many of the answers to this Pinball  mystery can be found in these early builds,   and specifically, Windows Longhorn build 4051.  Trust me, this is where the story gets a bit wild. Build 4051 was originally compiled in late 2003,   and released at the Professional  Developers Conference that same year.   It’s one of the earliest known AMD64 builds  of Windows and also it’s one of the few builds   that compiled all three architectures. As  such, we can compare them side by side. As I already implied, Pinball is present and works  just fine on the AMD64 version of Build 4051. Just   like all other 64-bit builds, there are a few  minor graphical glitches, but it otherwise plays   just fine. However, the real interesting thing  here is in the 32-bit version of this build. As one might expect, the i386 version also has  Pinball, and it shouldn’t surprise anyone that   this version is flawless. However, in  an attempt to ascertain if Pinball had   notably changed post-XP, I ended up doing the  same binary patch to enable 64-bit precision.   This is what caused the ball clipping bug before,  but on Longhorn, it does something different … That is to say, it works just  fine. This was my first solid   evidence that real code changes had been  made to fix Pinball on 64-bit Windows. In fact, this build of Longhorn tells us  quite a bit more than one might realize.   It suggests that Pinball was fixed in  the process of creating the AMD64 port,   and it’s likely no one realized that fixing  it for x64 also fixed it for Itanium. The IA64 version of Build 4051  leads credence to this theory.   Getting it to install on my zx6000 was not a fun  experience, and to add insult to injury, remote   desktop seems broken, so you’re going to have  to deal with me pointing a camera at a screen.   In keeping with everything we’ve seen thus  far, Pinball is still disabled here, although   as one might expect, the files are present, and  after manual extraction, it does indeed work. So in short, there is no real  change from XP 2003 Edition. My next theory was that Pinball was removed on  the basis of feature disparity. Microsoft may   have wanted to have identical feature  sets across all versions of Windows,   which may have prompted  Pinball's complete removal. However, this theory doesn’t hold water  for one simple reason: Microsoft never seems concerned over the feature drift between  the Itanium, and x86-based versions of Windows. As we already saw, XP 2003 Edition  was missing the Luna theme,   and later versions of Windows were missing  even more bits. Infact, a TechNet article   shows that for Windows Server 2008, Microsoft  had nearly stripped the Itanium port bare. I did get Server 2008 R2 to install on my  zx6000, and well, I’ve always been surprised that   Microsoft shipped such a bad port at all. This  machine isn’t *officially* supported, and well,   let’s just say Aero looks really bad in  16 colors at 640x480 although I supposed   it’s excused by the fact that Aero on  Itanium only exists in the installer,   and can’t be installed via the Desktop  Experience option either. Fortunately,   it’s possible to shoe-in the XP Ethernet  drivers, so I can at least use rdesktop again. That being said, to give you an idea of how  much is missing this is, opening the Add Roles   wizard shows that the only things that are  supported are application servers, and IIS. That’s it. I think the sad state of the IA64 port  is pretty clear proof that Microsoft   didn’t care if something didn’t work on Itanium.   That being said, I’ve been weirdly tempted  to try and see if I could port PHP and MySQL,   but I probably should get back to the topic at  hand before moving onto future pain projects. However, having ruled out both 64-bit  incompatibility, and feature disparity,   then what’s left? Why was Pinball removed? Well, to answer that, we need to  enter the final chapter of this story. While Longhorn was intended as a quick followup to  XP, the project was marred with stability issues   and feature creep. Ultimately, Microsoft would  reset Longhorn’s development, eventually creating   what would become Windows Vista. Windows XP x64  would be released as a stopgap product in 2005   which represents the only official 64-bit  version of Windows that shipped with Pinball. Given Longhorn’s tortured development, part  of me had wondered though if Pinball had been   removed by mistake. It was possible  that during the development reset,   someone removed Pinball without being aware  that it was fixed on 64-bit platforms. Looking at the post-reset builds of Longhorn,  Pinball itself lasted until build 5048. Build 5048   only has AMD64 and 32-bit versions available,  but both have Pinball installed by default.   This seems to rule out the possibility of  accidental removal during the development reset.   However, there’s one thing in  particular that caught my attention. Build 5048, in addition to Pinball, also  has the same classic games as Windows XP.   For instance, Solitaire remains almost unchanged  from its XP incarnation, which itself is virtually   the same as the version from Windows 3.0.  However, Vista didn’t ship with these games. Skipping ahead to the final release of Windows  Vista, it instead shipped with revamped DirectX   versions of the games with updated graphics.  This was likely intended to demonstrate the   new compositing engine, as well as make things  look consistent with then new Aero look and feel.   This is especially evident when comparing  XP’s Solitaire side by side with Vista’s. If we load Space Cadet Pinball on Vista, it  clearly looks out of place, and it doesn’t help   that Pinball's art was intended for a 640x480  display. For Pinball to properly fit in with   the new aesthetic, it would likely involve  replacing and revamping all the graphics. This however presented a problem: Space Cadet  Pinball wasn’t written by Microsoft. It was   instead licensed from Maxis, as something of a  demonstration game for their Full Tilt Pinball   collection. My guess would be that Microsoft  didn’t either have the rights to replace the art,   or, more likely, simply didn’t want  to dedicate the resources to do so. As such, I suspect Pinball was actually axed  because it didn’t look very good on Vista. What I can say for certain is that  Pinball was fixed for 64-bit platforms,   and did infact ship on XP x64. I suspect that no  one noticed the graphical glitches, as they’re not   very evident unless you’re very familiar with how  Pinball should look. Even if they were noticed,   it would likely be a fairly straightforward  fix. As such, I’m inclined to believe that   Pinball was not cut for any technical reason  . Unfortunately, we may never know for sure. As always, I welcome you to come to the comments  section to help theorycraft. I spent literal weeks exploring and disassembling various versions of  Pinball for this video, so if you’ve enjoyed this content, please like and subscribe. If you really  enjoyed this content, consider supporting me on Patreon. With that all said, this is NCommander,  signing out, and wishing you all a pleasant day.
Info
Channel: NCommander
Views: 880,724
Rating: undefined out of 5
Keywords: ncommander, retrocomputing, space cadet pinball, windows pinball, 64-bit windows, win64, raymond chen, microsoft, michael mjd, windows xp x64, windows xp, pinball, vintage computing, itanium, epic, ia64, windows xp 64-bit edition, windows xp 2003, windows vista, longhorn, longhorn build 4051, windbg, ghidra, floating point precision, minecraft far lands, tech mystery, windows symbol files, server 2003, server 2008, hp zx6000, ipf, itanium processor family, dave's garage
Id: 3EPTfOTC4Jw
Channel Id: undefined
Length: 29min 6sec (1746 seconds)
Published: Fri Sep 03 2021
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.