If you used Windows through
the 90s or early 2000s, then these should be familiar sights and sounds. They are, of course, the welcoming chimes of
Space Cadet Pinball, a game that shipped as part of the base system for over 11 years, before being
removed with the release of Windows Vista in 2007. The closest anyone got to an explanation
as to why this beloved game was removed was when Microsoft insider, Raymond Chen, wrote a
blog post about it that set the record straight. Specifically, he states that the game was removed
due compatibility issues with 64-bit processors, and that he spent many hours trying to fix
the game before giving it up as a lost cause. On the surface, this might make sense, but what
if I were to tell you that this wasn’t the truth? In fact, what if I were to tell you that I have multiple versions of a
64-bit Pinball game that work? Well, get ready, because this is your host
NCommander, and today, we’re going to chart the final days of Space Cadet Pinball, and trace
the chain of events that lead to its removal. Our journey will take us from
the classic Windows XP desktop, to Intel’s failed Itanium architecture,
the Longhorn project, and so much more. So I recommend you all get comfortable, and
let’s wind this back to where I started, which was with a video posted
by Michael MJD a few weeks ago. If you’re not familiar with his channel, Michael
MJD did a video demoing a pre-release version of Windows XP x64. During his exploration,
he showed Pinball running just fine, which immediately got my attention, and
prompted me to start my own investigation. The easiest thing to do was to see if Pinball
had persisted to the final release of Windows XP x64. After completing the installation process,
I can say yes it did, and it’s even listed in the Games folder. I played a few test games, and
it seems to be just fine. Or well, mostly fine. If we look closely, we can see there’s some
graphical corruption. Several lights are misaligned, the left flipper is clipped, while at
the same the right flipper has gained an outline. At the same time, the ball plunger
appears to be a bit camera shy. That being said, the game appears to
play just fine. Still, this means there was indeed a 64-bit version of Pinball, and
while not perfect, it’s obviously playable. Just to confirm, I did actually load this
version of Pinball into the reverse engineering tool Ghidra, and confirmed that yes, it is
a native 64-bit version through and through. Since this is a good time to bring this up,
Microsoft actually shipped what are known as symbol files in their debug kits for almost
every version of Pinball I examined, which drastically aided my investigations. It’s also why
I have function names both here, and in WinDBG. Still, with a native 64-bit Pinball in hand, it
would seem that Raymond Chen’s post is wrong. Or, well, so it would seem Let’s actually read the relevant post together:
In it, Raymond writes, “One of the things I did in Windows XP was port several millions of
lines of code from 32-bit to 64-bit Windows so that we could ship Windows XP 64-bit Edition.
The 64-bit version of Pinball had a pretty nasty bug where the ball would simply pass through
other objects like a ghost. In particular, when you started the game, the ball would be delivered
to the launcher, and then it would slowly fall towards the bottom of the screen, through the
plunger, and out the bottom of the table.” Raymond then goes on to say that he and a
coworker tried to debug the game to no avail, and that he believed the breakage was due to an
obscure floating point rounding issue. Writing a patch was complicated by the fact that Pinball
was originally created by an outside company, and that it’s source code was incredibly messy. In the end, they made the executive decision
to cut the game instead of trying to fix it. However, I played quite games on the 64-bit version of Pinball, and I didn't encounter any sort of collision or clipping issue That means, on the surface, it would seem that
reality, and Raymond’s blog post disagrees with each other. However, this discrepancy could
potentially be cleared if we take a closer look. Specifically, the operative key words
here are “Windows XP 64-bit Edition''. In a move that would cause numerous headaches, Microsoft released multiple 64-bit versions of
Windows XP with frustratingly similar names. What we’ve been looking at was officially
called, “Windows XP Professional x64 Edition”. However, in practice, a better name might have
been “Windows XP for AMD64 Compatible Processors”, and it was the basis of the second and
ultimately successful attempt to bring 64-bit computing to the masses. Today, all
modern PC processors, even those from Intel, are AMD64 compatible, and are frequently
referred to by the term x86_64. I normally wouldn’t need to make this distinction,
but to unravel this mystery, we need to dig into Intel’s original 64-bit platform. That is to
say, we need to step into the world of the Intel Itanium architecture. Itanium was originally
intended to be a replacement for Intel’s aging x86-based line of processors, but failed to
have much impact. Honestly, the best thing I can say about it is that it provides EPIC
examples of both how not to design a processor, and how an entire industry can shoot itself in
the foot, but those are stories for another day. What’s important to know here is that Microsoft
did have a relatively short lived port of Windows XP to Itanium, known as “Windows XP 64-Bit
Edition”, and this is what Raymond Chen is likely referring to. When I started this
investigation, one of the first things I wanted to check is if Pinball was present on any
version of Windows for Itanium. As it turned out, when I checked the Windows XP 64-Bit Edition
CD, there was a surprise waiting for me. Hidden in the ia64 folder is this file called
pinball.ex_, which is a compressed installer file. Decompressing it on Windows 10 shows that it is
indeed Space Cadet Pinball, and an examination of the file shows that it’s indeed a native Itanium
version. As one might say, the mystery deepens. However, testing this binary is slightly
problematic, as Itanium binaries can’t run on normal x86 machines, nor is there
any emulator available. That however, isn’t a show stopper for me. As it so happens, I
actually own Itanium hardware, specifically, an HP zx6000, which has admitly has seen better days.
Still, this system works, or well, it did. I had to do some maintenance off camera to
get it running again. While it was in storage, it seems one of the memory sticks had failed,
and the second SCSI HDD had also checked out. I also had to replace the CR2302 battery.
Still, with some prodding it kicked to life with an EFI boot screen. Now came the “fun”
part, installing Windows XP 64-Bit Edition. My first attempt didn’t quite go to plan however.
With the Windows XP 64-Bit Edition CD in, the setup process would launch, run for a bit,
and then crash. This would then be followed by system error lights, and POST code beeps. It should be noted that while I’ve had this system for a
long time, I primarily ran Linux in those years. While I have installed Server 2003 and 2008
before, XP was a new experience for me. Well, as it turned out, I had missed something
in my research up to this point. While Windows XP 64-Bit Edition was designed
for Itanium hardware, it wasn’t designed for MY Itanium hardware. Instead, it was meant
for the first generation “Merced” systems, and is incompatible with my second generation
McKinley system. What I needed was the annoyingly difficult to Google “Windows XP 2003 Edition”,
and yes, that’s the actual product name. It’s even printed on the certificate
of authenticity slip for this system. With the XP 2003 CD inserted, things
start looking up! This time, instead of crashing the system, we’re now greeted
with a somewhat squished TXTSETUP screen. I had a few more hiccups, but I’m going to
gloss over most of the installation process, and save it for a dedicated video. Still,
in the end, I was successful! After the installation finished, we’re
greeted with a tiny login screen, a stark reminder that early versions of Windows
didn’t have great support for high DPI monitors. It’s at this point I’m going to transition using
Remote Desktop to record footage from this point out. It’s not ideal, but it’s going to make things
a lot easier to show and demonstrate. Even on the login screen we can already see quite a few
differences as compared to “stock” Windows XP, such as the revised branding. That’s not counting
things such as the Luna theme being unavailable, and the lack of fast user switching.
Infact, the entire system feels unfinished, and it honestly behaves more like
a beta build than anything else. I have to wonder if this product only exists
due to some sort of contract obligation. That being said, if we go into
the Start Menu, we can see that Pinball is nowhere to be seen. It’s not even
listed in the Windows Components wizard. A closer examination of the setup files on the CD
shows that Pinball was removed from the manifest, and thus would never be installed. That means the
existence of pinball.exe on the XP installation disk appears to be an accident, and thus
this is almost certainly a broken binary. However, I don’t need to take that on faith.
By looking at the installation manifest from other versions of Windows, I can determine all
files necessary to install Pinball by hand, and it was trivial to extract these
for testing. With a little effort, Now, assuming that Raymond Chen’s blog post
was accurate, we should have a mildly amusing glitch. Unfortunately, when I try to run it, it
simply crashes to the desktop. I actually ended up installing WinDBG and try to isolate the cause
of crash, and from what I can tell, it attempts to load several of the game resources, but then
ends up crashing with a memory access error. I had trouble finding the specific cause of
the crash, partially due to the fact that Itanium assembly code is notoriously difficult to
read. I did see that not long before the crash, the game tries to load a bunch of floating
point numbers relating to collision detection, but I’m not certain if that’s related. However,
it would fit in with what Raymond Chen wrote. It should be noted that I also managed to crash
the debugger multiple times while doing this, which should give you an idea of how
much pain and suffering I went through. Thankfully, I did eventually dig up a newer
version of WinDBG which was much more stable, but it’s never a good sign when
you need to debug your debugger. It was while I was in this debugger purgatory
that I had something of a revelation: since there were multiple
versions of XP for Itanium, it was possible that there were
multiple versions of Pinball. Sure enough, there is a compressed Pinball
binary on the XP 2003 CD we just installed from Compared side by side, we can see differences
in the file size and modification dates, but the biggest version here is that
this version actually starts up! That means we should be able to see Raymond
bug, right? I want you all to put in the comments what you think is going to happen
when I launch the ball. As a reminder, according to the blog post, the game’s
collision detector should go completely bananas. Well, that should be long enough. Let’s try it. Bet you all weren’t expecting it.
Infact, I wasn’t expecting it! I am not exaggerating when I said I spent days
trying to figure out how this was possible. It’s pretty clear though that
something isn’t adding up. First off, I want to make this clear, this
is a native Itanium version of Pinball, and not the 32-bit version in disguise. We can
clearly see that with the binary loaded in WinDBG. Furthermore, just like the version we saw on
AMD64, there are some minor graphical glitches. As far as bugs go, I also found I can’t launch
the ball while playing over Remote Desktop. I think this is a problem with how rdesktop
is handling the space bar though, and not a problem with Pinball per say. Even taking that into account I can play normally by
using the “Launch Ball” menu item. Regardless, I now have multiple 64-bit
versions of Pinball, and while not perfectly, they’re clearly serviceable. This would seem
to indicate Raymond Chen was entirely wrong on why Pinball was removed. However, as strange
as it may seem given the available evidence, I actually believe his account. I just
think there’s more to it than that, especially because I can prove part of his story. Let’s take a look at a version of
Pinball I modified slightly with Ghdira. Keep your eye on the ball. Did you catch that? I’ll
play it again in slow motion. The ball clipped through the paddle and drained.
I caused this by changing the FPU precision mode from 53 bits to 64 bits. The increase in
precision introduced the type of rounding issues that Raymond had speculated about,
and broke collision detection on the paddles. Furthermore, this FPU mode would
be similar, if not identical to how floating point numbers would behave on both
Itanium, and on the x64 builds of Windows. I think though before we go on, this
merits a bit more in-depth explanation. If you’re not familiar with
what a floating point number is, then the short version its how a computer
handles numbers that have a decimal point. The slightly longer version is that a floating
point number is an approximation that is useful for handling very large and very small
numbers. Now, I could bore you with math, but let’s go with a visual demonstration,
and for that we need Minecraft Beta 1.7.3. Now, you might be wondering why we're here ... Well, in this version of the game, if you go far enough
away from the center of the world, reality kinda breaks ... In Minecraft, the position of the player
and various in-world entities are handled through floating point numbers. For instance,
at 1 million blocks, we can see that the hit box is off center from where it should be,
and movement is no longer smooth and fluid. If we skip ahead to 2.1 million blocks out, we
can see that the precision issues have become noticeably worse. For instance, the hitbox
on the furnace is now considerably offset, and flames from the torch are no longer aligned. This is an example of how floating point
values can break down given extreme values. It should be noted that one of the most famous
glitches relating to distance in Minecraft, the Far Lands, is not a precision issue. They’re instead caused by what
is known as an integer overflow. However, since we’re now 12 million blocks away
from spawn, everything has become considerably more broken. At this distance, the hit box
appears to be drunk, and well, you can see the fire effects for yourself. Even the F3 screen
is having trouble with reporting our position. The “too long, didn’t watch” verison of all this
can be summed up as “decimal places are magic”. Now, to give you an idea of
how different precision modes change things, let’s migrate this world
to the current version of Minecraft. We can see the furnace, torch, and
Far Land chunks are still here, but things are considerably less broken. Our
hitboxes and particle effects are exactly where they should be. This is because in the intervening
versions, Minecraft changed its code to use double precision floats, which can handle
much larger values, hence the lack of jitter. Conversely, our Pinball problem could be seen
as writing code which expects the glitchy mess present in Beta 1.7.3, but is instead getting
the nice smooth behavior from Minecraft 1.17. I do hope this explains what is,
unfortunately, a non-trivial topic to grasp. However, the question is now, how did Pinball
get fixed? My first thought was that this was potentially an accident. If Pinball was fixed due
to changes in either Microsoft’s compiler or the C runtime environment, it might
be working by happenstance. Furthermore, if the XP x64 team didn’t realize
that Pinball was supposed to be broken, it might explain why it was left enabled. However,
as I continued to look deeper into it, it quickly became apparent that there were real code
changes, and this isn't some sort of accident. Over time, the pieces of a much larger puzzle
began to slot into place, and I ended up tracing the history of Pinball from its NT days, all
the way through post-reset Windows Longhorn. All my research led to one inescapable conclusion: Pinball was intentionally
fixed for 64-bit platforms. I do honestly suspect that Raymond Chen had
assumed that after Pinball got cut from Windows on Itanium, that was the end of it. We already know
that isn’t the case, but, it does beg the question I’ve been building up through all the video: just
what exactly happened to Space Cadet Pinball? We know that Pinball was removed from the original
XP for the Itanium release, and at the time, Pinball didn’t work at all. Meanwhile, a working, albeit still disabled version shipped in
XP 2003. That might seem rather strange unless one was to take a look at what else
Microsoft was doing in this time period. You see, in parallel to their work on Itanium,
Microsoft was also working on porting Windows to AMD’s new x64 platform. While this port
wouldn’t ship until 2005, the earliest known betas date to 2003, and it’s easy to assume
that work was being done well before that point. It appears Microsoft had intended to release
the AMD64 port with the next version of Windows, which was known as Longhorn. For various reasons,
this future didn’t quite come to pass, however, it can give us insight into
what Microsoft was planning. Many of the answers to this Pinball
mystery can be found in these early builds, and specifically, Windows Longhorn build 4051.
Trust me, this is where the story gets a bit wild. Build 4051 was originally compiled in late 2003, and released at the Professional
Developers Conference that same year. It’s one of the earliest known AMD64 builds
of Windows and also it’s one of the few builds that compiled all three architectures. As
such, we can compare them side by side. As I already implied, Pinball is present and works
just fine on the AMD64 version of Build 4051. Just like all other 64-bit builds, there are a few
minor graphical glitches, but it otherwise plays just fine. However, the real interesting thing
here is in the 32-bit version of this build. As one might expect, the i386 version also has
Pinball, and it shouldn’t surprise anyone that this version is flawless. However, in
an attempt to ascertain if Pinball had notably changed post-XP, I ended up doing the
same binary patch to enable 64-bit precision. This is what caused the ball clipping bug before,
but on Longhorn, it does something different … That is to say, it works just
fine. This was my first solid evidence that real code changes had been
made to fix Pinball on 64-bit Windows. In fact, this build of Longhorn tells us
quite a bit more than one might realize. It suggests that Pinball was fixed in
the process of creating the AMD64 port, and it’s likely no one realized that fixing
it for x64 also fixed it for Itanium. The IA64 version of Build 4051
leads credence to this theory. Getting it to install on my zx6000 was not a fun
experience, and to add insult to injury, remote desktop seems broken, so you’re going to have
to deal with me pointing a camera at a screen. In keeping with everything we’ve seen thus
far, Pinball is still disabled here, although as one might expect, the files are present, and
after manual extraction, it does indeed work. So in short, there is no real
change from XP 2003 Edition. My next theory was that Pinball was removed on
the basis of feature disparity. Microsoft may have wanted to have identical feature
sets across all versions of Windows, which may have prompted
Pinball's complete removal. However, this theory doesn’t hold water
for one simple reason: Microsoft never seems concerned over the feature drift between
the Itanium, and x86-based versions of Windows. As we already saw, XP 2003 Edition
was missing the Luna theme, and later versions of Windows were missing
even more bits. Infact, a TechNet article shows that for Windows Server 2008, Microsoft
had nearly stripped the Itanium port bare. I did get Server 2008 R2 to install on my
zx6000, and well, I’ve always been surprised that Microsoft shipped such a bad port at all. This
machine isn’t *officially* supported, and well, let’s just say Aero looks really bad in
16 colors at 640x480 although I supposed it’s excused by the fact that Aero on
Itanium only exists in the installer, and can’t be installed via the Desktop
Experience option either. Fortunately, it’s possible to shoe-in the XP Ethernet
drivers, so I can at least use rdesktop again. That being said, to give you an idea of how
much is missing this is, opening the Add Roles wizard shows that the only things that are
supported are application servers, and IIS. That’s it. I think the sad state of the IA64 port
is pretty clear proof that Microsoft didn’t care if something didn’t work on Itanium. That being said, I’ve been weirdly tempted
to try and see if I could port PHP and MySQL, but I probably should get back to the topic at
hand before moving onto future pain projects. However, having ruled out both 64-bit
incompatibility, and feature disparity, then what’s left? Why was Pinball removed? Well, to answer that, we need to
enter the final chapter of this story. While Longhorn was intended as a quick followup to
XP, the project was marred with stability issues and feature creep. Ultimately, Microsoft would
reset Longhorn’s development, eventually creating what would become Windows Vista. Windows XP x64
would be released as a stopgap product in 2005 which represents the only official 64-bit
version of Windows that shipped with Pinball. Given Longhorn’s tortured development, part
of me had wondered though if Pinball had been removed by mistake. It was possible
that during the development reset, someone removed Pinball without being aware
that it was fixed on 64-bit platforms. Looking at the post-reset builds of Longhorn,
Pinball itself lasted until build 5048. Build 5048 only has AMD64 and 32-bit versions available,
but both have Pinball installed by default. This seems to rule out the possibility of
accidental removal during the development reset. However, there’s one thing in
particular that caught my attention. Build 5048, in addition to Pinball, also
has the same classic games as Windows XP. For instance, Solitaire remains almost unchanged
from its XP incarnation, which itself is virtually the same as the version from Windows 3.0.
However, Vista didn’t ship with these games. Skipping ahead to the final release of Windows
Vista, it instead shipped with revamped DirectX versions of the games with updated graphics.
This was likely intended to demonstrate the new compositing engine, as well as make things
look consistent with then new Aero look and feel. This is especially evident when comparing
XP’s Solitaire side by side with Vista’s. If we load Space Cadet Pinball on Vista, it
clearly looks out of place, and it doesn’t help that Pinball's art was intended for a 640x480
display. For Pinball to properly fit in with the new aesthetic, it would likely involve
replacing and revamping all the graphics. This however presented a problem: Space Cadet
Pinball wasn’t written by Microsoft. It was instead licensed from Maxis, as something of a
demonstration game for their Full Tilt Pinball collection. My guess would be that Microsoft
didn’t either have the rights to replace the art, or, more likely, simply didn’t want
to dedicate the resources to do so. As such, I suspect Pinball was actually axed
because it didn’t look very good on Vista. What I can say for certain is that
Pinball was fixed for 64-bit platforms, and did infact ship on XP x64. I suspect that no
one noticed the graphical glitches, as they’re not very evident unless you’re very familiar with how
Pinball should look. Even if they were noticed, it would likely be a fairly straightforward
fix. As such, I’m inclined to believe that Pinball was not cut for any technical reason
. Unfortunately, we may never know for sure. As always, I welcome you to come to the comments
section to help theorycraft. I spent literal weeks exploring and disassembling various versions of
Pinball for this video, so if you’ve enjoyed this content, please like and subscribe. If you really
enjoyed this content, consider supporting me on Patreon. With that all said, this is NCommander,
signing out, and wishing you all a pleasant day.
