If you think you need a GPU-enabled development board or SoM for your AI toaster / robot /
intergalactic space cruiser, chances are, you will be looking at the Nvidia Jetson
series. BUT. The more you look at them, the more confused you get. Which one do I choose?
Xavier? Which one? Nano? Orin? Orin Nano? Let me explain all of it to you
- no bullshit, straight to the point - and stick around
for an important update from NVIDIA GTC 2024. We'll start with the dinosaurs. Okay, not
literal dinosaurs, but rather Jetson TK, TX1, TX2 and to a certain extent Jetson Nano.
These were the first attempts of Nvidia to get into edge computing as far as 2014. Big deal at
that time, now - not as much. Floating point only calculations, no deep learning accelerators. Jetson Nano and its cousin Nano 2 Gb were a bit of a special
kids here, since Nvidia tried to capture the low-cost segment of the market. It looks like it didn't work out, as no next generation
low-cost model was developed after that. Short message from my algorithmic overlords
at YouTube: If you're enjoying this video - remember to hit the like button.
Xaviers. Made the video about Xavier NX when it was just out, mispronounced the
name. Awkward. Unlike the Nano series, which was aimed at hobbyists and students, Xavier
series was meant to replace the TX modules for industrial customers, building robots, drones
and other AI enabled devices. AGX placed a strong emphasis on automotive and was a beast at that
time, with 512-core NVIDIA Volta GPU cores and 2 NVDLA engines, delivering a total of 32 TOPS
of compute power in its maximum configuration. Also was pricey as hell. NX SoM and boards
offered a bit less compute (21 TOPS), mostly due to lesser amount of Volta cores, but also
smaller smaller footprint and lower price point. In 2022 Nvidia announced the Orin
series, which came as updated versions for the boards I talked about earlier: Orin Nano,
Orin NX and Orin AGX. They all use the new Ampere architecture GPU and deliver 5 to 8 times
more compute than the previous generation SoMs. Orin Nano 4Gb is the lowest tier in this line
up - it only has 512-core Ampere GPU and none of the Deep Learning accelerators. Orin Nano 8 Gb
- this is the version in the developer kit - not only has more RAM, but also beefier GPU - same
architecture, but double the amount of cores. Orin NX also comes in two variants: 8 Gb and 16
Gb. The main difference between them apart from RAM is the number of deep learning accelerators:
8 Gb version has one while 16 Gb version has two NVDLA v2 engines. Finally AGX Orin boards
have up to double the amount of GPU cores, depending on the configuration
- and faster GPU and CPU clocks. And what about Jetson Thor, which was unveiled at GTC 2024 at Jensen's key note speech? There really only few details available as of now we know it includes next-generation GPU, based on NVIDIA Blackwell architecture We know it's optimized for transformer inference and delivers 800 TFLOPS of 4-bit floating point performance to run multi-modal generative AI models, like GR00T. We do not know yet the price point or exact hardware specs. Which one do
you choose for your project? For the older modules - if you can get
one for free or really cheap, why not. If you are doing something for yourself, at home,
learning about embedded systems and Edge AI and building some DIY robots - it's okay still
to get Jetson Nano. It's only slightly more expensive than the latest Raspberry Pi, but it
does have a GPU for neural network acceleration and 4 Gb of RAM. The only disadvantage
really is the lack of software updates, so it is kind of stuck in the past. Xavier NX and Xavier AGX are in a bit of a gray
zone in my opinion. On the one hand they are still pretty capable, specs-wise, but Nvidia does not
put too much effort into developing new features for those boards when Orin is the main player on
the stage. So I would not recommend developing a new product with these modules - on the other
hand, if you are working on a project for yourself, as a DIY thing you need to keep in mind that Xavier NX and AGX developer kits have reached End of Life. If you are a hobbyist and you need a board with a lot of compute, get the Orin Nano Developer Kit, which is based on an 8 Gb
module. A lot of newer generative AI examples in Jetson AI lab are written and benchmarked for Orin
Nano 8 Gb. You will need NX or even AGX if you are trying to run more than one model at the same time
- or if instead of an edge device, you are trying to use it more as a server with multiple clients
processing their video footage/sound or text on Jetson. Or if you want to use for running inference with LLMs 35 - 70 B parameters, in that case you will need much more RAM and much more compute so yes, NX or AGX are recommended. Same advice pretty much goes to people testing
the SoM for product development after initial testing you might be able to find the ways to downscale your applications and use cheaper SoM, which
are usually not available as a Developer Kit. but can only be purchased as standalone SoM (see the note about 3rd party carrier boards!) Speaking about downsizing - there
are Large Language Models and there are Small Language Models. Check out the video about
benchmarking one of them on Raspberry Pi!
