- And then the iOS developer says, what are our next steps? - Awful! - That's so bad. - Hey, wait!
Check out my new background. - Ohh, nice. - Very good. - Yeah, I got one too. - Oh, you got one too. - Hang on. - Nice, Dan's got the document. - Very nice. - Okay, everyone. You all look great. So let's all get ready
to go then, right? Yes? - (All agree)
We're good. - All right then. Let's do it. Virtual high-five, everyone. - Hello. I'm Dev Ittycheria,
President and CEO of MongoDB. Welcome to MongoDB Live. We are thrilled to have you join
us today and can't wait to share what we've been working on. Before we start, I want to take
a moment to acknowledge all those who have been deeply
affected by COVID-19. Being headquartered
in New York City, we have seen first-hand the
devastation inflicted by this virus in our community
and the broader economy. My thoughts and best wishes go
out to you and your loved ones during this difficult time. Working through this pandemic
has not been easy but I'm so proud of our
2000-plus employees all around the world for the dedication, perseverance, and adaptability to continue serving our customers under clearly
trying circumstances. Our employees are also
finding ways to combat the coronavirus directly. For example, our developer
advocacy team has built an open data resource in Atlas for
developers to query a critical COVID-19 data set from Johns
Hopkins University for free. Furthermore, we're offering
free credits on Atlas for anyone building apps to
detect, understand, and stop
the spread of COVID-19. So if you're working on
a project to help combat Coronavirus, we hope you find
these initiatives helpful. I also want to acknowledge
all the employees that worked tirelessly to bring together
this conference, MongoDB Live. We essentially shrunk
the work of twelve months into twelve weeks,
but I'm really pleased to say the hard work and long hours have had the desired result. The content is outstanding. And I feel quite confident
that you will leave this conference with new,
innovative ideas on how you can use MongoDB to drive your organization and your business. What's more, we have repurposed
this entire conference to be completely virtual and designed
it for a global audience. There are over 100 sessions
focused on deep dives and new products, updates to
existing products, tutorials, sessions with engineers who
actually wrote the code, and sessions with customers
using MongoDB in unique ways to transform their business. I should note that live sessions
are being offered at different times for attendees in
nearly every time zone, and every session will be
recorded with subtitles for key sessions in nine
different languages. So no matter where you are,
all the content provided at this conference
will be incredibly useful and is available on demand. At MongoDB, our mission is to
free the genius within everyone by making data stunningly
easy to work with. It's clear that the future of
every business, large or small, will be powered by software. Every company must engage
with their customers, partners, suppliers,
through modern, high quality digital applications,
built on a highly flexible scalable and cost-effective architecture. As a result, our focus in
solving developers' data problems increases in
relevance every day. We've been fortunate to have
a very large community of developers who love MongoDB, due
to the way MongoDB's document model and query language make
it so easy to work with data. Our community server
has been downloaded over 100 million times. To put this number
in perspective, there are approximately 25
million developers in the world today, and Stack Overflow's
comprehensive worldwide developer survey recognized
MongoDB as the most wanted database by developers
on a worldwide basis three years in a row. We also now have more
than 18,000 customers, from the largest companies
in the world to cutting edge startups, in more than 100
countries across nearly every vertical industry,
using MongoDB for almost every conceivable use case. We've also seen
customers embrace Atlas, our global cloud database, which
is the most widely available cloud database, as it's
available on the three largest cloud providers:
AWS, Azure and GCP. Today we have over 2 million
free tier accounts on Atlas, and Atlas is the fastest
growing part of our business. We are truly humbled by how
popular MongoDB has become, and how large the MongoDB
community is today. While we're pleased
with how far we've come, our appetite to do more
has only increased. Applications of the future will
dramatically expand in their functionality and scope. Future applications will enable
continuous engagement and access to massive amounts of real-time
data among key constituents, be it users, customers,
partners, or suppliers. The availability of
instantaneous operational data, along with the
insight of that data, will increasingly drive
business decisions. This means that traditional
operational workloads will need to incorporate
new functionality, such as real time analytics. In addition, as data on
the edge continues to explode due to the increasing
use of mobile and IoT, future applications will require
the ability to effortlessly synchronize data between
the edge and the core. So what you'll hear over
the next 90 minutes will show you how MongoDB evolved
from a database company to a data platform company
to address these needs. To lead this discussion, I'm
pleased to introduce Sahir Azam, our Chief Product Officer,
along with members of our engineering team to
showcase what we have built. Sahir? - Hey Dev. - Are you ready to
tell everyone about what we're announcing today? - We can't wait, Dev. - Great! I hope you enjoy
the conference and all the content we have for you,
and that you are as excited as we are about the future
that we can build together. - Thanks, Dev, and thank
you all for joining us. like Dev said, our mission
is to make data stunningly easy to work with. Why? Because data is the foundation of every application. It was the ability to harness
data that powered many of the technology revolutions
of the past decade. But, as the prevalence of
data-driven applications has grown, so have the challenges
to leveraging all that data. Eleven years ago, our founders
saw that the biggest issue facing developers
was the database. The databases of the time
were a poor fit for this new breed of applications,
which needed to deal with enormous amounts of
data in many formats from disparate sources and adapt quickly to compete in an increasingly
demanding marketplace. Traditional relational
offerings were rigid, fragile, and couldn't scale horizontally,
making them slow to develop on and expensive to maintain. They set out to build
a new, open source, general purpose database
built on three core pillars. First, the document model. The document model makes
working with data easy, because it's flexible, suited
to a broad set of use cases, and maps well to the way
developers work in modern, object-oriented
programming languages. Second, distributed systems. Horizontal scale, redundancy
and workload isolation should be table stakes for databases. So a distributed
architecture was necessary. And finally,
the ability to run anywhere. So you can start
developing on your laptop, run it in your
corporate data center, or manage it
in a public cloud. Of course, traditional
databases did offer capabilities vital to mission
critical applications. We made sure to
incorporate those, like transactional guarantees,
secondary indexing, powerful aggregation
capabilities, and security and
management tooling. The combination of these
capabilities now allows MongoDB the ability to power any
application at any scale. And it's why MongoDB is
now the database of choice from millions of developers
and tens of thousands of businesses
across the world. Modern applications also
require modern infrastructure, so we built MongoDB Atlas,
an elastic MongoDB cloud service that runs in over
70 regions across every major cloud provider. MongoDB Atlas gets you up
and running in minutes, seamlessly self-manages scale,
and keeps itself up to date. Atlas is programmable,
integrates with popular infrastructure management
tooling, monitors your usage, and advises you of
performance issues. Atlas offers entirely unique
capabilities like global clusters, which allow you to
scale worldwide with the flip of a switch, and client-side,
field-level encryption, giving you fine-grained control
over your most sensitive data. Today we're going to
show you that MongoDB Atlas isn't just a cloud database. We'll show you how, along with
some astounding new features we're announcing today,
MongoDB Atlas is the data foundation for
modern applications. But first, we have to talk about
something that's been troubling us more and more
these past few years. You see, we're seeing the same
pain of working with data making its way into every part
of the modern tech stack, and we're tracing that pain
to the same causes that our founders set up to eliminate
when they started MongoDB. Let's look at a typical
modern application. It starts with
cloud infrastructure, which means flexibility, pay as
you go pricing, quick startup, and low operational overhead. That's a huge step up from where
the industry was a decade ago. The heart of any
application is an online transactional database. Things like log data get
archived to cheap storage or a data lake, and in order
to study that data, there are analytic systems
or data warehouses, which require ETL
processes to load. Search is now a basic necessity
for any modern application requiring its own
specialized database. Let's call all of that together
"the data tier." There's a middle tier for business logic,
generally composed of micro services that are managed
by independent teams backed by independent databases. Explicitly coordinating
these would be a mess, so event streaming technologies
give this service as a way to obtain and supply
the data they work with. A set of APIs and web services
go along with each microservice to operate and scale them. There's a lot of data
movement in the middle tier, but that's not where
it begins its life. The ever-rising tide of data
originates from the front end, meaning web-based or
mobile client applications, and increasingly, sensors
in the world of IoT. Integrating front-end data is
the last mile problem of the modern application, currently
requiring clients to coordinate independently with the many
back-end services they need. And since the benefit you offer
users generally means getting data to them, this is a
bi-directional challenge. Finally, a modern application
handles visualization using a dedicated business
intelligence system. Now, this provides a lot more
flexibility in scale than the monolithic apps of the 2000s,
but there's actually a ton of pain in these architectures. Each one of these components
represents code you have to write and maintain,
and none of it is what you really want to
spend your resources on. It's all basically plumbing
to get data from place to place and format to format. The data architecture of a
generic cloud application looks a lot like the legacy
data architecture, only the ease of spinning up
new services and integrations ultimately creates a snowball
of complexity that's hard to reason about, with conflicting
representations of the same data and a large
surface area to secure. These are familiar
problems to MongoDB. It's obvious to us that
this complexity is the result of the same old data
challenges masquerading as best practices in cloud application architecture, creating cognitive
overhead, wasting money, and slowing down your
ability to build amazing differentiated apps. And what do
we do at MongoDB? We make data stunningly
easy to work with. We're fixing this data
architecture problem. We're giving developers
a new holistic data platform for building applications that relieves them of unnecessary plumbing. It's called MongoDB Cloud, and
its architecture seamlessly spans the data core,
the middle tier, and all the way
to the front end. Data in the MongoDB Cloud
platform goes where it needs to go, automatically and securely,
and it doesn't make you work to groom it for
specialized systems. MongoDB Cloud is open and
provides seamless integration point to the critical
services you need to build your applications. MongoDB Cloud consists
of three components. MongoDB Atlas, our
global cloud database, gives developers a single
consistent data foundation to build on. Atlas clusters handle
operational and real-time analytics tasks
on the same data, while global clusters
keep data close to the users it belongs to, automatically. Atlas search applies the power
of Lucene's full text search to the same data seamlessly,
and archival storage and advanced analytics is alive again through Atlas Data Lake. You use the power and
expressiveness of MQL, the MongoDB query
language, for all of it. Search and Data Lake were
introduced last year, and today you'll see how they,
together with some incredible new components, form a powerful
integrated data foundation for your application. MongoDB Charts is document
native data visualization, connected directly
to your Atlas data, and available
anywhere it's needed. Whether that's in a dashboard
for business intelligence needs, embedded publicly
in a blog post, or built directly into an
application and securely showing users personalized data,
MongoDB Charts makes it easy to build live, responsive
charts that look beautiful and bring
insight from your data. I can't wait to show you how
the latest updates to Charts, including an even better way
to embed the power of MongoDB Charts into your applications. MongoDB Realm automates
and simplifies the way you leverage data across all
the layers of an application. It combines the Realm mobile
database and sync capabilities, which we acquired last year,
with the server list platform formerly known as
MongoDB Stitch, to create an entirely new
way of building applications, without the tedious
work of getting data where it needs to be. MongoDB Realm makes it trivial
to regulate access to data, synchronize it between devices,
act on it in real time, and integrate it into all
the services you build into the rest of your application, minimizing the need to translate it from format to format. We announced our road map to
unify MongoDB Realm a year ago, and today we're going to
amaze you with what it can do. Atlas, Charts and Realm,
the three components of MongoDB Cloud. Today we're going to show you
exactly how to use this unified data platform to build
applications without the tedium of shuttling data
anywhere in your stack. We'll also preview some of the
great new features coming up in the next release of MongoDB,
and the latest updates with operations and new integrations. At MongoDB, we love to
demonstrate what we've built in our key notes, but the release
of a whole new way to build applications requires something
a little more substantial than just a handful of demos. This year we decided
to build a complete, working reference application
that shows exactly how MongoDB Cloud can be used to build
applications that focus entirely on their differentiating value,
without getting bogged down in tedious plumbing. But we couldn't build something
that was essentially a feature checklist, only an application
that solves a real-world problem can really prove our claims. And since we have to solve
a real-world problem, we thought it was best to solve
one that has global impact. WildAid is a great champion of
conservation efforts and the marine division is fiercely
committed to protecting Earth's largest most important
habitat, the ocean. We're incredibly proud to
be involved in their effort, and I'd love to welcome
the head of the program, Meagan Brosnan to
tell you about it. - Thanks, Sahir, it's
great to be here. We believe the world deserves
abundant, healthy oceans, where wildlife is
safe from poaching, and where coastal
communities thrive. To achieve this, we are
building the world's most effective and well enforced marine protected areas, or MPAs. My team created a video
to tell you more and I'd love
to show it to you. - I'd love to see it. - [Music] "50 years ago,
we thought the ocean was indestructible. Today, we have lost half
of the world's marine life, thousands of miles of coastal
ecosystems have been destroyed, and climate change has triggered
a vast die-off of coral reefs. The ocean can recover, but
we need to protect it, now. Like national parks, marine
protected areas, or MPAs, are one of the best conservation
tools we have for our oceans. Although more and more
MPAs are being created, nearly 60% of them are
not well-protected, meaning they're little
more than lines on a map, or paper parks that lack the
resources and capacity to provide real and lasting
protection for marine wildlife. WildAid Marine is
working to change this. Using our comprehensive
blueprint for MPA success, WildAid works with
local partners, guiding them to
the best solutions. WildAid Marine's experience in
creating effective enforcement is unparalleled, and we are
now working with partners all over the world to
implement our model, creating regional centers
of excellence globally. With your help, we can reach our
ambitious goal of strengthening enforcement in 250 MPAs
in the next five years. Together, we can deliver the
protection our ocean needs." - Meagan, we love
what WildAid is doing. - Thanks. It's a true privilege to support
the people who are on the water protecting our oceans every day,
and in order to accomplish that mission, it's important that
they have tools that are easy to use and reliable. Those partners are facing
some challenges with... let me see if I
get this right... digital transformation? - Yeah, there's
some industry-speak. - Great! So a couple of members of my team, Bob and Karima, are going to explain why
that transformation has been so challenging. Bob is a former Fisheries
Enforcement Officer and currently an Enforcement
Consultant for WildAid, and Karima is a Project
Manager for WildAid. - The job of a fishery's
officer is challenging. We literally jump from one boat
to another and we don't always know what will happen when
we get on a fishing boat. Safety is a big concern for us. After boarding, I collect
information to make sure they're operating by the rules. As I collect all the information
on things like proper licensing, verifying the catch,
I usually have to write things down on
a piece of paper. I might even make notes
on the back of my hand. - Once all these forms and wet
scraps of paper get back to the headquarters, we have to
decipher the smudges and messy handwriting, and put it into
some sort of usable format. This can be very difficult,
and it takes a lot of time. We'd rather spend our
time making an impact, not doing data entry. - Officers don't have access to
that information, or if we do, it's months later. It would really be helpful
if we could know how dangerous boarding a vessel will
be before we board it. Most of the existing
digital tools we have tried are focused on compiling
the data back at the office. We want to give
the officers information they need in real time. - Digital record keeping allows
us to have metrics on what's effective, and we can visualize
it and draw conclusions. We can identify areas
that have a lot of activity and send officers there,
instead of wasting fuel looking for vessels
that aren't there. - We've looked into
some existing applications, but we have specific needs
that are not met by them. We wanted an app
that solves all of our needs. - And of course,
we need all our requirements met on a
not-for-profit budget. We have a goal of protecting 250
Marine Protected Areas by 2025; that's a scale of almost
five times more than we do now in only five years. We can't scale like that
if we have to pay for multiple software licenses. We need the funding from
our donors to be allocated to making a greater impact. The flexibility of the MongoDB
Cloud platform is going to let us comply more
easily with the regulations of all the
countries we work in. Also, a lot of
the areas where we work, simply don't have robust
technological resources, so our tools have to be
simple and easy to deploy. By any name, digital
transformation is exactly what WildAid is pursuing for
MPA enforcement agencies. But it's immediately clear
that this is a tall order. You're probably imagining
all the components already; offline capable mobile
applications for use in the field, some way to
sync that data with the back end and all the other
mobile apps, and a back office to report on and
visualize it, for starters. The data model has to be
flexible enough to handle a variety of areas,
and the infrastructure needs to be as simple and cheap
to manage as possible. This is the perfect proving
ground for MongoDB Cloud. For the past few months,
we've been using it to develop a solution for WildAid, and now we're going to pull it apart
and show you how it works. MongoDB Realm simplifies the
challenges of working with data across your application. Declarative access control
rules regulate access to data. Mobile Sync moves data between
devices and an Atlas cluster, handling offline access
without complex networking and conflict resolution code. You can act on it in real time
with Serverless Functions, where you can integrate
with all the services you need to build an application. And GraphQL gives developers
a single endpoint to access exactly the data they need,
nothing more, nothing less. Alexander, as a
founder of Realm, can you give us a
quick review of how it fits into MongoDB Cloud? - Absolutely. So much of today's data
problems are, as you said, just around getting the data
to where it needs to be. MongoDB Cloud
makes this easy. Realm allows you to have
an actual database on mobile devices, which seamlessly
synchronize with the data in your Atlas cluster. This makes cross-platform
development so much simpler, while opening up for
powerful new capabilities. Having the data on the
device makes everything fast, allowing local access
with zero latency, and it ensures that your
app works even when offline or when there's bad connectivity, which happens all the time
for mobile apps. And since sync happens
transparently in the background, you're saving yourself from
writing tons of error-prone networking code. Also since the Realm database
is a real object database, it fits right into all the
existing mobile frameworks, accelerating your development
and removing the need for all the boilerplate code you
would otherwise have to write. On top of that, Realm now
includes all the functionality formerly available
in MongoDB Stitch, so you also get authentication,
serverless functions, triggers, and everything else you need for
a fully serverless experience. - MongoDB Realm extends
the reach of the Atlas data foundation all the way
across the application stack. For the enforcement
agents WildAid is helping, we could quickly build highly
performant cross platform mobile apps that sync in real time
and work even when offline. - Let's get a look at
the basic functionality, recording the outcomes
of vessel inspections, and syncing
that data to Atlas. Bob and Karima are going to
show this off now, right? - Yep, ready when you are. - And Aaron, you'll chime
in to show us how everything is built, right? - Gladly. - When we're out on patrol,
we'll select the vessel for possible inspection. With the app on my iPhone,
I can search for previous boardings by me
or any of the other officers. What I'm looking at now,
I can find the most recent boardings are here, but I'm looking for a boat called "Predator," which isn't here. Looks like it was boarded
about a month ago, let's see the details. Looks like they have
no fishing license, now I know what
I'm getting into. Now I jump on the boat. The app was built to match
the officer's workflow. As I go through my inspection
filling out the details about the vessel and crew, the
activities they're engaged in, I can add details about
the catch they have on board, and also note if they
had any violations or if there was a seizure. I can also add an assessment
of their risk level. There were no violations, so
I'm going to keep this as green. As I inspect everything,
I can also make notes. I'm going to write "the crew
was friendly." I can also take pictures if I see
anything worth photographing. When I'm done, I submit my
report and the app saves all the information. If there was a problem
with the boarding, like a safety or
fisheries' violation, the next officer will know
before they get on that boat. - I'm looking for
the record on my Android tablet and it's not there. - Oh right. When I'm in an area
with no service, I put my phone in airplane mode. Then it will avoid
searching for a network and draining my battery. The app allows it to submit
records even when we're out at sea and offline. - Oh, got it. That makes sense. - Okay. Let me leave this boat
and come back in range. - Hey, here it is! - And if I run this query,
we can see the record is in Atlas, too. - This is fantastic! Instead of a backlog of paper
that needs to be inputted, we have the officers'
reports before they even return to shore. - Love to see it! Aaron, that must have
been really hard, right? Maybe a few hundred
lines of code? - Actually it's super easy,
barely an inconvenience. - Oh, really? - Well, it used to
be really complex. I have to code multi-tenant,
offline-online sync, with conflict resolution. But with Realm, it
amounts to--in fact, let me show you the code. These lines here, that's all. That's sync. It's almost as if the app's
code is devoted to, you know, the app's code. - Three simple lines
of code to implement bi-directional sync
between a mobile device and MongoDB data
in an Atlas cluster. It's amazing. If you build synchronization
logic without Realm, you have to build something
like you see in this diagram. And that really doesn't even
cover the intricate networking code we need to write. We replaced this complex
architecture with MongoDB Realm. Realm Sync is now
generally available, and this one feature powers
a lot of capabilities in an application. Like backup, cross-device
and cross-platform sync, and offline capabilities that
allow you to continue using an app even when not connected,
and automatically merge any changes back to Atlas
once you're back online. And if you develop for
both Android and iOS, you know that it's not easy
to sync data between the two. Because of how hard
it is, a lot of apps, even if they're available
in iOS and Android, don't have a way to sync
data across platforms. Unfortunately, that can
lead to bad user experience. If a user wants to switch
from one platform to another, they'll have to face
losing all of their data. MongoDB Realm turns
this into a non-issue, because syncing data across
platforms is trivial. That means happier
users for you. - You just saw how easy
it is to extend the reach of your data platform
to mobile devices and forget about writing a
bunch of data sync code. Aaron showed how nearly every
line of the code for an app is devoted purely to implementing
user-facing features. That was the warmup,
now it's time to see what it really means to have
a unified data platform, taking data routing
and manipulation responsibilities
off of your hands. I told you how these
responsibilities are hiding there in plain sight, right? So let's look more deeply
at one of the features in our app that you probably
didn't spend a ton of time thinking about when
we first showed it to you. Bob, what do you take pictures
of during inspections? -We take pictures of the vessel
so we can identify it the next time we board it, and we take
pictures of the captain or crew, and whenever we need evidence. The app allows us to attach
pictures to almost every field. The more pictures we have,
the better informed we can be. - Makes sense. So you take plenty of pictures,
but most of the time when you need to see those images,
they're coming from inspections that you didn't do
yourself, right? - That's correct. When I'm preparing
to board a boat, I might want to review
images from inspections that the other officers conducted. - Ingesting images and making
them available to client-side applications has really been a
table stakes feature for a lot of applications for decades,
and it's a great example of a seemingly simple feature
that has to move data through a bunch of different layers. Exactly the kind of plumbing
MongoDB Realm is built to automate, Alexander, take
us through this, okay? - You got it. So, getting images from
users, storing them, and then serving them back at a
variety of sizes from thumbnail to full resolution is about
as common of feature as an application can have. And back before
the cloud took over, that was a thing everyone
had to deal with themselves. Although a content delivery
network could take some of the sting out of it,
you still have to write the code that stored,
resized, and served images. - I mean, this is the
epitome of comodo code. It's a thin logic wrapper or
an image processing library, with an API for talking to it. - These days you'd never
dream of writing your own infrastructure for
something like that. For one thing, storing
and serving image files is a job that cloud optic
storage is perfect for. Only, as you'll see in a moment,
using cloud infrastructure and optic storage in
2020 looks barely different to a developer than
writing an image server and using a CDN
looked back in 2000. - This is an example of
architecture using cloud infrastructure and object
storage for ingesting images. We have a few calls in API, the
image metadata goes to database, and there's an image
processing job that needs to be coordinated, and afterwards
the image is stored in cloud storage, and the database entry
for it needs to be updated. Once the full-res
image is stored in S3, if only to delete it,
and only keep a thumbnail on a reference density object, yet another process. All the while your image service
needs to make sure it's only taking legitimate requests,
so you must be mindful of security concerns
and ensure the uploads are only coming
from legitimate users. So this is a complex, multi-step
process involving the issuance of temporary credentials. - The crazy thing about
this is that it's a whole system of this application
we have to write and maintain. And whether it's image
servers or image service, it's still added complexity. Fundamentally this is just
another data prompting responsibility pretending to
be a feature of an application. And now, Aaron's going to show
you what's basically a magic trick, because using
the power of MongoDB Cloud, he's going to make all
that complexity disappear. - So the app's diagram
stays the same, but most of the other server
stuff is going to be replaced with Realm, which will also
handle moving the image to S3, and all the other housekeeping. We still have the phone
that takes the photo and stores it in
the imbedded Realm database. This works, whether we have
internet connection or not. WildAid's boats are often
operating miles offshore, and so officers can't rely on
having a stable connection. Once the device does have
an internet connection, Realm Sync takes care
of getting the photo from the device to Atlas. We're not planning on letting
it stay there, though, as you'll see in a minute. Once the image is in Atlas, the
next step is to move it to S3. This process is
easy to automate, by adding an Atlas
trigger to detect when a new photo is inserted. Let me show you the code. This is a trigger. Whenever a new photo appears
in Atlas, it calls a function, "uploadImageToS3," which
stores the image in S3. Once that function returns with
the URL of the image from S3, it adds the URL to
the Atlas document and deletes the binary
image in the document. The Atlas document automatically
syncs back to the mobile device, freeing up storage space. And this is the Realm
function the trigger called. MongoDB Realm has built in
integration with AWS services, like S3, and so the function to
put the image in the S3 bucket is pretty short. Overall, it's really not a lot. - We've been able to work
with the images natively on the mobile platforms, relying on
Realm Mobile Sync with its built-in authentication and
access control to get the image data into Atlas without needing
to implement sync and deal with offline edge cases. Realm's triggers let us
react the instant new images appeared in Atlas,
and serverless functions, which now have the ability
to include library code with dependents in resolution,
gave us an integration point to move the images to Amazon S3. The flexibility of the document
model lets us replace binary image data with S3 URL
references in the same document, which means that Realm Mobile
Sync also took care of purging the unnecessary binary data
from the mobile devices, while keeping the client
code drop dead simple. - One more thing, if I want to
use an MPM package inside one of these functions,
that's easily possible with dependencies for
Atlas functions. I just upload it here,
and use it in my function. - Very cool. So we've extended the reach
of our data foundation to the mobile environment. But the picture isn't complete
until we've done the same for web applications and figured
out how to easily expose data to external consumers. For something like that, MongoDB
Realm incudes a GraphQL service. This makes it easy to get
up and running with GraphQL, the groundbreaking API query
language that gives developers a single end point to access
exactly the data they need. Nothing more, nothing less. - Lucky for us, Realm allows
us to automatically set up a GraphQL API close-expose
the data we want to share. Here I've created a
second Realm app and pointed it at the same cluster. Over here, I can
generate a schema, and in the GraphQL section,
I can dial in the schema from my application. And I can even test
out my GraphQL interface in this explorer. And just like that, see,
I can access the data. And because it's all in Realm,
I can rest assured knowing the Realm rules I have in place
will control data authorization. Nobody will see any data
they're not authorized to. - Just like the Realm Mobile
Database and Mobile Sync gives the mobile developers
the best way to work with data on a mobile device,
GraphQL gives web developers the best way to work with
data in a web application. On a last note, there's
a lot more that has happened on the Realm
Mobile Database site. We've really doubled down on
improving the database itself, with the release of an
all-new storage engine, improved query capabilities,
and Raspberry Pi release for IoT use cases, open sourcing
the sync client, and much more. So check it out. What do you think
about that, Sahir? - That was great!
Thank you to the team that has been working
and will continue to work on the WildAid app. Thank you also to our partners
at WildAid for giving us the opportunity to work
on this really special mission with them. It's been a pleasure
working with you. - It has, indeed. By the way, if you want
to program for good, and learn about how
to build apps as well, then you should join
us and contribute. The app is open source
and available in GitHub. Join us in saving our oceans
with WildAid and get to know a whole new way to develop
applications with MongoDB Realm. - And thank you
so much for joining us in protecting our oceans. The combination of this open
source application and the MongoDB Cloud platform means
we can use the resources of our generous donors
to make a much bigger impact. Furthermore. there are over 15,000
MPAs in the world, and honestly WildAid will never
be able to support all of them. But by making this
application open source, we are making the tool
available to every single enforcement agency
that needs it. Turtles and sharks
don't care about the borders we've drawn on maps. Coastal fishing communities
don't care if we have an existing project
in their country. The better the enforcement,
the safer our seas. - Thanks so much for joining
us, Meagan, Karima and Bob. We're looking forward
to getting this app into 250 plus Marine
Protected Areas by 2025. - So are we. - MongoDB Atlas, our global
cloud database service, has come a long way
since its launch in 2016. Walking every step of that
journey has been our EVP of Cloud Engineering,
Cailin Nelson. - Hi, Sahir! - Hi, Cailin. We couldn't be more excited
about the present and future of Atlas, and there's nobody better
to tell us about it than you. And Senior Developer
Advocate, Adrienne Tacke, will show you some of our
latest improvements in action. - Thanks, Sahir. I'd say I can't believe it's
been four years, but honestly, so much has changed. As much as we care about scale,
we care even more about offering the most
frictionless development experience of any database. How do we take all the manual
work out of running MongoDB? This has been a core goal
of Atlas since its inception, and we're continuing to
build on those foundations. For starters,
Atlas completely automates minor version upgrades,
so you never have to think about bug fixes
and security patches. Second, we thought databases
grow by default and storage is pretty cheap, you shouldn't have
to think about keeping up with the size of your data on
disk, so that's automatic too. Finally, anyone who's worked
with a growing data set knows that performance
is always changing, and you shouldn't have
to dig through your logs to figure out how to tune
your system performance. That's why we built
the Performance Advisor. We continually analyze query
performance, suggest indexes, and make it push button
easy to add them. In fact, can we show
the recent improvements we've made to index suggestions? - Hi, Cailin. I'd be glad to help
out and show this. Our friends at WildAid connected
us with another organization called Saildrone, whose fleet
of drones are equipped with atmospheric and
oceanographic sensors. Saildrone's mission is to create
the highest resolution ocean data set in the world and use
it to make global processes, such as weather
forecasting, carbon cycling, global fishing and climate
change, more predictable, visible, and actionable. They've been kind enough to
share their sensor data with us. We uploaded the data and started
to run some ad hoc queries. After several rounds
of testing we, found the queries
we want to use, and now we want to
know what indexes we should create for them. Performance Advisor analyzes the
existing workload on the data, and surfaces suggestions
based on the workload. Index suggestions are
now displayed and ranked by anticipated impact. Our top index suggestion advises
us to create an index on the time UTC field, which would
have an impact of over 9%. The metrics shown help
explain why performance advisor thinks in index is valuable. Knowing the indexes that are
already on the collection helps us decide if this index should
replace an existing index, or if the field could be
added to an existing index. Here's some examples
of the queries that triggered this suggestion,
so we know exactly which queries will be
helped by the index. If you need to change the query,
instead of adding an index, this sample will
help you pinpoint it. I think it would be a
good idea to make an index on the time UTC field. We can inspect the actual
"create index" command that will be sent to the servers
before issuing the command. We can even build it
right here with no downtime using a rolling index build. And there we go. The index is being created,
all with just a few clicks right in the UI. - We think the updated Atlas
Performance Advisor is going to be a really big help in getting
the most from your MongoDB experience, but we're
just getting started. If we could automate storage
resources and we can automate cluster management, then why
not automate scaling entirely? With the general availability
of compute autoscaling, Atlas can monitor workload
performance and automatically scale cluster resources up or
down to match changes in demand. No more having to watch the
dashboards and decide when to resize, and you can rest easy
knowing that your cluster will scale back down, too,
so you're not wasting money just to handle your peak load. - Let's go into the cluster
configuration settings. Storage auto scaling? Check. Cluster tier auto scaling? Let's turn it on. Notice you can manually
set upper and lower limits. If this is turned on,
the default minimum cluster size is an M10 and I will
allow my M20 to go as low as an M10. On the higher end,
I will limit the cluster to go only as high as an M40. That should be enough to
cover even high loads. Atlas is intentionally much more
conservative with downscaling. We don't want your cluster
shrinking due to a temporary lull in traffic,
which could cause a bottleneck if load picks back up. And that's it. - We built MongoDB to make
building with data easier, faster, and frankly, more fun. We've been overwhelmed by
this positive response. For three years running,
developers have voted MongoDB "the most wanted database,"
according to Stack Overflow's annual survey, and more people
are trying out MongoDB for the first time today
than ever before. Most people coming
to MongoDB are used to the "rows and columns"
mindset, and for them it's easy to skip
past schema design. And anyone who scaled MongoDB
before will tell you that your schema is just as important
in a document database, even though it's
intentionally flexible. That's why we have one more
feature to show you: Automated Schema Suggestions,
available in Atlas today. Let's take a look. - In the format we got it in,
the Saildrone data is flat, and we wanted a
richer experience. In this collection, we modified
the data so that each document represents the data
from one drone. One document represents all the
sensor data collected by one drone, The drone's identifier
is the field called "trajectory," but it's
actually just a label, not where sensor
or GPS data goes. Each drone sensor data is
stored in a data array. This record has a few
items in the data array. Now that we have
our new collection, Atlas may suggest ways to
make our collection better. The Performance Advisor, which
we used earlier to get index suggestions, also has a
Schema Suggestions tab. Yikes, looks like
the scheme advisor says we have some really large arrays. In fact, we have some arrays
with more than 10,000 entries. That's not good, let's see
what's going on with our data. Oh, I can see that this document
has lots of array records. We grouped the flat data
on a per drone basis, but that's led to over
10,000 array entries. Let's head back to our
schema suggestions in Atlas. Atlas not only brings issues
like these to our attention, it also helps us fix them, so
to fix this particular issue, we'll have to think about
adjusting our data model for our Saildrone data. Now I have a great start
in the right direction. I'll have to think about how
to group this data differently, so that there's a more
reasonable number of array entries. - Whether you're just starting
out, just reaching large scale, or already an expert
looking to improve, Schema Suggestions helps
optimize the structure of your data for optimal
performance and usability. - In MongoDB Atlas, we've worked
on a lot of amazing features, as well as improvements
regarding performance, resilience and reliability. Of course, our global cloud
database is built on the world's best modern general-purpose
database, MongoDB. The power of the document
model and the MongoDB query language led to the inception
of the post-relational movement. It drove massive worldwide
adoption to MongoDB and spawned a
field of imitators. Just like Cailin, EVP of Core
Engineering, Dan Pasette, has been part of
that entire journey. - That's true, so here
MongoDB Server is reaching version 4.4 this summer. The beta is available
for evaluation now. - We developed version 4.4 with
a laser focus on the things you've been asking for; easier
scaling, faster performance, and more flexibility to
solve even more use cases. Let's start with scaling. The minimum production ready
cluster you can deploy in MongoDB Atlas has
his three nodes. Over time, your
cluster might grow, especially if your
business is going well. - As your workload grows,
there are some important choices you have to make. Essentially, you need to
decideabout the scaling model of the distributed
system for your database. - With MongoDB Atlas, we aim to
make all of this easy for you. You can move between
cloud providers, seamlessly scale your clusters,
and use global clusters to distribute your
load geographically, in order to keep data close
to your users and prevent it from leaving
the region if necessary. All of this is great, but
there's one thing we have to talk about at the server level. - Sharding, the best
way of scaling data. MongoDB gives you
flexibility to choose exactly how you want to scale. This means that you might
need to make some choices, and the most important choice
is choosing the right shard key. Let me quickly recap
what a shard key is. A shard key determines
the distribution of the collections documents
among the cluster shards. In MongoDB, the shard
key is an indexed field or group of fields. The ideal shard key allows
MongoDB to distribute documents evenly throughout the cluster. Let's say your data has
a customer ID field. You may, for example, choose to
split your data onto different shards based on
that customer ID. Say you have two shards, and
you randomly allocate a number between one and 10,000
as your customer ID. Behind the scenes, different
ranges of customer IDs will be allocated to each shard,
and this might work just fine as you get started. There's a lot of pressure to
get the selection of shard key exactly right, and not just
right for your current use case and workload, but also
right for for the future, regardless of how much you
may scale or how your data or workload may change. Let's return to our example with
the shard key "customer ID." Now it could happen
that as you grow, some customers
amass a lot of data, which would have to be
stored all on the same shard, meaning we cannot split
their data over more shards. This can lead to uneven load,
putting more pressure on just a single shard. Now, there was a way
to fix this beforehand, but it required
quite a bit of effort. You probably guessed
where I'm going with this. Yes, we've solved
this problem for you. You can now refine your
shard key at any time, allowing you to adapt data
distribution across cluster as your database grows
and your application evolves. You still need to think
about your shard key, but you don't need
to plan for all possible future needs up front. With refinable shard keys,
you have the flexibility to evolve data distribution as
your requirements change, all without downtime. Now, getting back to our example
with the customer ID shard key. You can now add the order ID as
a second part of the shard key. This allows the
big chunks of data, that each belong to
a single customer, to be split and moved
to other shards, leading to a more even
load across the cluster. So far, so good. We may still have a
problem with this, though. Let's say the order IDs are in
monotonically increasing order. Maybe there's a customer that
is a big online retailer, that places orders
with the platform as they come into that retailer. If a lot of orders come
in for a big customer, they will all go to the same
shard, causing uneven workload. The answer here is
hashing the order ID. You could do this in
the application before, but now the database
can also handle it for you with support for
compound hashed shard keys. This will ensure a
more evenly distributed load across your shards. - Sharding can improve the end
user experience but mainly, these improvements focus on
making scaling easier for you. The next improvements are all
about ensuring your users' experience is stable
and performant. - If a user performs an
action in your application, the response should be
as quick as possible, and that's where we've driven
some real innovation in 4.4. Up until version 4.2, if a
query was sent to MongoDB, one node would be responsible
for sending the answer. However, if this node was
busy with something else, maybe a hardware issue
or disk slow down, the user could see
higher latency. In 4.4, we minimize P-95
and P-99 long tail latency by dispatching queries to multiple members of a replica set where possible,
and returning the result from the quickest
responding node. We also
worked on resilience. This is to reduce user impact
due to replica set elections after a node goes down,
whether due to planned maintenance or
unplanned failures. We do that with the
help of mirrored reads. Mirrored reads keep the caches
of electable secondaries warm by continuously mirroring a
subset of reads to them. Now if the primary
then goes down, the secondary that takes over
as the new primary will already have a warmed-up cache,
reducing the impact of the failover
on your customers. - These are exciting changes
that will make MongoDB even more performant,
resilient and reliable. Let's look at the
query language now. As you know, MongoDB
allows for natural and flexible data modeling. With MongoDB 4.4, you have
even more flexibility in how you query
your data as well. Dan, take it away. - Thanks, Sahir. MongoDB is awesome as a unified
data platform and unity is all about bringing things
together in the same place. We've done that inside
MongoDB as well. For example, we added the
$union aggregation pipeline stage. Union allows you to blend
data from multiple collections and pipelines, giving you more
ways to explore and query data. Rather than just
stuffing all your data you want to work with
into one giant collection, you can keep data in separate collections that make sense, and bring them
together at query time. Union is another way of pushing
much more work to the database instead of doing it in
your own application. Something else we've done that
allows you to push more work to your database, is allowing
you to embed custom JavaScript functions right inside
your aggregation pipelines. This was something that
was allowed in the past, but with limitations. MongoDB's new custom
aggregation expressions are much more robust,
and give you the flexibility to do more work in
the place that gives you the best performance. This is especially great
for any calculations that you are currently doing in
your application, by pulling in potentially
large amounts of data. My words aren't going
to do this justice. Adrienne, can you
show us an example? - Of course, Dan. One really neat thing
we're introducing in MongoDB 4.4 is the new
$function operator, which you can use with custom
aggregation expressions. Most things I need to do with
data are covered by the existing MongoDB operators,
but not everything. Let's say I want to get a crew
roster for the last three days. That is, every vessel boarded
in the last three days, and a list of their crew,
in alphabetical order. In our record, "crew" is an
array of subdocuments with lots of information,
but all we need is the name. And then we need to
sort the array by name. Trying to do this in MongoDB
means we would have a lengthy and confusing aggregation. Using the $function
operator makes this simple. With it, I can create a
function right inside my custom aggregation expression
that returns the list of vessels and crew members. I can do this complex
query with just a few simple lines of JavaScript. Add in a match for the
past three days at the top, and a projection for the
fields I want at the bottom, and then I run the query. And here you see the results. Here's the list of vessels
and their crew members from the last three days. Crew members are in
alphabetical order, and I only had to use a
fairly simple aggregation. - Thank you, Adrienne. With custom aggregation
expressions you can do more processing server side,
delivering higher performance to your users, and you can
precisely customize MongoDB to your specific needs. There's a ton more that's
new with the query language, letting you rely more on MongoDB
to handle your work for you. We support MongoDB
drivers for all of the most popular languages,
and we are really excited to add some new drivers as well. The latest additions to the club
are the MongoDB Rust driver, as well as the
MongoDB Swift driver. Both are generally
available after having been in beta for a few months. So, if you use Rust or Swift,
it's now even easier to work with MongoDB. - Thanks, Dan. MongoDB 4.4 packs in
the features and improvements most demanded by you; an
ever-richer query language, the flexibility to refine your
data distribution at any time, with the most
sophisticated latency and security
controls anywhere. Now that we've looked
over the improvements coming in MongoDB 4.4,
can you tell everyone about the last piece in
the Atlas data foundation? - We want you to love
using MongoDB Atlas, and when it comes to management,
the best database is the one that you
don't have to think about. Another thing that we've
learned from our community is that MongoDB
and search go hand in hand. Today's search is an integral
part of nearly every application experience we touch. Netflix, YouTube, Twitter,
Instagram, LinkedIn, GitHub, Stack Overflow, you name it. True search capability is
more than just running queries against a live database. It's about full-text
indexing, intelligent recall, and relevance-based ranking. Given that search is such a
fundamental expectation for modern user experiences, and
that search is all about data, it just makes sense for
us to take the problem on. Search isn't easy. Another technology to
install, maintain and scale, another data format, another
query language for accessing it, another process for
shuffling data from database to search engine. We set out to make
searching your data as simple as querying it, right in Atlas,
and super simple to use. First, we wanted Atlas
Search to scale transparently with your database. No second service to manage
to add configuration. Second, we wanted a single
unified query language and data model. Third, we wanted to offer
you the full power of Lucene, the premier open
source search engine and the industry standard. If you attended
MongoDB World 2019, you saw a limited beta version
of Atlas full-text search. Today we're thrilled
to announce the general availability of Atlas Search. It now has support for more
data types, not just text. You can index data
in more languages. You can now use filters
and facets to offer advanced search interfaces. We've even added type-ahead,
autocomplete, fuzzy search, and dozens of new operators
and options for building advanced search queries. The reporting back end we built
into the WildAid application is a typical example of
the need for diverse and rich search functionality. - To give the WildAid staff
a way to visualize boarding activity data, pull reports,
and inspect or correct specific records,
we built an app for the web using MongoDB Cloud and React. Let's say I want to find
a record in the web app. I remember that either
the name of the vessel or the captain was Mya Jane. Autocomplete helps me
find results even before I'm finished typing,
and I find the Mya Jane, even though
I spelled "Mya" wrong. Fuzzy matching took
care of that for me. I want to
see all the boardings. Notice how on every page
the data that matched my search
string is highlighted. Now I'm seeing only
boardings matching "Mya," but I'm looking
for a specific record, one where a citation was issued,
which means I want to filter on risk level "red."
Now I have two records, and get more information
on both to figure out exactly which was the record
I was thinking of. That's a lot of
functionality in one search. Let's have a look
under the hood. Atlas Search uses a
special full-text index which you can
find in the search tab. As you can see here, we've
created a search index for our boarding reports collection. The index has the default
settings dynamically mapping every field in the collection,
so that when we do a full-text search, we can do it
across all our fields. Right now, the default
search index works just fine for our current application because we only have a few thousand records. For a larger application,
or when specific sets of fields are used more than others, mapping to these specific fields is beneficial. Now that we've seen
the full-text index that powers our application, let's
see how to use it. Let's look at the code
for the web page search. To use Atlas Search,
there's a new stage in aggregation expressions. This stage used to be
called "search beta, " but now it's just $search. Here, we query the search
term against the vessel names, the crew names,
and the captain name, and violation codes
and descriptions. The fuzzy option tells Atlas
Search to look for the text that matches
within one character. The highlight option is
used to indicate which texts match the search string. In the production stage, we
returned not just the fields, but the highlights. That's why "Mya" was highlighted
in our boarding results. When we filtered on
the risk level on "red, " we were using "faceted
search" in Atlas Search. Numbers, dates and geolocation
data can be used to filter results and influence
the relevant score. With Atlas Search
I can have the robust, fine-grained search
functionality that everyone expects, and I can add it in
my application in an afternoon, no third-party system needed. - We think you're going to
love working with Atlas Search. One more piece of the data
development puzzle snapping into place,
and we want you to use it. That means no limits,
no extra cost, and an integrated set
of tools to bring powerful search experiences to
every application. Atlas may be four years old,
but we are just getting started. We see a future in which working
with data is delightfully easy, from online
transaction processing, to advanced analytics, to
real-time distribution, to search, and beyond. - MongoDB Atlas now provides
a unified data infrastructure for transactional, real-time
analytical and search workloads, with fully autonomous scaling
and even more intelligent performance optimization. We created
Atlas for all of you. We can't wait to see
what you build next. We have invested in
transforming organizationally, and we set the bar high when it
comes to security and compliance at MongoDB, so that you
can be sure that we are completely worthy
of your trust. To that end, our CSO
works relentlessly to give us and you
the security we need. Lena? - Hey, Sahir. - Can you give everyone an
overview of what security looks like at MongoDB now, and what
we've done recently to give our customers even more confidence
in us managing their data? - Sure, Sahir. So we were involved
in the earliest stages of the systems development
life cycle at MongoDB, and we work hand-in-hand
with engineering to produce, maintain, and enhance all
our security-based offerings. Strong security is often
considered to be at odds with ease of use. But the truth is, baroque,
onerous security requirements will lead to surprisingly
insecure consequences. At MongoDB, we understand
a complete security design has to put
the developer at the center. The strongest encryption in
the world is useless if developers can't
figure out how to turn it on correctly. Client-side field
level encryption, a feature we added to MongoDB
last year with the 4.2 release, is a perfect example of this
developer-centric approach to security, especially in
the context of cloud database services like MongoDB Atlas. There's some data that's
so sensitive that you want to encrypt it before it
even leaves your network. Take this basic document with a
user's social security number, for example. This data is so sensitive that
the typical mechanisms that secure data in the
cloud are inadequate. You could encrypt the entire
document on the client side, but then the database wouldn't
operate usefully on any of the less sensitive data. MongoDB's client-side field
level encryption lets you specify which fields are
sensitive and who can see them. So developers work with data
the same way they always do, but that sensitive document now
looks like this before it even leaves the client application. This feature has been generally
available since December last year, and since then, we have
added more drivers supporting Javascript, Node.js, Python,
Java, Java Reactive Streams, Scala, C#,
.NET, and Go drivers. More are going through testing
and will be available soon. Secrets management is another
area where security best practices and developer ease
of use are complimentary. HashiCorp Vault is a centralized
system for storing and controlling access to secrets,
such as passwords, certificates, and encryption keys for all
the services and applications. It allows compliance and
security teams to set policy, and gives developers a single,
unified way to work with the secrets for their cloud
services and platforms. And HashiCorp Vault runs
on any major public cloud. Now naturally, we want your
dev-ops teams to be able to make use of the most effective tools
to manage your Atlas cluster secrets, so we built
MongoDB Atlas Secrets Engine for HashiCorp Vault. The Atlas Secrets Engine makes
it easy to manage and control access for database users,
while reducing security risks, and increasing
developer productivity. On the other hand, if your
infrastructure is concentrated in AWS, you're definitely
using AWS identity and access management to manage
those services, and access to MongoDB Atlas
is an outlier in your security responsibilities. So today, we're announcing
AWS-IAM Database Authentication in MongoDB Atlas. Using IAM, you can now allow
applications, containers, and serverless functions to
authenticate to Atlas clusters using temporary
AWS-IAM credentials, in the same way your
application would authenticate to other AWS services. This means that developers do
not have to switch context for authentication between their
applications, any AWS services, or MongoDB Atlas clusters. If you're in dev-ops, it means
that you have fewer secrets to manage, and you can apply
consistent policies across the secrets used
by the application. Using AWS-IAM authentication
works across Atlas, so you can use the permissions
you define there to control access to clusters deployed
in Microsoft Azure, and GCP as well. With this, your authentication
mechanism across AWS and MongoDB Atlas clusters can
also be unified, paving the way to faster
application development, and less operational
overhead for dev-ops. We've spoken about cluster
secret management as well as authentication, which are
important building blocks to secure applications. While all traffic from
users in applications to MongoDB Atlas is always encrypted in
transit with TLS, enterprises can require this
traffic to use private network connectivity made available
by cloud providers. This is where Atlas
Private Endpoints, powered by AWS
PrivateLink, come in. If you're using AWS PrivateLink
for private network peering, you can now also connect Atlas
to your AWS applications to ensure private connectivity
between your clusters and all your AWS services and accounts. With this in place, your
operations teams no longer need to update or maintain
a separate set of access controls and security
protocols for MongoDB Atlas. Another win for compliance,
and a valuable aid in maintaining those very
important access controls. Atlas Secret Engines
for HashiCorp Vault, Identity and Access Management
Database Authentication, and AWS PrivateLink all help
to make your applications more secure by simplifying
your system and providing integrations with services
and tools you know and trust. This year, our main focus was
to simplify data security in the cloud for MongoDB Atlas. We've integrated with leading
identity and access management technologies, and secrets
management providers to make your lives easier. - Thank you, Lena. Rock solid security is
only one of the foundations of a modern application. Some others are orchestration,
programmatic automation and the ability to revert
backups when needed. If you tuned in last year,
you heard about our backup re-architecture project. We're happy to announce
that we now support all the most commonly requested
backup features. We speed up all your backup
processes and allow you to restore to any point in time. We introduced the MongoDB
Enterprise Operator for Kubernetes last year. It allows you to deploy
and manage MongoDB clusters from the Kubernetes
API and use OPS Manager, the management
platform for MongoDB, for the database automation,
monitoring and backup. This year we made
things even easier. The Enterprise Operator
can now deploy OPS Manager in containers. This will dramatically simplify
running containerized MongoDB deployments from your
Kubernetes Control Plane. But what if you're just
getting started with databases in containers? Don't worry, we've
got you covered. The new Kubernetes Community
Operator is open source and can deploy replica
sets of MongoDB community. If you're using Atlas,
there are also some great new integrations allowing
you to leverage configuration as code to automate cloud data
infrastructure deployment. Historically, our cloud products
have had two ways to interact with them; a GUI
for people, and a REST API for programmatic integrations. The new CLI for MongoDB Cloud
is for those that don't want to work in GUIs but want something
that's more approachable, interactive, and easily
scriptable than REST APIs. As such, the CLI is an
ideal middle ground. A simple interface that allows
for both ad hoc task execution and easy script ability
for a current operation. With the new MongoDB CLI,
you can also easily switch between managing your
Atlas environments and self-hosted clusters in
OPS Manager or Cloud Manager. If you're using HashiCorp
Terraform to manage your infrastructure, the updated
Atlas provider automates your data infrastructure deployments
by making it easy to provision, manage, and control Atlas
configurations as code. And we continue to expand
the Atlas features supported. Beyond just
provisioning clusters, you can now automate additional
tasks like creating projects and managing database users. Last but not least, we also
got something for you if you're building data
pipelines with Kafka. The MongoDB Connector for Apache
Kafka enables you to easily integrate MongoDB with Apache
Kafka so you can build powerful real-time data pipelines. We've worked very closely
with our partner, Confluent, and are pleased to announce the
preview availability of MongoDB Connector on Confluent Cloud. Now your Kafka and MongoDB
experience can be entirely managed in the cloud,
no spare parts for you to operate yourself. We're excited to have worked
with these partners to make sure the MongoDB Cloud platform
easily integrates into the rest of your
infrastructure and processes. MongoDB Cloud provides
an integrated data foundation for creating and growing
the most advanced, demanding, and powerful applications. One truth we hear over
and over is that whatever you're building,
once you have data, you're going to
want to understand it. If you're using an off the shelf
business intelligence tool, our BI connector makes it easy
to connect directly to MongoDB. But we want you to be
able to get all of your data work done in one place. That's why we built the fully
integrated MongoDB Charts. Charts allows you to visualize
data right where it lives, no transformation required. - Charts works directly
on your Atlas cluster data, and it understands your
data's content and structure. - Here's a dashboard that shows
the most important information for WildAid. At the top are the
critical stats at a glance. Next, we have an interactive
map that shows boardings. I can zoom in to see boardings
near the Galapagos Islands. - Charts aren't just about
making it easy for you to analyze your own data,
they're also a great tool for visualizing data in user
facing applications. You could always roll your own,
but we make it easy to embed live updating charts directly
into web application interface with just an iframe. - All I need to do is
copy this imbed code. I have a simple website with
information about threats to the Galapagos Islands here,
so I'll paste the embed code in my web page and save it. Now, when I refresh my web
page, I can see my chart. Pretty simple, right? Without Charts I'd probably
have to refresh my D3 skills or learn something else
to visualize this data. Not what I want to do
with my time right now. Because I'm embedding a chart,
I don't even have to be the one to create
or edit the chart. The data visualization
team can easily create the visualization
in Charts without me. That's a win for everyone. - Thanks, Lauren. We love embedded charts but
there's one thing you've been asking for for a while;
JavaScript rendering. The new MongoDB Charts
embedding SDK is exactly that. Bob and Karima from WildAid
are here to show you how we use the new embedded
SDK in the WildAid App we created together. - Hi Cailin. - So, what do you want
to see in your app? - A good indication of
the effectiveness of our enforcement
is compliance rate. We also want to see
our boardings on a map, looking at which zones are
potentially more dangerous, and be able to drill down into
the details of the boardings. The map will help us make
decisions on where to focus our patrols to be more effective,
and target our efforts so that we can optimize
our time and fuel. - Building that with the
embedding SDK was easy. The SDK lets me control
the size, theme, filters, and refresh behavior,
as well as the integration with auth tokens
directly in the app. Bob, can you
see if it works? - Sure.
Seems to be working. Earlier you showed me how we
can zoom in and move the map, so let me try that. I'll zoom in on
the coast of Ecuador. Ah, so here we can see
where we've been patrolling, and a big spot we may
need to focus on more. Very good work. - We're really excited
to see how you can use embedded charts to bring live data into
your own applications. When you need to really
dig into your data though, you're going to want
to use the full power of Charts and Dashboards. - Here's a dashboard. Bob, can you explain
what we're seeing? - Yeah, this looks like
data from Saildrone. The chart on the left
shows the water temperature at different times. Sailfish can be targeted
by illegal longliners. The ideal temperature for
sailfish is somewhere between 23.5 and 25.8 degrees Celsius. By using the chart on the right,
we can see temperature plumes that help us figure out
where the sailfish might be. The points in blue
are in the ideal water temperature range for sailfish. And the points
in green are not. So we now have a visual
aid to protect the sailfish from the illegal longliners. - What I'd really like to
be able to do is map where the ideal sailfish
temperatures are so we can plan our patrols
to optimize our efforts. - MongoDB Charts on Atlas
now supports dashboard level filters, allowing you
to quickly change the context of an entire dashboard to get
actionable insights faster. - Let me build that for you. I'm going to apply the filter
for our water temperature. I'll set the min to 23.5,
and the max to 25.8. Now the chart on the left is
only showing the blue points that show the ideal
sailfish temperature, because the filter
is being applied. Karima, what would you
like the default to be in terms of
how recent the data is? - The last
week would be great! - Okay, give me a moment. Now that we have the data,
let's zoom out so we can see where we are. Here you go. - Wow, perfect!
Now we know exactly where to send our patrols. - Exactly. You can visualize subsets
of data across the entire dashboard, and anyone else who
has access can do the same, making collaboration very easy. - MongoDB Charts
is always evolving. Take a look at our
newest features, including the new
Charts Embedding SDK, Dashboard Filtering,
and Public Link Sharing. Charts lets you visualize your
document data with an intuitive drag and drop interface,
allowing anyone to build and share powerful dashboards
internally and externally, and developers to code
them into your apps, without having to waste
time pushing pixels. - Charts allowed the team
to seamlessly start analyzing data without any custom
transformations or integrations. Data is rich and heterogeneous. You should be able to visualize
it with a tool that understands its structure natively,
and with Charts it's never been easier to enrich
your applications with live visualizations. The database for modern
applications needs equally modern tooling, because
you want to move fast and solve
problems for your users. We want to make sure that
engineers who develop their solutions on MongoDB
platform have a seamless and integrated experience. - That's right, Sahir. And that's why we have
placed a special focus on improving the user experience
around using MongoDB. So we were looking into
what developers love most. Exactly!
Fancy keyboards! And that led us to thinking
about the way we type instructions. If MongoDB makes it
easy to work with data, the MongoDB Shell makes it
easy to work with MongoDB. The MongoDB Shell is the most
used tool to connect to our database, run queries,
and manage clusters. To improve the user experience,
there were a few things we wanted to do. So let me introduce you
to the new MongoDB Shell. The new MongoDB Shell gives you
a speed boost with autocomplete, and improved readability
with syntax highlighting. On top of that the new Shell
makes it easier to get back on track when you
encounter errors, with more
descriptive error messages. And last, but not least,
you can now get help directly in the Shell. The MongoDB Shell is
wherever you need it, whether that's on your favorite
terminal or in Compass, the graphical user
interface for MongoDB. The MongoDB Shell will be
integrated into Compass to give you the same simple, fast
and powerful experience. Previously, IDE users couldn't
use our shell seamlessly in their workflow. This somewhat defeats the
integrated part of using an IDE. In the new Shell,
this is changing. When you're coding
database queries, you often want to
validate your work against your sandbox database. If you previously used to
switch between terminal and IDE, you no longer have to. We've built an integration
for Visual Studio Code. Using it you can navigate your
databases and collections, try out queries
and aggregations, and integrate them
quickly in your code. - But Grace, I use IntelliJ. Can I have a shell
integration, too? - Grace, I use Webstorm, can
I get shell integration too? - What about me, I use PyCharm? - I use Golang. - I use Datagrid. - I use PHPStorm. - I use Appcode. - Okay, okay, chillax folks. We've got it covered. We've been working
closely with JetBrains, the maker of these tools. You'll get the same power you
have with the MongoDB Shell directly in those IDEs. - And there
was much rejoicing. MongoDB Shell is the best
shell for modern applications development with MongoDB. You'll be able to get
to your commands faster, wherever you need them. You can get the new Shell from
our downloads center or with your favorite package manager. The integrations
are available as extension managers of your IDEs. Aside from improving your
experience using MongoDB, we've also been hard at work
making our learning resources better for both beginners
and advanced learners. MongoDB University is our
online learning program, which has 1.5 million-plus
registrations in 196 countries since its launch. The completion rate is five
times the industry average, which shows just how
valuable our students are finding these resources. All technical MongoDB employees
go through the MongoDB University courses as
part of onboarding. We want the best for
our employees' education and for your education as well. But I also know that self-guided
learning can be daunting. Depending on whether you're a
developer or a database admin, you also need different
learning materials. You've been asking
for more directed guidance to help people find
the appropriate learning tracks through
our online courses. We're thrilled to show you
what we've done to provide it. We've introduced MongoDB
University Learning Paths to provide people learning
MongoDB guidance on what courses to take or to start,
and what path to follow that suits their role and needs. So now you're able to follow
a prescribed path and see how you're progressing. Please try it out and let
us know how you like it. You can get started at
university.mongodb.com. As great as academic
material is, working developers
also need real world resources that stay up to date
with the latest developments. Our recently launched Developer
Hub is a great resource for developers who tackle real-world
MongoDB development challenges across a wide range of
development languages with code examples, step-by-step
guides, and tutorials. We've got weekly Twitch
streams you can join and ask us questions, along with
podcasts and YouTube content. Check it out for yourself
on developer.mongodb.com, and we welcome you to
contribute sample code and best practices
on the site as well. And a cool side fact; the
Developer Hub uses our own technologies, Atlas
and Realm Server, as part of our static
publishing engine. If you have any
questions, suggestions, or just want to chat
with other MongoDB users, you can start a discussion
on our community forum. Community experts and MongoDB
employees from our product and other technical teams
all over the world are actively engaged on that site. There have already been
more than 1000 discussions since we launched
the forums three months ago. We are also now hosting virtual
meetups around the globe, and would love to see you
at our offline MongoDB user group meetings when
those happen again. Overall I just want to say we're
deeply committed to helping you, our community, succeed
in developing applications with MongoDB. To that end we want to make
our tools as flexible and frictionless to work
with as a database itself. We're excited to hear what
you think about the new Shell and MongoDB University
Learning Path, and we look forward to hearing
from you on our community forum. - Thank you, Grace. I look forward to continuing
our investment in making our developer experience
stunningly easy, and I can't wait to meet more
of you on our community forum. Today's digital reality is that
there is more data and more formats available to us than
we even know what to do with. MongoDB and the document model
provided consistent and unified experience for working
with all types of data. MongoDB Atlas Data Lake enables
you to bring disparate systems and services together in one
simple consistent interface for finding, analyzing,
and manipulating data, no matter where or
how it's stored. Data fragmentation doesn't only
occur when trying to work with data from different sources. You may be working with
fragmented data because you're optimizing for price
and performance. As usage of an
application grows, some data is naturally used
more frequently than other data. To optimize storage cost, you
want to hold only frequently accessed data in the database. To achieve this you
traditionally had to either delete data from your cluster,
or manually archive the data by moving it out of the
cluster to an alternate, cheaper storage location. - That's right, and what ends
up happening is that in order to reduce costs we end up making
some data less accessible, or even completely inaccessible. The reason we keep all this
historical data around is that we know it has some intrinsic
value, even if we rarely use it. Having all data accessible
also improves user experience, either because users expect
older data to remain accessible, or because you can provide a
better service with all of the data easily accessible. Essentially, if you're moving
data you solve for one problem by replacing it with another. It doesn't have to be that way. Today we are very pleased
to announce the beta of Atlas Online Archive,
allowing your archive data to remain online and queryable. Atlas Online Archive allows you
to seamlessly archive live data from your MongoDB Atlas
clusters into fully managed cloud object storage. All you need to do is specify
the rules for archiving, and from then on, it's all
automatic and transparent. Karen, can you give
me a hand here? - Hey, Cailin. - Let's show everyone how easy
it is to enable Atlas Online Archive for a running
Atlas cluster. - Sure, Cailin. We're going to use Saildrone's
sensor data again to show Online Archive in action. Right now, the data is all
in a Saildrone collection and there are nearly 500,000
documents in there. I'm going to use the
new MongoDB Shell that Grace talked about
to query the data. I'm going to do an aggregation
that gives me the average water temperature grouped by month. We've got thousands
of records per month, dating back to January 2019. Most of our queries only
care about the recent data, but if we just archived
older data to save money, we lose the ability
to query it easily. Online Archive literally gives
you the best of both worlds. So let me set this up
to archive all records older than six months. We can configure an online
archive here in the new Online Archive tab on
the clusters page. I'm going to create a time-based
rule to determine which documents get archived and
which stay here in Atlas. Our namespace is our database
name plus our collection, "Ocean Watch Saildrone,"
and will use the time UTC field as the
date field to archive on. The field we choose needs
to have an index on it, and luckily for me Adrienne
already added an index on the time UTC field earlier. And the archiving threshold,
six months or 180 days, this auto-generated query
will show us which documents will be archived. Let's see, more than 300,000,
that's a lot of records. That sounds great, and it's the
right data, so let's move on. To get the best performance,
we enter the most commonly queried fields, which will
partition our archive data. In our workload, our query
patterns often concern the average barometric pressure, and
the average water temperature. And that's it. Let's begin archiving. MongoDB will automatically
archive our data. There is no need to
run his periodically, and since we're migrating
so many records, this will take some time. So while that happens, I'll
just be here hanging out. - Thanks, Karen. What makes Atlas Online
Archive so exciting is that it allows you to save
on transactional database storage costs without
compromising on having that data available and queryable. - Now our archive is active. And our Atlas Data Explorer
shows that we now only have 150,000 records sitting in our
cluster, not the nearly 500, 000 we started with. However as Cailin said, all
of our data is still queryable using a single
connection string. In the Online Archive tab,
we have the option to either connect to cluster, Atlas data,
or connect to cluster and Online Archive. This is that magical
connection string. We can tell the difference
because it has "Atlas Online Archive"
in the name. Anyway, let me run
the same query here, just to prove it works. See? And even though not all the
data is in the Atlas cluster, it's still there
in Online Archive and it's still queryable. I didn't even have
to change the query, just the connection string. - Hold on a second. Did you catch that? - "But if we just archived
older data to save money, we lose the ability
to query it easily. Online Archive literally gives
you the best of both worlds." - I want to let that
really sink in. Before Online Archive, you
had two choices for your data. You could keep it in
the database, high cost, high accessibility,
or you could put it in cloud object storage,
low-cost, low accessibility. Online Archive adds
a middle ground. You can store your
data at low cost, but retain the same
easy accessibility you get from your database
without any code changes. - "And our Atlas Data
Explorer shows that we now only have 150, 000 records sitting in our cluster, not the nearly
500, 000 we started with." "In the Online Archive tab,
we have the option to either connect to cluster,
Atlas data, or connect to
cluster and Online Archive." - Imagine what you'd have
to do to get this functionality without a feature
like Atlas Online Archive. We've turned that into
a single configuration parameter and a single branch
statement that chooses between the online only and the full data connection strings. Okay, back to the show. - "I didn't even have
to change the query, just the connection string." - Great, but we're not done. Industrywide we have a
growing volume of rich, multi structured
data being generated. Connecting the right data
at the right time in the right place is harder
than it should be. Atlas Online Archive
allows you to tier and access historic data. But what could your application
do with even more data? Data from other systems,
other applications, from other parts of the
company, other partners? This is, of course, already
possible and people improve the application experience users
have by pulling in more data. They generally do this by either
connecting to different data sources and processing data
in the application layer, or by moving data around,
maybe transforming it. The point is it's not as
straightforward as it could be. Bringing external data into
real-time applications adds complexity, when fundamentally
it's a data problem. This is what we
want to help you with. Last year, we announced
Atlas Data Lake, a service that allows you
to query data in AWS S3, as though it was
in Atlas cluster. This year we have taken
Data Lake to the next level. We are very pleased to announce
that Atlas Data Lake is now GA, and supports federated queries. Federated queries allow you to
run queries against your live cluster data as well as against
data you have in cloud object storage at the same time. - Here, let me show you. This is my S3 bucket,
named "S3 WildAid" with CSV and parquet
files in an Atlas. We already have a Data Lake
connecting with those files. My Data Lake connection
strings are found here, but I've already
connected in Compass, and we could see our S3
collections from Compass. Data Lake doesn't import
any data it looks at it where it lives,
so here these collections are really files in S3. Atlas has never
stored that data. In order to query both
Atlas and my S3 bucket, I need to configure the
Data Lake to use both. Right now you could see
that there's only one data store set provided by S3. The S3 WildAid bucket
and the data store has been given the name "S3 WilsAid."
The name field is what we will use later in
the configuration file to refer to the S3 data source. At the top we have
a list of databases, and this database name
"S3 WildAid" is the name of the S3 bucket we
connected to our Data Lake. The collections in this
database, BoardingRecords, BoardingReports, and Crew,
are all listed here as well. This was automatically
configured by Data Lake. Now this last collection
is a wild card, and it creates collections
for everything in that bucket's path. So now we want to add our Atlas
data to this configuration, so we first go to
the array "stores, " and add the Atlas data store. The provider is "Atlas,"
my cluster name is "OceanWatchData,", and this
is my Atlas project ID. Finally, I name it
"ATLASwildaid." Now we can use this data store to define a
database in our Data Lake. the data source is coming
from Atlas data store which we just defined
as ATLASwildaid. This collection is coming
from a regular Atlas database, and we use the existing
BoardingReports collection. So this should now show up in
the Data Lake as a database named "wildaid-new-data."
Let's check this in Compass. So this is great,
now Data Lake has our S3 data and our Atlas data. But we need to
query them together, that's the whole point of
federated queries after all, so let's set that up now. Let's change the collection
that we just made to combine data
from both Atlas and S3. So let's add another
data source, the S3 data, to this new Atlas WildAid. In our Data Lake configuration,
we add two more data sources from the S3 WildAid store,
"BoardingReports.CSV, " and then the parquet file
with the "BoardingRecords." To reflect that this is now
WildAid data together, I'm going to change the database
name to "wildaidAll." In the Atlas data, we have
rich documents with nested arrays and embedded objects,
but CSV and parquet files, that's flat data. So now we run a federated
query that combines both. So back in the MongoDB
Shell, we'll use the database "wildaidAll" and do an
aggregation to check for compliance records on this
new, modified collection. And here we have the results. We ran this aggregation to get
the number of compliant records for every year and month,
and I'd like to store the results later. In normal aggregations you
can use $out to save results to a MongoDB collection,
now you can use $out to save a JSON or a BSON file
to your S3 bucket right from inside Atlas Data Lake. So now I run the same
aggregation with $out stage appended to the pipeline. This should create an
"allcompliance" file to our S3 bucket. So now let's look
again at our S3 bucket. Ta-dah! And as we can see
the "allcompliance". JSON file that we just
wrote from our Data Lake. I'm really excited about
how easy it is to query all of our data in the Data Lake
alongside my Atlas data. I could go on forever. I haven't even shown you some
of the other cool new features, like how to define a
view in Atlas Data Lake, and SQL support. And Data Lake now supports the
aggregation stage currentOP, which shows you
information about currently running operations. But even though we don't
have time for that, I honestly hope I've blown
your mind a little bit with what I've shown you,
and you'll be able to learn all about those other
things in sessions today and tomorrow. - With these changes, Atlas
Data Lake helps you bring cloud object storage data together
with your MongoDB data, while keeping it in place
in its native format, without data movement
or transformation. - With Atlas Data Lake
and Atlas Online Archive, you can have a consistent and
unified experience for working with all types in tiers of data,
providing easy access to all of your data wherever in
the cloud it lives. We think Online Archives
and Atlas Data Lake are going to revolutionize how you
tier and use your data, and we're excited to explore
this future with you. Today, along with our
partners at WildAid, we've shown you the incredible
capabilities of the new MongoDB Cloud Data platform,
and how it lets you build modern applications with
the seamless data architecture, spanning every
layer of the stack. You've seen how MongoDB Realm
makes it trivial to write cross platform mobile applications
that sync to the cloud bi-directionally
with barely any code, respond to changes in real
time, and integrate easily with services you need
to build your back end. In MongoDB Atlas we improve
performance advisor, introduce full compute
and storage autoscaling, and show schema suggestions
that help everyone get started with MongoDB Atlas in
the best possible way. And Atlas Search, now
generally available, searches your Atlas data
with the flip of a button. MongoDB Atlas is the modern
data foundation to build on. MongoDB 4.4 will deliver
the features and enhancements you've asked for the most, including refinable shard keys, compound hash shard keys,
as well as hedged reads and mirrored reads. 4.4 will also feature additions
to the query language in the form of unions and custom
aggregation expressions. We simplified data security
for MongoDB Atlas with Atlas Private Endpoints, powered
by AWS PrivateLink, integrated with AWS-IAM for
database authentication, and added support for
HashiCorp Vault with the Atlas Secrets Engine. With Ops & Integrations, we
talked about the Kubernetes Community Operator,
the backup re-architecture, the CLI for MongoDB Cloud,
the Terraform Atlas Provider, and the MongoDB Connector
for Apache Kafka. MongoDB Charts; document
native visualization has added dashboard filtering as
well as an embeddable SDK to integrate directly
with your applications. You saw some great new tools for
developers like our new MongoDB Shell and its integrations into
VS code and JetBrain's IDEs, and some great learning
resources like MongoDB University Learning Paths,
and the new Developer Hub. And finally, you saw the general
availability of Atlas Data Lake in new industry redefining
data tiering capabilities and Atlas Online Archive
and federated queries. Dev, I think we showed off
some amazing stuff today, and hopefully gave everyone
a sense of how we can simplify the data architecture for
modern applications. - Sahir, I couldn't be more
proud of all the great things we just talked about. As I said at the beginning
of this presentation, every business' future
will be powered by software. Everything we do stems from
our desire to take the friction and hassle out
of working with data, so developers can focus
on what really matters. We are incredibly excited
about what we are building at MongoDB. And I hope over these
next two days you'll find that information helpful, as you
think about what you want to build for your organization. Whatever your needs and
wherever you are in the world, we hope you recognize that
MongoDB is the modern, general purpose data
platform that lets you build for the future. Thank you to all the members of
our community, our customers, our partners and our employees,
and thank you for joining us today, and I truly hope the
next time we meet is in person. Until then, from all of us,
stay safe, stay healthy and now, please enjoy the conference. Bye. - Thanks. - Bye. - Thank you, bye. - Thank you everyone. - Bye. - How do I...? Oh man, the
button always changes... uhhhh. Aw, forget it.
