CRAIG LABENZ: Hello, everyone. Welcome to another episode of
Observable Flutter or as we could call it today
Debuggable Flutter. Actually, we're not really
going be doing anything with Flutter today, just Dart. My name, as always, is Craig
Labenz, and I am your host. And before I get
started, let's move through all the housekeeping. Super important. Remember, folks, that
it's very important to me, and I hope to all of
you, that we live up to the standards of the
Flutter community here. So we're under the
YouTube terms of service and the Flutter
code of conduct, so let's be very kind and
respectful to each other, as always. And I think that's actually it. We're pretty much
ready to go now. So I promised you
today live debugging, and that is what I'm
going to deliver. So it was brought
to my attention by the creator of
Serverpod, Viktor Lidholt, that there's a problem
with connection pooling coming from
Cloud Run on GCP going to a Postgres database. And at first glance, that may
not seem like a giant problem to everyone, but it's
basically a showstopper for deploying Dart
code to Cloud Run that wants to talk to a database,
because connection pooling is absolutely critical. So if you're not familiar
with connection pooling, basically if you don't use
it, then every single request you make to your app will
probably naively spin up a new connection
to the database, and that will both be slow
and expensive, and could hammer your database with just
crazy amount of connections. So connection pooling
means that each worker node kind of maintains this
pool of connections and reuses them, and
actually stay open, so it's much faster to
talk to your database, and the amount of connections
going to the database is more consistent
and more predictable. So connection pooling
is a just best practice of all the best practices
in server side development. And it's not working for
Cloud Run and Postgres in the Dart Postgres
pools package. So let's get going here. So this is the error that you
get after a second request. It works after the first
request, and then it times out. And this works from
other environments, so like Google
Compute Engine, GCE, if you deploy your app,
there it works just fine. But from Cloud Run, we get
this issue, this response. I didn't look at what
the error code is here. That's not what I want. I want this. Then we'll go to Network. And I probably have
to refresh here, so it might work the first time. There we see, because Cloud
Run had turned off since I reproduced this bug last night. But now, if I refresh
it's just going to work. What? Refresh. OK, now it's hanging. Does it work twice? That would be a surprising
number of it working. Let's see. Randall says, connection
pooling is "very important when the amount of work done
to connect and disconnect dwarfs the actual query." Yeah, absolutely. If you're just-- and that'll
be true for most queries that you want to run in
a request response cycle. Any long query is,
hopefully, just running on a clone
of-- a read-only clone of your database that you use
for analytics and whatnot. Anyway, we can see this hanging. It takes a while,
so we won't get to see the status
for a minute here. But then we will-- I want to make the-- there we go. OK, too big. We won't get to see
this until it times out. It might be several minutes,
but we'll see the actual status code eventually. So let me show you what I
did to reproduce this bug, and then we can
begin to debug it. And I have no idea where this
episode is going to take us. I didn't even add a
single print statement once I produced the bug. So it's all waiting
for us here today. So I ran, last night,
Dart Create dash t server dash shelf, and
made a new server side app called connection pools. And let me go full desktop
here, so I'm not blocking stuff. So I just created this
app, connection pools, and then that generated some
code pretty similar to this. But I just made a
few changes to it. So in my pub spec file, I
added the Postgres package and the Postgres
pool package, that enjoy a nice little bit
of version symmetry there. And everything else-- oh, I
added GCP, but I didn't use it. I think I'm actually
going to get rid of it. I thought it had a
good logging utility, but I actually couldn't quite
figure out how to use it. So yeah, I added
these two libraries. And that is it. Yeah, that's all I did in
here, other than the unused GCP package. And so then back in server,
I made a Postgres pool. And this just comes,
by the way, from the-- if we go to pub.dev, and go
to packages Postgres pool, we'll see in the
example what to do. So we make a Postgres pool. We pass it a Postgres endpoint. Set some settings. And then when we actually
want to run a query, we access our Postgres
pool, we call dot run, and that gives us a callback. And this is important
because the pool has a fixed size of connections. And so let's say it has four
connections as specified here, if we have five
queries active right now, maybe because we've
parallelized them in our app or there's just a bunch
of requests coming in, that fifth one, while
the other four are out, it's going to have to wait. It doesn't open
a new connection. So it might seem like,
well, that's slower, but over time, in aggregate,
it's faster to do this, and way more performant,
and easier on the database. So that fifth one is just
going to have to wait a second. So of course, we
pass a callback. And then whenever the pool
has an available connection, it will use-- you know, it will
kind of run the next callback in the queue. And hopefully, that callback
actually runs a query, otherwise, then you're
really just abusing your connection pool. And so yeah, and
then we don't really have to worry about the futures
thing that they're doing here. This is just kind
of showing that you can parallelize these queries. So that is kind
of what we were-- oh yeah, so that's
the documentation. So back here. So I made a Postgres pool. And then down in
the main method, which was generated by
the command that I ran, Dart create, in the main
method, right here-- let me get rid of that window-- I'm making a Postgres
pool, and then I'm just kind of farming out
the endpoint to a method. And the reason
that I'm doing that is simply so that I can
have some local host versus production branching. So if there's a DB host
environment variable, then that tells me
that I'm in Cloud Run. And so then I'm going to do
this kind of arcane thing that I'll talk about to
get my host for where the database lives. And if there is no DB
host environment variable, then I just assume I'm sitting
at my computer, right here, and then it's local host is
where the database lives. And I'm just running a
database from Docker right now. And then all of
this stuff, again, I'll take a database
name, and user, and password from
the environment if they're available,
otherwise I'll just use Postgres, Postgres Admin. And then of course, when
I deploy to Cloud Run, I supply these
environment variables. So I got this all working last
night, just because I didn't-- reproducing the bug
isn't debugging the bug, and wanted to debug on the show. But now, just walking
through it quickly. So we have the PG endpoint
that sets up the Postgres pool. I'm just using the same
settings from the documentation. And then all the rest
of this is, again, generated by Dart create. So I didn't touch
any of this code. But then back up top here,
now, we just have one route. So the home root says get time. And it runs that the
pool that we made, the Postgres pool dot run. And we pass it a callback. And our callback is
incredibly simple. We are just asking
the database what time it is, not particularly high
value usage of a database. But it will at least
run a connection. It will use a
connection, and run a query, all that good stuff. And then all these queries
return lists of lists, because, in general, queries
are going to return rows and each of those roles
are going to have values. Even though our query merely
returns a single value, it'll still be packaged
as a list of lists. So we just kind of drill
in, and get the time. And that was how-- well, we saw the time before,
but now it is timed out and we've gotten a 504, which
I guess, is just time out. I'm not going to lie,
I have not seen 504. Yeah, gateway time out. OK. It's funny, I feel like
normally that ends up being a 502 bad gateway. I don't think I've really
seen a 504 very much. Interesting. That may even be our first
bit of data to debug. I don't know. Let me look at some chat here. "Where are you
from," Samandar says. Well, I'm originally
from a little place called Michigan in an area
around a city of Detroit. But I'm currently in
California, where the weather is much warmer than Michigan. Randall says, "determining
the number of parallel items is a tuning issue and
will vary between setups. I've seen the number
between 5 and 50 personally. You just have to watch where
queries are mostly blocked." Yeah, database administration
is incredibly involved. There's no-- you just
have to get into it and look at what's going on. There's no other
way around that. OK, also we see here the
time was five minutes. And that's probably-- I
wonder where that comes from. I wonder if that's in this
settings variable here. Where are you? Let's go into the PG
settings class, and see is there any-- oh, query
timeout, five minutes. OK. So the library thinks
the query is timing out. And that's another
piece of data. So 504-- the server sent a 504. I wonder if the 504 was
omitted, like, I wonder if the client bailed
and set the 504, and nothing came
back from the server. I don't know, these are
interesting questions. I don't know the answer to that. Waiting for a response
from server, five minutes. And then we did get a
little content download. So upstream request
timeout, that probably did come from the server. And I haven't debugged a web
thing like this in a long time. You don't do this
very often in Flutter. All right, so we have a
couple of things to go on. And yeah, let's
get started here. Where should we begin? Now, one thing that I already
was thinking last night, but I didn't add that
print statement-- but the first thing
that I was wondering is do we even get
into this method? Does the pool try to
run another query? And I'm guessing it
does, but I don't know. So let's just print
running query. Seems pretty simple. Now, we get to redeploy
to Google Cloud. So that was stuff that I
also set up last night, because I didn't, again,
want to kind of slow down this stream with that. So I have this makefile here. And this isn't a very
Windows friendly pattern, because I don't think
makefiles work on Windows. But they're still
pretty nice overall. if nothing else, as just a bit
of documentation for yourself on like how does
this work, again. So there's two
steps that we need to run to redeploy
our code to Cloud run. The first thing is
that we need to use Docker to repackage our
code and upload that to Google's servers. And actually, it happens
in the other direction, we upload and then it
builds on Google's servers. And that happens with
GCloud builds submit, this command right here. Now GCloud builds
submit will use a Cloud build dot YAML file to
specify how and what it builds. And that's just an
implicit file name. If you have a different file
name, then you pass a flag. So in Cloud Build
dot YAML there's some more arcane stuff here. Actually, some commented
code, because I decided to separate
my build and deploy. So I'm just going to
delete that right now. So there's two
steps to this where we're going to build an image
from our code using Docker. And then we're going to push
that image to a container registry on Google's servers. And again, I'll just get rid
of some more commented code. So if I run, I can either
type GCloud builds submit or I can run my make
command, and either of those will rebuild the app. So I'm going to do that now. And I'll use make. I'm going to say make build. And this syntax here
means-- so the build command depends on the project command. And the project command
just sets what project runs. So that'll it'll be redundant
after every other time that I use it. But if you're switching back
and forth between projects, it's pretty nice. And then I've just kind of
set a couple of variables up here for myself. So make build. We'll see that it first
sets that project, and then it just dives
right into all of this stuff about rebuilding. We really don't need
to study this output, because it is just a
bunch of Docker output that is being streamed back
to us from Google servers. Vikas asks, "what is the
theme of this in VS Code." And it's a Synthwave 84. And it's really
quite nice, isn't it? Synthwave 84. That will give you this
juicy, purple goodness. All right, so Docker
is still running, which means we'll
read some more chat. "Makefiles date back to
Unix V7 45 years ago." All right, so that's
an interesting factoid. And my question
for you, Randall, is did you have to Google that? Because, boy, would
I have had to. Dayamoy, awe, well, shucks. I'm really glad to hear that. Thanks for the kind words. All right, we're still building. This is all, again, all
happening on Google servers. And this actually important,
building in this way, especially if you're
using an ARM like Mac-- like I'm currently using-- then building on
Google's servers is very important, because it
builds with the architecture of your current machine. So the first time I
got an updated Mac that has the ARM, Apple
Silicon architecture, I built a Docker artifact. And I sent it to Cloud Run. And it broke. And it exploded immediately. And I got some very weird-- maybe the word architecture
did appear in the error. I don't remember. But it was just, like,
what the heck is this? And then realized, like, oh, my
Docker file was built locally or my Docker image
was built locally to run on an ARM
architecture of my laptop, and then I sent it to Google's
computers that presumably are running Intel architecture. And it exploded. So building on Google servers
gets rid of that problem. OK, so the first step is done. We have built. And now
if we just scroll down on our makefile, again, I
was building and deploying kind of separately. I didn't want a deploy to
be dependent on a build. So I have a separate
command here, deploy. And it depends, again, only on
project, not on Build, again. And it just runs this
big long deploy command. So make deploy. And this will now run that
command, and deploy the app. And this is very fast,
because it's just moving-- everything's already
on Google servers, it's just kind of hopping
over into the Cloud Run spot. So it's pretty fast. And we're done. Great. So this should also mean the
old worker has definitely been turned off. So the first request,
it should work. And this means it will
get to look at some logs. So I'm going to refresh the app. And we got a time. And when I look in my logs,
let's have this refresh. I think, yeah, it's
still just showing the logs from last night. There we go. Now it's loading
at the bottom here. And when I scroll down,
we see running query. Great. Well, it's funny that
running query came through-- oh, it basically--
oh, no, listening-- server listening on
port came through after. How did that happen? Server listening on port comes
from the bottom of the file. I mean there's some-- you can't really predict
the order of log statements sometimes, but this
is a real surprise that this came 6
seconds after, right? 9:19:48-- this is
California time-- versus 9:19:42
for running query. I have no guesstimates
about that. Anyway, let's see when
I refresh and it hangs, do we get that log
statement again? It's hanging. And let's try to
refresh our logs. Nothing. No new log statement. All right, this is
our first data point. Run is not-- here we go. The callback is never executed. Hmm, let's dive
into run and start to see how on Earth this works. So it runs a function
outside of a transaction. OK. There's probably
a sibling function that runs it in a transaction. And so here's our function
that we pass, retry options, or else, retry if. Interesting. I mean, I don't think we want
to retry after a timeout, and then we'll just wait for
another round of timeouts. So I don't think
that's going to help. Trace ID, session ID. If we get further
into this and we make some progress
debugging this, maybe we'd use some
tracer session IDs. I'm not sure where
those end up going. OK, so there's
retry options, yeah, so we're probably using
some default, like, don't retry that
comes off the shelf. But anyway, so retry
options, dot retry. And so with connection,
let's think here. I'm guessing with
connection is-- so I think if we put a
print statement here-- and we're editing code
not in my app right now, so this is generally a bad idea. But actually, I might
have to pull this in. If I want to add print
statements to Postgres pool, I might have to just copy
the code into my own app and pretend it's my code. But anyway, I anticipate
that coming shortly. I bet if I put a print
statement here, it would print. And then if I look
in with connection, that's where the error would be. Because there's
just nothing here that could have
really broken yet. I mean, I guess retry
could have broken, because this is also
taking a callback. Let's peek at retry. Yeah, so we're looking
at some attempt logic. And this will be attempt zero. And then it just immediately
runs the function. Yeah, so we don't
have any retry logic, but that doesn't kick in
until after the function runs, so that's not the issue. So it is going to be
in with connection. So I'm not going to bother
making the print statement here, deploying, seeing it. I just see no way that the
error could come before this with connection function. So as we dive in here,
executor scheduled task. OK. And then this says
use or create. So we have two things to look
at, scheduled task and use or create. Oh, you know what? Also, there's a couple
other high level things that I want to check. I was thinking about this as I
was falling asleep last night. I want to check some high
level networking things in Google Cloud. I can't believe I
almost forgot this. So let me open my database page. We're going to
come back to this, but I want to rule out
a couple other things. Because, remember, we
know that this code works from Google Compute Engine. So if there's a bug
in here somewhere, it's very subtle
and sneaky, and it's just some intersection
between this code and, like, Cloud runs
networking, or something, right? It's going to be very
specific kind of narrow needle to thread. So I do want to rule out
a couple other things. So I'm going to switch over
to the My Database Settings. I didn't actually
mean to close that. So I'm going to
keep my logs open. And then in this tab, we'll
switch over to Cloud SQL. And in this pool's
database, I want to play with a
couple other things here, like getting rid
of the public IP address. I don't know if that's going
to change how this works. Also, a little
primer on database connections with Postgres. So remember there was, back in
server dot Dart at the bottom, I had all of this nonsense. And if you've not connected
to a Postgres database before, especially on Cloud Run, or
just using the socket file, you might see this and
think, I don't know. So what's happening
here is a common way to connect to databases,
especially Postgres, is through a socket file
that sits on your computer. And I'm not going
to lie, I don't even know how socket files work. I think they're black magic. But this socket file, it's
not even really a file, it's like a fake, virtual file. And it's essentially
just a connection to some other computer, and
it keeps that connection open. And when you have a database
connected to a Cloud Run instance, which we can see. If we go back to Cloud run, and
we look in our edit and deploy, blah, blah, blah. And then I scroll all
the way to the bottom, we see here that
Cloud SQL connections is pointing to this database. So this bit here is
enough to tell Cloud GCP, like, hey, these
two things should be allowed to talk to each other. Cloud Run is on the
list when he shows up at the database's bouncer,
it's like what's your name? And he says Cloud Run,
they're like, OK, you're in. So they're able to talk
by way of this connection. And then when you do
that, the next time you deploy Cloud Run, by the
magic of all the stuff going on behind the scenes,
it will add a socket file at this exact location,
and it's slash Cloud SQL, so from the root of the
runtime operating system, slash Cloud SQL. And then all of this is
the name of your database. So that comes from
your database page, and it's literally this
connection name right here. So I'll just copy this in. And we'll have a
commented version of this. That not how you make
a comment in Dart. So it's slash Cloud
SQL, slash all of that, and then this part is
actually just pure Postgres. This is very standard Postgres
stuff, so this follows you everywhere you go. So at this location, on your
running Cloud Run workers, there is a file right there. And you can use that as a
host, just like an IP address, just like a domain or something,
right, if you had mapped it that way. Even though it's
on your computer, you pass it as the host. And then you'll often,
in the wrapper libraries that set all this
up, you just tell it that it's a UNIX
socket, and then they do some other things
differently based on that. And again, I'm
deriving whether or not this is a UNIX socket, again,
by the presence of this DB host. This is just a convention
that I invented for myself in this project. This is not a global concept. So anyway, that's
how we're connecting. And this makes me think-- right, remember, I said that
this connection, the socket file acts like a host, like an
IP address for your local code. Well, I'm just wondering--
these things are timing out, something's going wrong
with the connection. What if this didn't have
a public IP address? Would that put it in
a different profile in terms of the Google
Cloud networking setup that will maybe unlock this? I have no idea, but
I want to find out. So it's been a minute
since I've done this. Maybe it's in the
connections tab networking. Yeah, I'm going to unassign
this public IP address. It will no longer
have a public IP. I have to select a
type of connectivity. I hope if I put on
private IP address that the socket will still work. Yeah, because now I'm
going to need to add a VPC. And I can, I guess, add
the Cloud Run instance to the same thing. Let's see here, I don't know. I'll put it on my VPC. I have no idea how old that is. This sandbox project is not new. Sure. My VPC. I could be barking up the
wrong tree here, don't know. Let's see. Kuasai says, "I'm curious about
the code created in one file, how many lines of code?" Which file are
you talking about? And I'd love to know
what you mean there. Well, let's see. There's a whole lot
of chat going on here. Are you guys solving
the issue for me here? Oh, Randall, didn't
have to Google it. He was there. Don't tell me to do that,
when I saw it happen. First typed to ls in 1977. Randall, I was a mere negative
9 years old at that time. All right, there's chat
about background tasks, OK. I'm going to hide myself here. We see in the bottom right that
this change that I just made is going through. So as soon as
that's done, then we can try to reconnect
and see how this goes. I don't know, I might have to
redeploy Cloud SQL-- or sorry, Cloud Run. I might have to
redeploy a Cloud Run if the connection path to the
database is different enough. Let's see. Hopefully, this
finishes quickly. I don't know if it will. While it runs, I'm
going to just try to brainstorm other things
that could be going on. Let's look at the
code some more. Postgres pool. Where were we? Why are we all the way
down here in this file? Where was run? There's run, there's
with connection. There we go. All right, so yeah, we
had two new players here, the executor and scheduled task. Also, concurrency,
that's interesting. So settings
concurrency is written to executor concurrency. And this is the maximum
number of concurrent sessions, which defaults to one. But we remember that it had been
upped to four in our settings. So four concurrent
connections we shouldn't be approaching
that, as this is merely our second query, and
the first one is done. So it shouldn't
be an issue there. OK, back to run. And then back to
with connection. All right, so
that's on an issue. Executor scheduled task,
let's see what this does. It schedules an async
task and returns with a future that completes
when the task is finished. The task may not get
executed immediately, indeed. So we need to find the
implementation of this. Abstract executor, so what
is our kind of executor? Let's find where
this comes from. We just saw that
this was abstract. Yeah, you-- oh, there's a
factory, which implements-- oh, it sends underscore executor. OK. How do you work? You must be in this file. You were. You're all in the
implementation file, is that like a part situation,
part implementation? Yes. OK. What method are we doing
on the executor, again? With connection. So it's executor scheduled task. So let's look at scheduled task. Scheduled task. There we go. All right, so we're
not getting this error. So it makes an item waiting. OK, this is just a list. And then trigger
is probably going to try to run something
off the first thing. Oh, man, it just doesn't
feel like the error is going to be here, you know? Because it works on GCE. Still kind of instructive,
still a little educational. So yeah, make sure that we are
following the concurrency rule. And waiting is not empty,
because if waiting is empty, then we can just get
out of this method. Something about rate. If rate does not equal null,
that seems like it's fine. A little bookkeeping on how
performant these things are, presumably. We have timeouts, right? So we need to know this. Started is not empty, right. So it's kind of a
queue essentially. So we're going to take the
first one off, and then run it, make a timer. Looking good. Started. OK, so we put it on a list
of things that are in flight, and then we continue. And use a completer. I always love using completers. I think it's really, really
fun to use a completer. All right, so then
it goes into running. And we call trigger, again. I don't know. It's a little confusing. I don't think the
problem is in there. Let's see if our
database updated. Nope. Still killing time. Anoj, we are debugging a
problem with connection pooling from Cloud Run to Cloud SQL. And I have no idea
where the error is. It may be so hidden
from us that we never find any hint of the
problem in this live stream. We should all be comfortable
with that possibility, especially because I've just
made this setting change. It's taking five
minutes to propagate. And I don't know what it's going
to look like to flip it back. And it's still going. All right, so anyway,
blah, blah, blah. I just it's really
hard to imagine. I haven't fully passed
this into my brain yet, but I don't think
it's the issue. I don't think it's
what's going on. All right, someone says-- Sarbador says, "what
is that scheduled task, recently I got that
problem in debugging." What? What do you mean? Say more. Identify yourself? All right, scheduled task. Let's go back to this. That's where we were. But I went into
trigger there briefly. And then we're waiting
the completers future. That's good. Result item, so
time out exception. But we did not get
this exception. We didn't see
anything about this. Except I don't know if that
would have really been logged. That's interesting. OK, let's keep that in mind. And then we await the task. So now we're finally-- is closing. Who are you? Where do you come from? Closing, does anything
set you to true? Close, we didn't call close. OK, that's not it. That was an efficient rule out. This was scheduled task. OK, so it's not that issue. It's not is closing, so
we can jump down here. We await the task. We mark it as complete. I love completers,
they're just so fun. I mean, it doesn't
seem to complete. So I think we're
awaiting the task. But this task is
probably just the method that we passed in, right? Yeah, so scheduled task came, it
was passed into scheduled task. And scheduled task came from-- oh, it's our method, which
was body, wrapped in this use or create. Oh, oh, oh, use or create. This is going to
be use or create a connection, because
this is a connection pool. [GASP] This is crazy. This is crazy. CryptoWolf says,
"what's the exception?" Well, we didn't get one. It just timed out. We got no logs. Nothing. "Maybe some networking like
virtual networks and access rates." Well, the first query works. Yeah, these are good hints. These are very good ideas. You may have missed hearing
that the first query works, then at times out. This also works from other
runtimes, so GCE, it works. GC-- I don't know if
Cloud Run has an acronym. But from Cloud
Run, it just starts to hang out only when we
use this connection pool. So let's see if this
database is updated. It is at last. And I don't even know,
it might have just broken our whole runtime here
or our Cloud Run app. But we're going to find out. So if we go back, there's some
button that's just like deploy again, even though
nothing changed. Oh, it's literally
right there, deploy. So this will just kind
of rebuild everything. And it will allow us to try
to reconnect to the database. One thing I don't
remember is, like, I think the socket will still work
even in the private IP universe that we just jumped into. I think this will still be fine. But let's find out. OK, we've updated. So I switch back
here, I refresh. It's timing out immediately. So I'm probably just
in a broken state. I don't think we should
read too much into this. Maybe when you have a private
IP, then you can't use this. Oh, no. No, no, no. No, no, no. No, no, no. We put this on a VPC. Yeah, I put it on a
VPC, MY VPC in fact. Which means I probably need
to do the same thing here. So if I go to a networking,
VPC network none. Yeah, OK, how do I edit you? Probably here. Let's edit, networking,
there we go. VPC. Why are you not showing My VPC? Connect to a network. I have to make a connector. I really don't like these. I just never know
what I'm doing. On my Boring Show, doing
a bunch of this stuff with Simon a year ago, we
got to a VPC connector. And I was just
like, Simon, help. And thankfully,
he knew everything off the top of his head,
which was really helpful. OK, so I accent,
I added the API. Serverless API,
make a connector. I don't think I really
want to do this. All right, let's try
briefly, at least. West2, that's what I picked. So this is My VPC connector. The network should
be My VPC, very good. The subnet, I have no idea. Can you just pick a reasonable
default for me, please. Create, no. I have to pick a subnet. I have no idea. Custom IP range. Can I just copy this default? I don't have a clue. Maybe that'll work. Don't have a clue. Don't know anything
about networking. Tec-X has noticed that
the blue of Flutter is very similar to the
green of my green screen. Good eye. Randall says, "the problem
may be a reuse connection. Maybe there's some
meta information that GCR does differently." Yeah, I mean, I think it's
going to be something like that. Just don't know what. All right, so we're
making this connector, which then maybe means here. Access resources,
learn more or create-- so you can learn more or create
a serverless VPC connector. I didn't. I could have learned
more, I guess. I'm a slow reader though. All right, let's go back
to what we were looking at. All right, we have this. What was it called? Get or create or something? Use or create? Use or create, yeah. So this method was called,
and it takes our function. This is what we passed in
right here, up to body, this goes all the way
back to this code here. This method is passed into run. And in run, it works its
way down to with connection. And it's the function because
this is what it's called here, function. So with connection
takes a function that wraps our function. Woo, callbacks on
callbacks on callbacks. So then we get into
with connection. And so this is already, again,
a wrapped version of ours. And we get or create-- or sorry, use or create. That's the body there. So this is a function
wrapping our function. And that function that
wraps our function takes the connection,
which makes sense because it's got to pass
that into our function. Not confusing at all. And yeah, this
looks interesting. So try, acquire
available, or await open. This seems like maybe where
the code is going to hang. So I'm going to, at
this point, I think, copy this code into my directory
and add some print statements. Let's try it,
because I'm not going to be able to add print
statements here, and then ship them up to the
server and have it work. That isn't how Docker works. Remember, it's going to build
for my pub spec dot YAML. And so even if I edit my
local copy of Postgres pool, it's not going to do anything
when I ship it to GCP. So I'm going to open
this lib directory here. Can I, like, is there a
show in Finder or something? How do I do this, please? OK, we're going to
have to do it manually. So what is this? We go to pub cache hosted
pub dot dev hosted-- here we go-- pub dot
dev and Postgres pool. Postgres pool. Here we go. And I guess I'll go
into lib as well. So this is-- and then
it's all just in one file. Oh, what is executor in? That's in a different
thing called executor. This is getting complicated. Randall says "copy path on right
click on editor window tab." Oh, right here. Copy path. Nice. All right, so I also
need this executor. How much do you have? It's also just one file,
plus the implementation. Why did I-- oh, right, no. It's two files. All right, we're going
to move all these over. It's going to be
a little tedious. So I'm going to copy Postgres
pool to my dev folder, my-- where are you? Where am I? Don't remember where this-- dev folder. I think it might be in my
Flutter dev reel folder. Connection pools, there we go. There we are. And then I think it's
the same folder again. And then we go into bin. And there it is. Copy. OK, one down, two to go. So now I'll go cd out of
this, out of Postgres pool, and into executor. OK. And then if we look
in lib, we just see its source and executor. So I'm going to copy the lib
directory to again out of this. No, do I have to-- how do you copy it? It's like copy recursively lib? I don't know if
it's going to work. Out of the executor folder, no. Root dev Flutter dev
rel connection pools connection pools bin. Let's see if it works. Well, OK, great. So now let's just cd
to that directory. And look what we got,
ll lib Postgres pool. Oh, my gosh. I did it right. Crazy. So now, I'm going to
close these other files that I have apparently edited. OK, now you're unedited. Now you're unedited. And I'll unedited this. Great. So in my-- let's lower this. And oh yeah, my connector. Let's check that out. Did the connector land? All right, My VPC
connector, looking good. So I want to make
connect to a network. Maybe have to refresh
this page, because I made the VPC connector
after loading the page. Network. Do you see it? You do. Wonderful. All right, only
requests to private IP through the VP connector. That seems correct. Let's try that one. OK, and now we
continue on our end. So in our pub spec
dot YAML file, I'm going to get rid of Postgres
pool, which included executor. Keeping Postgres, even though
I'm not really using it. And now, in server dot Dart,
we don't import Postgres pool. We import local
Postgres pool dot Dart. And in local Postgres
pool dot Dart, we just import local executor. Oh, we need retry. Son of a gun. I was so close. I hope retry is small. OK. Import. I guess we should go here. So this will be local executor. Oh, no, it's lib. I put it in lib for some reason. It made no sense. Lib executors dot Dart. So that gets rid of you. Now I need to do retry. Boo. All right, I'm going to go
back to the previous directory in my terminal. And I'm going to cd out
of this and into retry. OK, please be small. So we do lib. One file. Copy, retry to development
Flutter dev rel connection pools connection pools. Then go. OK, go back. Oh, I cd'ed into
the lib directory, so my go back one is broken. Dev Flutter dev rel
connection, connection bin. There we go. We got retry. So now this is just
retry dot Dart. This is crazy. I'm probably doing this-- have you ever heard
the expression go around the block
to get next door? Pretty sure that's
what we're doing here. All right, I wonder
if this will work. Do you think it'll work? I have no idea. Refresh. Cool. It worked. So we're on the VPC. I'm going to refresh again. Is it going to work? No. Changed nothing. Good though, we ruled it out. Actually, it's bad, because if
that had been the problem then we could have just filed a bug. But we're still hunting. So OK, so Postgres pool
is now fully migrated. This is some exciting stuff. And PG end point. Right, we were in executor. So we went from the run
file into with connection. With connection
had use or create. Oh yeah, we're still in the
same file but we needed to-- that's right. That's right. That's right. We just needed those
other dependencies. OK, so I'm going to perform
a little surgery here. And I just want to say,
so this is like, we'll call this option one, OK? Also, I'm an out of print here. And this is in use or create. So we have option one. And then we're going
to have option two. And this is option
one or option two. And then we can say resolved
option one with option one. We don't need those curlies. Resolved option two
with option two. And then our context is context. Oh, we're talking
about cascades. Yeah, cascades are pretty great. So cascade is a
shortened version of-- it basically is a Dart language
way of allowing everything to chain. So we'll have a real
quick demo into this here. I'm going to keep this fast. Language Dart. So normally, let's say you
have some class, A, then you have a method
void, whatever, right? And then you have
a version of A, so a equals A. Then
there's some attributes as well, or something, right? So let's say there's an int
count equals 0 or something. Let's say you want to call
whatever and set the count, well, you'd kind of have
to do A dot whatever, and A dot count equals 5. So this is three lines
of code to do that. Cascades allow
everything to chain. So if you want to void whatever
to be chainable, first of all, you'd have to return this. And then you'd say this is A,
is now my return type here. Now, I can say A dot
whatever dot count equals 5. This would work, right? So now I'm down
one line of code. But I couldn't go
in the other order, and what if that was important? What if I needed to do this? Because I have kind of
not good code, and this depends on count. So this code is bad, and
you shouldn't write code like this, where there's this
kind of sneaky dependency on the whatever
function on the count. But you know, we
all write bad code. So A dot count equals
5 dot whatever-- like, this is just incoherent, right? So we couldn't do this
in one line of code, even though whatever chains. We're back to A dot whatever. So cascading says whenever
I type 2 dots for this, then at the end of this
statement, actually, just like the thing that's
available right now in scope is whatever I started with. So A dot count equals 5-- like if I say dot here,
what am I working with? It's A, again, it like
pops one off the stack. And you can just do
that for methods. So A dot whatever means
if I say dot here, what is IntelliSense
going to give me? It's not whatever was
returned by whatever. OK, this is getting awkward,
a little who's on first. It's not the item
that was returned by the whatever
method, you know, unless they happen
to be the same. I don't really want to
save this file, now do I? It's not the item
that was returned by the whatever method. It's A, because it
pops one off the stack. So that's cascading. And it's nice. And it allows a single
line of code here. You can even do it off-- [INAUDIBLE] constructor
it was obvious. So we make the settings
object, and then we set the connection age. But we pop that off
the stack, and then we just set the concurrency
all in one line. And there it is. That's cascading. All right, where were we? What are we doing? I really should have started the
build before I went into that. Really, really should have. All right, so we have updated. We've added these
print statements. I'm just going to say make
build and make deploy. I'm in the wrong directory
for that command. OK, what should we talk about? Folks, we have a
few minutes to kill. Good question, by the way. Yeah, Randall says they
are required reading or required listening
if you happen to be watching right now. Yeah, Yageshrin,
says, "you might have to fix the import for
executor implementation." Oh, boy. Thank you for paying attention. That isn't in this file. That's probably in
executor dot Dart. Did I not fix that? Oh, I did. No, oh, it just
happens to be the same. That's why I copied the
folder over that way. Yeah, yeah, yeah. Good eye, though. Yeah, because it wouldn't
even compile if that was the case, but great point. Karan, there's not
a Discord for this, but Flutter has a Discord. And if you go to
Flutter.dev/community, you too can find a link to
that Discord and join it. I've not thought about making a
Discord for Observable Flutter, honestly. I kind of feel like we're all
in enough slacks and Discords. That's my initial
thought on the matter. Terminal. Oh, nice it was fast. It's almost done deploying. Almost done. So let's remind ourselves
what we're hoping to learn. In the use or create
method, I theorized that we were getting blocked
on getting this connection that we're either acquiring
an available or opening was bombing. And I just want to
find out exactly where in that dense line of
code this was happening. And so I broke it apart to
see where it's happening. And the deploy is done. And we're back at it. So refresh. It works. Let's see that
our log statements worked by going back
here, looking at logs. Scrolling to the bottom. OK, we see it. So resolved option one null. This absolutely makes sense
for the first request, because it make sense if
there wouldn't be an available one yet for the first request. So this instance of-- this is
really not very helpful other than whether or not it's null. So we got all of our statements. Absolutely makes sense. And then running query
happens at the end, of course, because after
it does all of this stuff, it finally gets to
our method, which is where I added the
running query log statement. So now I will refresh again. And it will hang. And we will read the logs. Coming in after 9:58:35. So scrolling. So refreshes. The problem is
before this method. Wow. The problem is
before this method. Really? Nothing. What does this mean? Double rainbow. What does it mean? OK, so use or create
is not the issue. Scheduled task may be. We're just going to
add things everywhere. Print in the scheduled task. Print. With connection. I'm shameless now. I'm so shameless. Print in with connection. OK. We're not calling close. Let's go back to run. In run, remember
when I ruled out that it could possibly be
retry, because that'd be nuts. Print-- we're going
to find out-- in run. And here we are. This is where this has to go. So this is with
retry, we'll call it. OK. Think a typo c with ctx. That would be in use or create. I think you're talking
about the difference between this variable
here c and ctx here. But they're actually
different variables. So this is just a function
that accepts a ctx, and we're calling it c. Apologies, by the way, if you're
thinking of something else and I've misread what
you're talking about. But this is a function
that accepts a connection, and it's called c. And we're first
building that connection and storing it in ctx. And then I bet later--
yeah, we pass it into body. So body takes a
connection and we are passing in what we build. But it has a different
variable name. Oh, we could-- wait,
no, I was going to-- I almost got excited. We could put things down here,
like even in this timeout. But we don't even get
these print statements. So we're not going
to get down there. Dinesh says, "could it
be a problem with the web server running your container." Yeah, it absolutely could. That's totally,
totally possible. And it's quite-- I mean, it's kind
of likely, actually. Because this works
on a different thing, it runs on Google
Compute Engine. I have not re-created that. But I was told that
it does work there. So I just trusted that. All right, I'm going to
add a couple more prints, because they're cheap. Oh yeah, I have another
thing I wanted to test. Because I was so surprised
that even happened. Let's just do a little
more cascading here. And let's say echo. Actually, I don't
even really care. This is just like-- we'll say marco. And then I'll make
a new method here. It's just a response
called polo. It gets the request. And it returns a response. OK, polo. What's your problem? Oh, get. You have to write actual code. OK, I want to make sure that
the whole thing isn't broken. Because I really
thought we were going to get those print statements. I think the marco URL will work,
but we're going to find out. OK, so I'm going to
deploy all of this. Chat with you find people again. What do we think? Should we take bets? What's the last print
statement we're going to say we're going to see? I'd love everyone
to place your bets. So running query is
the very last one. We know that has no chance. We already saw that all
of these didn't work. I kind of wish I had
done this before. So those are all
out of the running. We have, first,
it drops into run. Well, that's not run,
that's run transaction. So first it drops into run,
where the first possible print statements appear in
run and with retry. So those are the first two
answers you could give. Then it goes into with
connection, where two more contenders enter the arena. The last print statement we get
could be in with connection, if that's the case, it means the
schedule task shenanigans fail, or the task is scheduled, and
this would be the last thing that we run. "DebugPrint versus print. My linter doesn't like print." Yeah, for sure. DebugPrint is better
for local stuff, but I actually want this to
run in a production build of this server. So I have to use print. Normally, you don't want print
statements in your production build, but we are
deep in the weeds. Hamza says, "what are
you exactly building?" Well, am exactly debugging
a problem connecting to a Postgres database from
Cloud Run using the Dart Postgres pool library. The library works
from not Cloud Run. So the library may be
super-duper innocent. All right, we're deployed. So I'm going to refresh. And we'll see all
the print statements. I'm going to go back. And I'm going to look
at the print statements. And we look at the
print statements. Here we go. OK. In run with retry. In with connection. Scheduled task. In user create. It all works. Lovely. Moment of truth, folks. Moment of truth. I refreshed the wrong page. OK, the real moment of truth. Refresh. Switch back. Is that a band? Isn't there a band
called Switch back? Nothing. Are you kidding me? Whoa. Now, we try marco. It's hanging. We don't have any
more debugging to do. Wow. Wow. In the words of
Owen Wilson, "wow." I'm just so blown away. Wow, we're done debugging. Great. What else do we want to do? We have quite a
ticket to solve-- or I mean file. Someone else's to solve. [GROANING] Nothing. The whole thing dies. I'm blown away by this. How is that even possible? What if the concurrency is one? Build. "How do we connect our
app to a running app to extract data to have
something similar to DevTools?" Oh. "Can you make any tutorials on
creating a DevTools replica?" Well, so you can-- basically, the
truest answer to this is I don't really
know the answer to how to make a DevTools replica. But I can tell you some
surrounding knowledge that I have. One, you would need to
talk to the analysis engine, or the analysis
server in the engine. I don't even know how to connect
to those things, to be honest. Two, in terms of
wanting to do this, there's a whole team, like
a sub team in Flutter, of incredibly
talented engineers, who build the DevTools. And it will be an effort
to replicate any percentage of what they've done. If you're envisioning
something that is missing, you could potentially file
some feature requests. I think you'd probably get your
feature request faster than if you cloned-- if
you did your own. Also, all their code
is on GitHub, actually. Flutter DevTools GitHub. If you don't want
that non-answer, the real answer is all
somewhere in this repository. But I have never looked
at this repository. "What if your Dart server died? Can you check inside
the container logs?" Well, yeah, that is
one of the issues, no more logs ever show up. So finally, some more showed
up, because I redeployed. But nothing happens,
nothing happens after that. All right, I'm going
to refresh again. Wait, no. Let's start with marco. No because the page
never resolved the URL, didn't get saved. I don't know why
that's a feature. Marco. Polo. Great. Now, I can refresh. Cloud Run works. Good. I just spammed the
lungs that was wasteful. OK, now we refresh here. And we get a time. Lovely. Now, concurrency
one, does it help? Refresh marco. No. It doesn't do anything. Refresh hangs. Wait. No, that came back. Now, it hangs. We saw that once before, where
it would like do one more, and then hang. So it's actually
slightly intermittent, because it did a time
or two before not hang until the, like, third
query, instead of the second. I think we have video proof
of that in this very stream. I said, "this is hacking." I might have mumbled. Does anyone remember
what I said? "This is hacking." I don't remember
saying those words. But I'd love to
explain what happened when I said something that
sounded like "this is "hacking. If I even know what happened. This has been an exercise in
me not knowing what happened. All right, so we're
going to get all those. Yep, this is when I was
just spamming control-r on the marco file. That worked. Then we ran one query, and
then we got that second query, and then it died. Super weird. I have no idea. "Can you Zoom in
to Cloud Run log?" That's what we're
looking at right now. But, oh, zoom in, right. I'm zooming in on the wrong
screen, because I'm a genius. Yeah, sorry. Good call. Good call. Hey, Viktor. What have you seen? We have uncovered some
shenanigans, let me tell you. These are proper shenanigans. Viktor, would you be
surprised to learn that a simple HTTP request
also hangs in this state? It's not just an
attempt to load-- to run another query. I added this URL, Viktor. Wrap your mind around this. Marco just sends back polo. Little summer, pool shenanigans. Hangs. All right, well, I think
we can basically call this. And I have a bug to file. Does anyone have
any other ideas? Oh, Viktor, you had
said another thing. "Perhaps it used another
instance when it works twice." Oh, that's totally possible. But it would be
really surprising if Cloud Run scaled
so early, right? But that's totally possible. Oh, that's a good idea. That's a good idea. That's a good idea. We can dive into this more,
because I bet we're going to-- no, I don't bet. I have no idea. But we might get some resource. Are you going to
tell us anything? Cloud Run revision labels,
like, our vision name. I'm wondering if we're going
to get an actual worker ID that could be differentiate-able
to tell if it did scale, this is on Worker 5,
this is on Worker 12, you know, if you had
a ton of traffic. Revision name, location, ID, no. None of this it would seem like
it's going to differentiate. Oh, instance ID. Hey, hey, good stuff. OK, so this starts with 00F. Let's go up to the
same thing here. 00f, OK, and it ends with 0b1. And that ends with-- where'd it go? It is a different instance ID. Genius. Not surprised. I wonder if it just does
that when it launches as like some precaution. Interesting. Yeah, I totally agree. It's something with Google
Cloud Run, and the proxy, and the database. Another thing we
tried, Viktor, was getting rid of the private
IP on the database. So I put it-- sorry,
getting rid of the public. So I put it on a private
IP, I added it to a VPC. And then I made a VPC connector. And then I added Cloud Run
to the VPC via the connector. Didn't work. Didn't help. Randal, "print some sort
of process ID or hostname each time." Well, I think we did
end up getting that. Though, that would
have been what we would do next if the
instance ID wasn't in the logs. But we got in here
labels, instance ID. This ended in e3f, and if we
go up to the previous one that worked, as well-- and we're
like, why'd it work twice-- does not end in e3f. It's a different instance ID. Oh. These are good theories. These are real good theories. Proxy needs a public ID
according to the docs. Well, those docs are wrong. I think-- yeah, those
acts are just stale. It needs a public IP or to
be on a VPC with something, and then it can
access the private IP. I don't know why those
docs are so abridged. Huh. Huh. Huh. Well, we learned something. I have a ticket that I can
file that I believe will-- I wonder how else I could get-- how else could I get
that fatal error? Oh. Oh. Oh. Oh. Oh. Oh. Remember-- remember. But we're not getting-- I wonder if it's
like here somewhere. I wonder if it's somewhere
down here, right? But the problem is
if it errors here-- oh, this would be
like on the way out. OK, this is so odd. This is so odd. It would have to
be in it's not-- the first request
completes successfully. So it would be
extremely surprising if adding any print statements
to all of these try catches actually printed, right? Because if we go
to the logs, we see the final, like, all
done here log, right? So this means we win. That request is complete. OK, I have, like, a lot of
thoughts popping in my head here. So that request is complete. So the trailing try catches
that are not in that file-- are they in that file? Well, they're
literally highlighted. All these try catches
are not going to print. And then on the next request,
nothing before them prints. Nothing prints. So the proxy
crashing the instance is a quite good theory. And it would happen like
if that's what's going on, it must be happening completely
outside of our request response cycle. Oh man. Man, oh man. All right, someone's
going to have some-- Cloud Engineer is going to
have some digging to do. Viktor also says,
"I think it breaks when you do two requests on
the same connection possibly. The pool will reuse
the first connection. Or maybe fail to
close the connection." Yeah, I totally agree
with that theory. Absolutely. Yep, totally agree. I think that's what
it is in some manner. It's-- well, I don't even know. No, no, no, no, no, no. No, I think that's
actually not it. I think it's more back to this,
because marco doesn't even work on the next one. So it's like it doesn't
even get a chance to reuse the connection. It could be this one-- fails
to close the connection and fails in a fatal way. Or return it to the pool,
right, the whole point of the connection pool
is it doesn't close it. "Tried to use one connection. It still broke." So I did a similar thing with
setting concurrency to 1, is that what you did? Yeah. And we're still in that world. I set concurrency to 1. I'm guessing that's how
you tried one connection. This is just bananas. It does feel like there's
simply a networking issue. Something's going on. All right, well,
folks, we debugged. And we didn't fix the bug. We don't have a PR
to submit anywhere. But we have an issue to file. And if nothing else,
when people say an issue with a good
minimal reproduction is a huge contribution,
they mean it. Because you just
saw how long it can take to do enough
investigation to be able to file an issue with
a minimal reproduction. So without the
pool, still broke. Oh, that's a great test. So the pool has
nothing to do with it. It's totally in the-- it's got to be in that proxy. But you know what? This doesn't happen
in other languages. It could be a hand-off between
the proxy and the package, except it also doesn't happen
on Google Compute Engine. It only happens on Cloud Run. So it doesn't happen
in other languages. It doesn't happen in Python,
doesn't happen in JavaScript. If it did, then that
would be the biggest bug in the history of Google Cloud. So it's some handoff
between some, like, aligning of the stars
just on Cloud Run. Probably the raw Postgres
library, because Postgres still uses Postgres library, and
you just replicated this bug without the pool library. So some hand-off between
the Dart-Postgres library and the proxy. But it's somehow
activated only by-- wait. No, I have another test. I have another test
we have to run. (SINGING) I have another test. I have another test. La, la, la. I have another test. What am I doing? I wish I still had a public IP. I wonder if it's faster to
make a whole new database or switch back to public. Oh, maybe we can do both. I'll just add a public IP. That still will probably
take a long time. "Didn't use the proxy on GCE." That's exactly what
I want to test. I want to connect,
not via the proxy, via, instead, the public
IP from Cloud Run. Actually, I kind of vaguely
remember that doesn't work. I wonder if we can
get the private IP. Does anyone know how to get
the private IP of something on a VPC? I do not know how. But I kind of vaguely
remember that you are always blocked from using the
public IP on Cloud Run. This sounds familiar to me. "Have we determined
whether it's two requests on the same run dies,
or to requests quickly from two separate runs dies?" I think we have not. Well, let me think about
this before I just say words. Same run-- I don't
think the speed matters. I don't think quickly matters. So two requests, same run, or
two requests quickly from two separate runs. Oh, that's another
interesting point. What if we ran another
query in the first request. That's another great test. Yes. Let's duplicate this. And whatever. And now here we'll print. Request one done. And we'll see if it hangs
but we get request one done. This is an excellent test. "It seems really tricky to
use private IP on Google Cloud Run." Yeah, I vaguely remember this. OK, request one done. Did I break anything else
or can I just deploy this? I don't think I
broke anything else. Deploy. What do we theorize here? I think we theorize that
it's going to break, right? The first request here
is going to break. That's the theory. If the whole thing comes
back, we're shocked. I'm shocked at least. I won't tell you all to be
shocked, but I will be shocked. Yeah, I think this
isn't going to work. What would it mean if it does? It would mean that
there's some-- see I'm still using the pool. But you created-- you,
Viktor, replicated this bug without the pool. That means it's not, like, when
the pool puts the connection back in the pool, because
you recreated the bug whether that whole
system wasn't in play. So I don't think it's going
to be like some end of request cool down phase, because the
raw Postgres library wouldn't have any concept of that. It has no idea about
request response. So I think it's just going
to be either the closing-- Yeah, it's going to be the
closing of any connection, the termination of something. That's my theory, if
this all happens, then-- well, it actually
it carves away a ton of other possible theories. So it would zoom in
our investigation. We would have a smaller target
of where the bug could lie. But I don't think that's
what we're going to see. "Does Cloud Run
pause the instance as soon as it is
listening on a port?" Pause as soon as it's listening,
what do you mean, Viktor? Like, as soon as it's done
listening, like as soon as the request ends? Say that with twice
as many words. This could be a failure
point of the proxy. Yeah, it's true, it could. All right, let's refresh here. Is this going? So marco works. Show ourselves that. The app is deployed. Load, just once. Nothing. Nothing. But let's see if we got
a full round of logs, and we got that request one
complete, or whatever I wrote. Logs. Scroll to the bottom. Yeah, request one done. Wait a minute. But then we did get more stuff. And then we just got
resolved option one, and we're hanging on option two. Oh, I did actually
change the logic of this. I didn't appreciate
that earlier. I did change the logic
of this, because it never would ask option two in
a normal universe, where it loaded option one. What file was that in? Retry? Resolved? Yeah, here we go. So it didn't use
to try option two. It never used to call open
if it got something here. So I actually changed the
behavior of this code. So I will say here this will
be option one or option two. Now, I have to deploy again. And this will now
mean option one and option two will be the
same thing if option one loads. But that's fine,
because we're not even getting any intel about
these, just as instance of. Re-deploy. Viktor says, "I'm thinking
it can be an optimization. If the instance is paused,
but the load balancer listens for connections
and starts the instance when it detects a connection." I see. Yeah, so Cloud Run does
scale down to zero, but not that fast. It's actually a trade a trade
secret how much the rate at which it scales down. And I literally don't
know the answer. But I know that
it's not that fast. I think it's in the order of
tens of seconds or something before Cloud Run
kills an instance. But I genuinely don't know. But it's not a half a second. OK. Here we go. Here we go. It's loading. It's not loading. It's building. OK, this will complete
in a more good way the test that we tried
to run last time, which was Randal's idea. Randal, I, at first,
glossed over the wisdom in that statement. The quickly is just brilliant. We've never run two
queries in the same thing. Aren't we all? Aren't we all? Yeah. Yeah, yeah. All right, it's deployed. Marco works. Hello, polo. Still timing out. Refresh works. Whoa. Refresh hangs. This told us something. Well, the pool is still tricky,
because what could happen here is the pool could
say, you know, sure, use one of the connections
for this query, and then this is so soon after
that first query is done, and maybe the first connection
isn't back in the pool yet. Maybe this uses the
second connection. And then maybe the
problem still comes shortly after, when this
connection gets, ultimately, terminated, which
means the time is now to remove pool from our setup. So I'm going to end
all of our pool usage. And to do that, I'm going to go
to pub.dev packages Postgres, and I'm just going to
steal the connection info. Here we go. It's Postgres Google connection
and their mix of positional and named parameters. What is this? Start discussion. The heck did that come from? OK. So we're not using
endpoint anymore. We're using
PostgreSQL connection. And its host goes here. Oops. Host goes here. Oh, why did I cut this? What did I just do? Oh, I pasted when
I meant to cut. Brilliant. OK, Dart test. This is the name
of the database. So we'll grab this code. Let's make all of this format. OK, so the name of the
database goes here. The user goes here. All right, this one is a
named parameter, username. And then the password goes here. And I can't type. And then I bet this also
has an is socket connection. Yes, is UNIX socket. I literally can't type anything. This is crazy. Is UNIX socket is a same value. OK, so now we have a
PostgreSQL connection. And in our main-- funny thing here is-- so let's just make a variable
conn equals PG endpoint. Still not a great name for that. OK, so then how do we use this? Connection dot open and
connection dot query, sure. So connection dot
open we'll say here. And then we'll have
this variable up here, this will be late. PostgreSQL connection conn. So we have the connection. We open it in the main method. And then in here, we don't
do any of this anymore. In here, we're now going
to say conn dot query. And we pass in all
of this, right? No, what am I talking about? I just pass in select now. And this is going to return
that same list of lists. Yeah, absolutely. So now we're fine. Whatever, fine. Let's print the first one. So this is going to be-- I'll wait. And then we wrap
you in parentheses. And then we grab zero, zero. What's the problem here? What am I doing wrong? I don't think I want-- huh? What's the issue? I'm missing a closing
parentheses that's what it is. OK, what's your deal? Void? It can't be used,
because you're on print. Oh, that's the issue. There we are. Oh, no, you're still unhappy. Future Postgres result,
you're outside of the await. Oh, goodness. This goes here. OK. I'm really looking-- I'm looking good here. Making this look easy. "Have you tried closing the
pool at the end of the call?" Wish I would have read that
before I ripped the pool out. Hi, Simon. Help. So I'm going to do this. And then we're going
to run a second one. And then we're going to
return that two string. Well, we don't need to print
that the first query ended, because we'll just
see the time print. OK. And then there will be
some connection closing shenanigans we can play around
with at the end of this. But I think this
will at least work. Let's run this locally and
just make sure it works. So Dart bin server dot Dart. Looking good. Come over to local host. Refresh. Get a time. It keeps loading. Good, good, good. And it should be
printing these times. It is. Awesome. It's printing the times right
next to a log statement, which also has the time. So not confusing at all. And obviously, marco works. OK, so this isn't 100% broken. So that's a good start. So we can build and deploy. Soon, I'm going to have to
take this debugging offline. But I think we're in a pretty-- we're continuing to
make progress here. And I'm getting good
ideas from you all. So I appreciate it. All right, I think I'm going
to just reason about this while it deploys. So there's an open question
here of closing the connection, which I am not ever doing. I don't know the
implications of this. The connection is
opened in main. So when-- I mean, it does kind
of seem like it should stay open the whole time, right? And then it will just
close when the server ends. I think this is reasonable. There's probably
a more elegant way to close it like
when the server ends to make sure it closes and
the database doesn't think it's still open or something. But I think this isn't going
to get in the way of us testing this bug. So we open the connection,
we run two simple queries. And Viktor, you've said that
you've caused this to error. Train of Thought, we
are debugging a problem with talking to a Postgres
database on Cloud SQL from Cloud Run using the
Dart Postgres library. It works on Cloud Run to Cloud
SQL with other languages, so other libraries, and it
works with the library, it works with the
Dart Postgres library from Google Compute Engine. It is just the confluence of
Cloud Run, the Dart Postgres library, and Cloud SQL, they're
somehow aligning their stars to cause this problem. Trying to rule out who the
guilty party is and isn't. OK. What did I just do? I deployed the app. So now when I refresh
here, we'll see something. And that's that it's pending. OK, let's look at the logs. Current time is 42. OK, scrolling down to 42. There is no logs, because
I pretty much deleted all the print statements. It was supposed to print
the time, though, right? Right there. This is supposed
to print the time. And it is not printing
the time, why? No, nothing prints there. Does marco work? I should have marco first. Marcos still works. Now it hangs. That's probably a
different instance. I keep switching screens when
I just mean to switch tabs. So we're going to
see marco load once. There it is. And we never did get the time. And we get no error. Nothing. No error. All right, it seems like
it's with the proxy. This really seems like
it's with the proxy. I know some folks that
work on the proxy, and I'm going to talk to him. But I think that is enough-- we got enough to call it today. I'm going to file a bug
and talk to the guy. And the proxy is written in Go. He's like this super Go expert-- a couple of other
people on the team. But yeah, I know the folks
who work on the proxy, and I'm going to talk
to them about it. Try to call in a favor. Try to convince them that I
even have a favor to call in. OK. Man, this was an adventure. I don't know about you all, but
it was a wild ride on my end. A wild, wild ride. Yeah, Viktor, thanks for
making Serverpod and being a bit of a Guinea pig
on a fairly advanced GCP deploy that you were almost
going to have no issues with, but for this. A couple other
Config things, we're working out the kinks on there. But yeah, this is the real
showstopper for the Cloud Run flavor of Serverpod. Anyway, I'm going to try
to get it figured out, because I know that I also
want to be able to just deploy Dart code to Cloud Run
that talks to a database. And right now I can't do that. So that's not great. Well, folks, thanks
for tuning in. Thanks for all the ideas. I would still be
probably just staring out the window wondering what things
test without your good ideas, so this really was
quite a team effort. Man, honestly, kind of, I think,
maybe the most fun episode on my end. I don't think we're going to get
the viewership of interviewing Remy last week. But debugging with
the boys and the gals, doesn't get better than that. So OK, I'll be back next week. I've got a guest lined up. We'll be back in Dart
Google Cloud land. It's going to be fun. Stay tuned. Until then, see everybody.
