In this video, we'll use AWS Lambda, which is a service that allows us to upload code that should be executed upon certain events. And in this video, we'll upload some code that will execute whenever a new file is uploaded into a certain S3 bucket. And then, the code which we'll use in this video will simply look into that file that was uploaded and output the content of that file; simply because it is a basic demo, but of course we could also store that content in a database, or change that content, perform some analytics on that content, do whatever we need to do. In this video, however, we'll see how we create and use such a Lambda function, how we can configure it, and how we can connect it to a S3 bucket, which acts as an event trigger for this function. As always, as with all the other videos in this series of videos where I provide hands on examples, this video is not there to get you started with AWS, or S3 or Lambda, instead, it's there to show you how these services can actually be used. In order to get started with AWS and get a thorough overview of all its key services, you might wanna check out my Cloud Practitioner course which gets you started with AWS and the key services, and also helps you pass your first certification; the cloud practitioner certification, in case you're interested in that. Now, for this video, I also, again, prepared some code; the code we'll use in the Lambda function. And you find this code attached so that you can follow along. If you want to. Now, in this video, as mentioned, we'll use S3 and Lambda. And since I'll use S3 bucket as the trigger for a Lambda function, I'll first create that bucket on the S3 service page. Now, S3 is a data storage service that can be used to store objects, so files, in so called buckets. And those buckets can be exposed to the public, but are private by default. And there are many services you can use S3 in conjunction with, and there are many things you can do with S3, as explained in my course. Now here, I'll create a brand new bucket and give it a name of acad-demo1. And this name must be unique across all AWS accounts. So you might need to try a couple of names until you find a unique one. Typically, if you include your personal or company name, you have a higher chance of generating unique bucket names. And it's also a global service here. You don't pick a region in the top right corner, but you do pick a region here then. So it is a regional service, but you don't use the selection up here, but instead, this drop down. So that's where this bucket will be created and where the data will then be stored. Next, you have various settings related to who can access the data. And by default, all public access is blocked. And you would have to disable this to then add other settings to grant public access to this bucket. Now, I do show how you could create a public bucket in my course. In the end, you have to disable this here. So you have to disable this block all public access feature, and then you could add a bucket policy, which does grant access by anonymous users. That's something you could do if you would use this bucket to host a static website, for example, which is something you can also do with S3. It's not what we'll do here though. Instead, I want a private bucket which is only accessible by me. So, by this account. Hence, I'll keep this setting and I'll also not use any of the other settings here, and instead, create the bucket like this. Now, again, S3 is a powerful service. There are many things you can configure on this bucket, especially all the ones that was created. You can, for example, turn on replication to make sure that data in that bucket is automatically replicated into other buckets possibly in other regions, to make sure that even if that bucket would be lost, thanks to some catastrophe which hit the AWS data center that contained this data, it would still be there, or to deliver it faster to different users in different areas of the world. And that's just one of the many things you can do with S3. Again, I do cover it in greater detail in my course. Here, we have this basic bucket, which is all we need. And now I want to make sure that whenever I upload a file into this bucket, for example, through the management console, a certain AWS land function executes. So, some code is executed when I upload something here. Now, we could spin up an EC2 instance, run some process on that instance, and then in that process have some code that queries that bucket from time to time to see whether new files were uploaded. This would be rather cumbersome though, and that's why we have this Lambda service, because Lambda actually is a serverless service that allows us to upload code that should be executed whenever a certain event occurs. And there are all kinds of events you can listen to. So here in the Lambda console, we can create a new function. And we could use a blueprint to see some examples, but here I'll author it from scratch. Give it a name, like S3-upload. Choose the run time. So, the programming language that we want to use for the code that we'll upload. And in my case here, I got a Node.js application, hence, I'll use Node.js here. We can then also change the execution role, and I'll come back to that in a second. For the moment, I'll let it create a new role. And also enable some advanced settings, like adding a function URL, which would give this function a URL. And whenever we send a request to the URL, the function would be executed, which can also be great for hosting simple websites or micro services, for example, but here we don't need that. We can also attach or associate this function with a VPC, because by default, VPCs don't matter for Lambda functions, but you could do that if that Lambda function would need access to certain other services and things running in one of your VPCs. By default, a Lambda function has no access to your VPCs. But if it needed access, for example, to an RDS instance, you could use this enabled VPC feature here to associate this function with a VPC. And then, we can create this function. Now, once the function was created, you can write some code here right inside the management console. And that's the code that will be executed whenever this function is triggered. Now, we could write some simple code here, like a simple console lock statement, which prints a message to the console. We can then also test this code from right inside the console, and define a test event that we wanna simulate. Now, here, I don't really need any event data, but we could add some data that's passed along with the event that occurred, and this data could then be processed in the Lambda function. But here, I don't care about this, so I'll just give this a dummy name, save this test event, and execute the function with this test event. And then we see our output here. So that's how we could test our code and execute it upon a button click. Typically, though, we'll not execute code when we click a button, but when a certain event occurred. And that's what we set up up here. Here, for our Lambda function, we can add a trigger. And that is the event source, which we wanna attach to the Lambda function. Now, there are all kinds of things we can use as event sources, but S3 is one of the sources. And depending on which source you choose, you then get more options here. For S3, we have to choose which bucket should act as a trigger. And here, I select the bucket which I just created. And then, whether you care about object creation events, deletion events, which kinds of creation events and much more. And here, I'll simply select all object create events. You could then also filter by prefix, which has to be part of the file name of the file that was created in order to trigger this function, or look for files with certain extensions. So that, for example, this function only executes if JPEG files are added to the bucket, and other files would be ignored. Here, I don't care about either of these settings though, I only care about the bucket as a whole. And then, the function should execute no matter how a file was named or which extension it has. And we have to check this box, where we acknowledge that if our function would upload or create files in that bucket to which we're listening, we could cause an infinite loop. We are aware of that and we have to acknowledge that here, because that could lead to high costs. It won't be the case for this demo function though. So with that trigger attached, this function will execute whenever we upload a file to this bucket. It's also worth noting that in the bucket, if we inspect that bucket again, there is a special setting that was added under properties. There, under event notifications, this entry was added. Event notifications is basically a S3 feature that allows you to set certain recipients for certain S3 events. And the event notification that was set up here basically just says, that for this bucket, whenever objects are created, this function will be executed. So it's actually a setting in S3 that's changed to make this Lambda S3 connection work as intended. Now, you can also add a destination, which basically allows you to automatically send certain records to a destination of your choice. For example, to another Lambda function that should be invoked whenever this Lambda the function is done, but it's a feature which we don't need here. Now, besides choosing a trigger and defining the code, with which we're not done yet, by the way, there also are other things you can do here. You can, for example, monitor function execution, and also view logs in CloudWatch, which is another AWS service that helps with monitoring. Now, this CloudWatch log group and log stream, as it's called, was created automatically by AWS when this function was invoked. And here, you then see some log output for the function execution. Under configuration, you also got more settings you can tweak. For example, under general configuration, you control how much memory and storage is allocated to this function. Now, the more memory you allocate, the faster this function might execute, but the more you'll pay per execution. Though, you actually get a generous amount of free executions every month, but that amount is exhausted faster if you choose more memory. And for this example, this lowest possible memory value will do, which ensures that we won't pay for this example if you got no other functions in your account, at least. You can also set a time out after which the function will stop executing. And we might wanna set this to 10 seconds here, but it depends on the exact work you plan to do in a function, how much time you wanna set here. You just wanna make sure that you don't accidentally stop a function just because it ran out of time, even though the work it does simply takes a bit longer. The execution role is also important. That's a role that was created automatically when we created this function, and this role determines which permissions this function and the code and this function has. And if your code accesses another AWS service, for example, you must make sure that this role grants the appropriate permissions. It's actually something we'll have to tweak in a couple of minutes. Now, for the other configuration settings here, you again see your permissions and the role here under permissions. You see possible destinations, which I showed you a couple of seconds ago under the destinations area. If you enable the function URL feature, you see more related settings and information here, and you could, for example, also set environment variables that can be accessed inside of the code that's uploaded for this function. There also are more features, as you can see, many of them more advanced and not important for us here. I just also want to note that you could, for example, also attach an EFS file system. And I did cover EFS in an another video here, and of course, also in my course. But if your function code would need access to such a shared file system, it could get access because Lambda does support the EFS service. Now for us, the most important thing is that we add some code to this function. And here we could write the code inside of this interface here, inside of the management console, but I did prepare some code, which I want to use instead. And I will open that in my finder or in the windows Explorer, if you are on Windows. And here, you wanna select the index.js file and the node modules folder, which contains important dependencies of index.js, and bundle those together into a ZIP archive. If you don't have a node modules folder, you must run npm install inside of that lambda-S3-read-data folder as a first step. And that requires Node.js to be installed on your system. So once that ZIP file was created, you can upload this ZIP file, which contains the code. So choose that and upload that. And then this code, and also all the dependencies will be part of this Lambda function here. So that's then the code that will be executed when the function is invoked. So here's my code and my dependencies. And now, once this was deployed, this function is ready to be executed. Right now, however, it would fail, because in this function I'm actually accessing S3 to get some data from that file for which the function was executed. In my function code, I first of all extract some data from the incoming event, and that event is created and passed into the function automatically by AWS. And this event contains some information about the S3 object that was created, because keep in mind that this function is executed whenever an S3 object is created. Therefore, we automatically get an event with more information about that object by AWS. That event object which we get has various nested keys and ultimately it, for example, contains the name of the bucket for which the event occurred and the name of the file for which the event occurred. And then I use that data to read the file content and output the file content down there. Now, since I access S3 with this code, this function needs permission to access S3 and read S3 file data. And that permission can be granted with help of a role. Just as we did it in a number of video with an EC2 instance, which needed permission to access DynamoDB. There, we also created a dedicated role and then added the appropriate permissions through a policy to that role. We need to do the same thing here, but we already have a role. Under permissions, you see that we already have this auto created role. Therefore, now I just wanna make sure that we add the appropriate permissions to that role. So, here's that role. And now here, under permissions, we want to add more permissions. So here, we can click on add permissions and attach another policy. And I'll create a new policy here, which grants read access to S3. So here, I want to grant access to S3. And I want to grant read access, and specifically, the get object access. This action should be allowed. That's the only thing I need here. And I also wanna make it clear for which bucket this access should be granted. So here, we add an ARN, and it's the bucket name which we chose for our trigger. acad-demo1, in my case. I'll add this here. And then, I want to grant this access for all objects in this bucket. So this policy grants the get object permission for this bucket, and then for all the objects, all the files in the bucket. So, this is the permission I wanna create. I'll name it ReadDemoS3Objects. That's my policy name. And create this policy. And with the policy created, we can go back to this role here and add this newly added policy to that role. And that is the role that is attached to the Lambda function. And therefore, now the Lambda function has those extra permissions. Those read S3 object permissions for this bucket, and then all the objects in that bucket. So therefore, now with those permissions added, we don't need to do anything else because the role is already attached to this Lambda function. And since we added a new policy to the role, this function automatically has those permissions as well. And therefore, now we can give this a try. I'll go to the S3 bucket here. Into this bucket and upload a new file to it. And you can really upload any file here. It does not matter. In my case here, I'll upload the package.json file from this lambda-S3-read-data project here. So, this file here, I'll upload this package.json file. Click upload. And once it was uploaded and therefore created, this Lambda function should be executed. Now, we can tell whether it was executed by clicking on monitor. And we will see the executions here, but we can also click on view logs in CloudWatch to see those logs again. And as mentioned, a new log is added for every execution. So here, I see that log, and if I click into it, then here I have this content from the file being printed as part of the log, because that's what our function does. If you check the code, it ultimately logs, it outputs the content of the file. And therefore, that's what we see here. And this proves that this Lambda function did execute as intended for this uploaded file. And that's just one of the many things you can do with Lambda. Besides S3, there are many other sources you can add. You can also trigger it manually through the AWS SDK, for example, as part of a process running on an EC2 instance. You can also use SQS Or SNS as trigger sources. And these are other services I cover in my course. And Lambda in general is a highly versatile service that allows you to execute code upon the occurrence of certain events. And one Lambda function can also be connected to multiple event sources, of course, and you can therefore build highly flexible dynamic systems and microservices architectures with help of Lambda. Now, here in this case, I'm done. Therefore, I will delete this Lambda function here, because I don't need it anymore. And for S3, I will also delete that one file, which I uploaded from the bucket. Like this. And then under buckets, delete the bucket which is not needed anymore.
