Transfer Complex Python Objects via Sockets

what is going on guys welcome back in this video today we're going to learn how to transfer complex python objects via sockets so let us get right into [Music] it all right so we're going to learn how to transmit complex python objects via sockets and this is going to be a quite beginner friendly tutorial we're basically going to set up a simple client server architecture a client sending complex objects to a server the server receiving the object objects and using them on the server site and uh we're going to start here by first of all creating the two files client py and server py and those are of course two separate applications two separate scripts and in a real scenario they would be running on two different system so you would have one server running server piy one computer running client py and they could communicate via sockets also via the Internet if everything is set up properly everything is configured properly with firewalls and everything um in this case now I'm going to run on my system here so locally uh for demonstrational purposes now the idea is we're going to start first of all by sending a simple dictionary which is more than just a string but it's still a very simple python object and then once we have the basic code set up we're going to transmit a psychic learn machine learning model this is going to be our complex object and we're going to see how that works so we're going to start here by importing first of all the socket module we're also going to import pickle for serialization um and then basically the client is just going to send whatever object it has to the server so the client is going to say socket or maybe we should say client equal socket socket we want to have an internet socket so afin and we want to have a TCP socket so the uh connection oriented protocol which is addressed or chosen by sock stream uh and with this client now we want to connect to the server so we want to say connect or actually client. connect want to connect here to a tup pull off an IP address and a port now of course if you're running this on an actual server here you would specify the public IP address of the server or if you're running it in the same network the private IP address in my case it's all on the same machine so I'm just going to specify Local Host like this here so you can either provide the Local Host IP address which is this one or you can just type the word Local Host um and we're going to use port 9999 for this so the client is going to connect to a server that of course has has to be running um and then it's going to basically just send an object so we're going to say try my object is going to be a simple dictionary it's just going to have some keys key one value one key 2 value to very simple dictionary here and we want to send this now to the server now the sending part is actually quite simple so what we're going to do here in this case because we are not just going to send it as adjacent string object we're going to serialize it we're going to use pickle to serialize it into uh into bytes so we're going to say serialized is going to be equal to pickle dump s um and we're going to use my object here so we can actually see what this does maybe we can comment this out here I'm going to add the accept block here um we're just going to pass and this is basically what the serialized version looks like of this dictionary so this is what we're going to send via the socket now we're actually not going to do exception handling here because yeah we're just going to try and if it doesn't work it doesn't work but in the end definitely we want to close the socket so we're going to say client close so if you crash I still want you to close the client socket um and the sending on the client side is actually quite simple the whole logic and handling of the transmission happens on the server side because because the client for the client it's quite simple I have this object I know how large it is send it send all of it and this is actually the function that we're going to use client send all and we're going to send a serialized data uh that we created with pickle or we serialized with pickle so that is the client code very simple now on the server side we have the magic or I don't want to call it magic but we have the whole uh logic what is different here than when sending an ordinary string because usually what you can do is you can just say client send server receive in this case now it doesn't work because complex objects we assume that they're going to have a certain size and this is in fact going to be the case when we uh transmit the machine learning model maybe it's not the case for the dictionary but for the machine learning model it will be the case we need some more um we need to handle more bytes than we just receive with one receive chunk so we're going to sa here again import socket import pickle we're going to say server equals socket Socket Internet socket so again afin net TCP so sock stream and then we're going to bind the server to the same address um that we specified here in the client now again if you're running this in um on an actual server and an actual scenario here you would have the address you want to connect to so the public IP address or the private if it's the same network and here you want to have now the private IP address even if it's online even if it's an internet not in the same network here you always want to have the private IP address so we're going to in this case use Local Host as well because it's all on the same machine and the port of course has to be the same as well uh and we need to pass this as a two pole like this all right so the server is now bound we're going to say server uh listen we're going to allow for exactly one connection so once the connection is handled and closed we can open a new one but we're going to only allow for one connection at a time and then we're going to say while true so basically an endless loop what we want to do is we want to print first of all waiting for connection and then we want to accept what this connection has to offer so we want to say first of all I mean before we accept any content we want to accept the connection itself so the connection and the client address or usually what we do is we say client and address is going to be equal to server do accept basically meaning when there isn't connection attempt so when someone tries to do do connect uh onto the server we're just going to accept it and then what we want to do is we want to try to actually get the data so we can print for example connection or let's just say connected let's not print the address um and now what we want to do is we want to get all the data so we're going to say data first of all is an empty stream of bytes so B and an empty string and then we're going to get chunks of the data until there is no more data now we don't have here in the server we don't have something like receive all this function does not exist we don't have a receive all function we have a send all function and the reason for that is we know how large this object is the server doesn't know that the server doesn't know how large the object is it cannot just receive all it has to receive until there is nothing more to receive so basically again we're going to say while true the chunk that we want to receive is going to be connection or actually sorry client receive and we're going to receive for example 496 bytes you can also pick a different number but that is our chunk size now and if not chunk so if there is nothing we received after this command so if there is just no content in the chunk we're going to break out of this endless loop in here otherwise if there is something we're going to append whatever was in a chunk to the data like this um and then basically we have the bite stream and now we want to unpickle the by stream so we want to load from this by stream the actual content again so we want to say um received object is going to be equal to and then we're going to say pickle load s not load but load s and we want to get the data this is going to be the basis for all of this and we can print what we received as an F string here so received object whatever um and finally we want to close the connection so clients close now of course this is now going to keep running so this Loop is never terminated so it's going to always accept new um new requests unless we keyboard interrupt or just terminate the script so it's always going to be able to answer new new connections so we're going to run this now and you can see it says waiting for connection I can run the client now as well and you can see here on the server side received this so the client terminated immediately without any messages and here we have waiting for connection connected and then we received the actual thing and I can run the client now uh can I somehow split this actually there you go so I can run the client again and again and you can see that all of this is transmitted all right so this works now let's go ahead and actually transmit the complex object quote unquote uh which is going to be our psyched learn model and for this we're going to train a very simple model the focus is not going to be on the machine learning here I'm just going to train a basic model on the iris data set so we're going to say here from SK learn by the way if you don't have sklearn Pip or pip 3 install scikit-learn this is the command to get sklearn onto your system then from SK learn. dat sets import load Iris from skq learn. Ensemble import random forest classifier and from skq learn. model selection import train test split like this and then we're going to basically say data equals load Iris then we're going to say X and Y is equal to data data and data Target then we're going to say extra train X test y train y test is equal to the result of a trained test split on the X data on the Y data with a test size of 0 point and now just to see that we don't get perfect performance we're going to use half of the data for training half of the data for evaluating we're going to train them all here on the clients so we're going to say model is going to be a random forest classifier and the model is going to to be fitted on X train and Y train that's it so the model is strained here on the client side and then the model shall be sent to the server so what we actually want to do here is we want to remove that and we want to dump as the model and then on the server what we're going to do is we're going to also import uh psych learn so we want to say here from psyit learn data sets import for um or actually do we want to do that yeah actually we're going to use all of the data set for evaluation because I don't want to do some fancy uh data set transmission now so we're just going to train the we're going to train them all on half of the data and we're going to evaluate it on all the data again the focus is here on the socket transmission not on the machine learning so I'm going to say here again data equals loow Iris XY is equal to data data and um data Target and then what we're going to do is we're going to uh After we receive this object after we received them all we want to evaluate it so we want to say print accuracy and we're going to print here received object. score on all of X and all of Y so again run the server run the client and you can see we have here a 98% accuracy now there is some Randomness involved so this should change when we run it multiple times but you can see we get a good accuracy so we train them all on the client we transmit the model and we can actually see what is being transmitted by changing or by adding a line to the client where we show what serialized look uh looks like so this is what we're transmitting to the server so this is the complex quote unquote object that we're transmitting here the model so yeah this is how you transmit complex objects via sockets and python so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting a like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you in the next video and bye

Channel: NeuralNine

Views: 8,664

Keywords: sockets, socket, python, python socket, python networking, python tcp, tcp, tcp sockets

Id: xtdS7flpaoA

Length: 13min 35sec (815 seconds)

Published: Sun Nov 26 2023