A tutorial on server-side deployment of a Python application without interruption _python

When you start deploying applications, the easiest way to do this is to restart My_app or all services with an administrator to upgrade the product to the current version. Everything is fine at first, but eventually you will find that once the application is started, attempting to connect during the reboot will result in numerous HTTP 503 errors.

Finally you may find that Gunicorn and Uwsgi can reload your application without closing the socket, so that when your application starts, the network request is just a little bit delayed. As long as your application does not take a long time to start up, it will work very well. Unfortunately, many of the existing applications may take 1 minutes to start, which is too long for a link waiting on the socket.

Gunicorn uses the Kill-hup $PID to reload by shutting down all worker processes and then starting them. However, the slow initialization process of a worker process often leads to problems. Uwsgi uses chained overloads, which start only one worker process at a time. I need support for tornado, which is not very suitable for uwsgi at the moment.

using Load balancers

A common technique is to remove a single server from the load Balancer, upgrade/Restart the application, and then load it back in. We are using the load balancer, but in order to schedule the entire process, we need to coordinate the use of haproxy to manage the socket in the configuration node. Our current deployment scenario is to deploy to all nodes at once, rather than one after another, a considerable change. While waiting for the lbs (load balancer) to move the node out of the pool, you can use the 404 ' ing status page to cheat the healthcheck. This is a little more than I wanted, and for each server, two Healthcheck failures are 5 seconds apart, which includes the time the Web process recovers after the upgrade completes.

Gunicorn overload + +

Gunicorn automatically restarts the failed Web process, so it may kill each process and hibernate in between until all of the child processes have finished executing. This works, but if the application starts to change significantly, we will either wait too long for the reboot, or wait for a long time and take some downtime risk.

Because Gunicorn contains a Python hook that points to an application, it is entirely possible to write a small piece of code that notifies the restart process when the worker process is ready. Gunicorn does not contain the required hooks, but making changes is very simple. It requires some modification before the new version is released.

The restart process now plays the fact that a single soket has multiple processes that accept connections. Rebooting only minimizes the service capability (1/n), but we can continue to process traffic without having to wait too long for the connection.

This kind of process is usually like this

For Child_pid of Gunicorn-master:
 kill Child_pid wait for
 app startup

My first version uses Shell and NC to listen for application-initiated UDP packets. Although integrating our process manager into the shell environment is a bit more troublesome than I expected, it works well.

The restart script should be invoked with Gunicorn PID, which is the masterrestart.sh $PID

Echo ' Killing children of ' $;
 
children=$ (Pgrep-p $) for child in
$children
do
 echo ' killing ' $child
 kill $child
 response=$ ( Timeout nc-w 0-ul 4012)
 if ["$response"!= ' OK ']; then
  echo ' broken '
  exit 1;
 Fi done

Post_worker_init the script in tandem so that the restart script is notified when the app is running.

Import socket
Import time
 
def post_worker_init (worker):
 _send_udp (' ok\n ')
 
def _send_udp ):
 
 udp_ip = "127.0.0.1"
 udp_port = 4012
 
 sock = Socket.socket (socket.af_inet, # Internet
       socket. SOCK_DGRAM) # UDP
 sock.sendto (Message, (UDP_IP, Udp_port)
If we have such a WSGI (Python Web Server Gateway Interface) Application:
 
from werkzeug.wrappers import Request, Response
 
@Request. Application
def application ( Request):
 resp = Response (' Hello world! ')
 if Request.path = = '/_status ':
  resp.status = ' OK '
 else:
  resp.status = ' 404 Not Found ' return
 resp

We can even check the/_status page to verify that the application is running.

def post_worker_init (worker):
 env = {
  ' request_method ': ' Get ',
  ' path_info ': '/_status ',
 }
 def start_response (*args, **kwargs):
  _send_udp (args[0])
 
 Worker.wsgi (env, Start_response)

Be careful not to try to run too many applications in this health test, if, for whatever reason, your post_worker_init generates an error, the worker process exits and prevents the application from starting. This can be a problem when you check for potentially defunct DB links, even if your application works, it won't start again.

Now with a one-minute application launch, we have implemented a rolling reboot without having to stop applying or discard any links!