Secrets of a WSGI master
17 likes13,367 views
The WSGI (Web Server Gateway Interface) specification for hosting Python web applications was created in 2003. Measured in Internet time, it is ancient. The oldest main stream implementation of the WSGI specification is mod_wsgi, for the Apache HTTPD server and it is 10 years old. WSGI is starting to be regarded as not up to the job, with technologies such as HTTP/2, web sockets and async dispatching being the way forward. Reality is that WSGI will be around for quite some time yet and for the majority of use cases is more than adequate. The real problem is not that we need to move to these new technologies, but that we aren't using the current WSGI servers to their best advantage. Moving to a new set of technologies will not necessarily make things better and will only create a new set of problems you have to solve. As one of the oldest WSGI server implementations, Apache and mod_wsgi may be regarded as boring and not cool, but it is still the most stable option for hosting WSGI applications available. It also hasn't been sitting still, with a considerable amount of development work being done on mod_wsgi in the last few years to make it even more robust and easier to use in a development environment as well as production, including in containerised environments. In this talk you will learn about many features of mod_wsgi which you probably didn't even know existed, features which can help towards ensuring your Python web application deployment performs to its best, is secure, and has a low maintenance burden.
![def application(environ, start_response):
status = '200 OK'
output = b'Hello World!'
response_headers = [
('Content-Type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]](https://image.slidesharecdn.com/secretsofawsgimaster-170806022036/85/Secrets-of-a-WSGI-master-4-320.jpg?cb=1504852163)
![Django framework integration
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'mod_wsgi.server',
] Add mod_wsgi.server](https://image.slidesharecdn.com/secretsofawsgimaster-170806022036/85/Secrets-of-a-WSGI-master-11-320.jpg?cb=1504852163)
![Build a container image
FROM python:3
RUN apt-get update &&
apt-get install -y --no-install-recommends apache2 apache2-dev locales &&
apt-get clean && rm -r /var/lib/apt/lists/*
RUN echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
RUN pip install --no-cache-dir mod_wsgi
WORKDIR /opt/app-root
COPY . /opt/app-root
EXPOSE 80
CMD [ "mod_wsgi-express", "start-server", "--port", "80", "--user",
"www-data", "--group", "www-data", "--log-to-terminal", "wsgi.py" ]
Install Apache
Fix Unicode Problems
Run mod_wsgi-express](https://image.slidesharecdn.com/secretsofawsgimaster-170806022036/85/Secrets-of-a-WSGI-master-18-320.jpg?cb=1504852163)
![A better container image #2
RUN pip install --no-cache-dir virtualenv &&
virtualenv /opt/app-root &&
. /opt/app-root/bin/activate &&
pip install --no-cache-dir warpdrive &&
warpdrive fixup /opt/app-root
WORKDIR /opt/app-root
COPY . /opt/app-root/src
RUN warpdrive fixup /opt/app-root/src
USER 1001
RUN warpdrive build &&
warpdrive fixup /opt/app-root
EXPOSE 8080
CMD [ "warpdrive", "start" ]
Create a Python
virtual environment
Use warpdrive,
it's magicUse non
root user
Non
privileged
port](https://image.slidesharecdn.com/secretsofawsgimaster-170806022036/85/Secrets-of-a-WSGI-master-24-320.jpg?cb=1504852163)
![Request Monitoring
import mod_wsgi
def event_handler(name, **kwargs):
if name == 'request_started':
...
elif name == 'request_finished':
environ = kwargs['request_environ']
response_time = kwargs.get('response_time')
cpu_user_time = kwargs.get('cpu_user_time')
cpu_system_time = kwargs.get('cpu_system_time')
...
elif name == 'request_exception':
...
mod_wsgi.subscribe_events(event_handler)](https://image.slidesharecdn.com/secretsofawsgimaster-170806022036/85/Secrets-of-a-WSGI-master-39-320.jpg?cb=1504852163)
Secrets of a WSGI master
- 1. Secrets of a WSGI master. Graham Dumpleton Graham.Dumpleton@gmail.com @GrahamDumpleton https://www.slideshare.net/GrahamDumpleton/secrets-of-a-wsgi-master
- 2. What is WSGI? Web Browser Web Browser Web Browser Web Server HTTP HTTP HTTP File System (Static Files) Python Web Application WSGI WSGI == Web Server Gateway Interface (PEP 3333)
- 3. WSGI is a specification for an Application Programming Interface WSGI is NOT a wire protocol WSGI is NOT an implementation of any anything
- 4. def application(environ, start_response): status = '200 OK' output = b'Hello World!' response_headers = [ ('Content-Type', 'text/plain'), ('Content-Length', str(len(output)))] start_response(status, response_headers) return [output]
- 5. Friends don’t let friends use raw WSGI
- 7. You still need a way to host a WSGI application The development servers builtin to a framework are not good enough
- 8. Installing mod_wsgi the easy way pip install mod_wsgi https://pypi.python.org/pypi/mod_wsgi
- 9. Run mod_wsgi from the command line mod_wsgi-express start-server wsgi.py
- 10. No Apache configuration required $ mod_wsgi-express start-server wsgi.py Server URL : http://localhost:8000/ Server Root : /tmp/mod_wsgi-localhost:8000:502 Server Conf : /tmp/mod_wsgi-localhost:8000:502/httpd.conf Error Log File : /tmp/mod_wsgi-localhost:8000:502/error_log (warn) Request Capacity : 5 (1 process * 5 threads) Request Timeout : 60 (seconds) Startup Timeout : 15 (seconds) Queue Backlog : 100 (connections) Queue Timeout : 45 (seconds) Server Capacity : 20 (event/worker), 20 (prefork) Server Backlog : 500 (connections) Locale Setting : en_AU.UTF-8
- 11. Django framework integration INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'mod_wsgi.server', ] Add mod_wsgi.server
- 12. Run with Django management command python manage.py runmodwsgi
- 13. Automatic code reloading python manage.py runmodwsgi --reload-on-changes
- 14. Interactive debugger python manage.py runmodwsgi --enable-debugger
- 15. Production configuration #1 python manage.py runmodwsgi --server-root /etc/wsgi-port-80 --user www-data --group www-data --port 80 --setup-only
- 16. Production configuration #2 mod_wsgi-express start-server wsgi.py --server-root /etc/wsgi-port-80 --user www-data --group www-data --port 80 --url-alias /static static --setup-only
- 17. Running in production /etc/wsgi-port-80/apachectl start /etc/wsgi-port-80/apachectl restart /etc/wsgi-port-80/apachectl stop
- 18. Build a container image FROM python:3 RUN apt-get update && apt-get install -y --no-install-recommends apache2 apache2-dev locales && apt-get clean && rm -r /var/lib/apt/lists/* RUN echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen && locale-gen ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 RUN pip install --no-cache-dir mod_wsgi WORKDIR /opt/app-root COPY . /opt/app-root EXPOSE 80 CMD [ "mod_wsgi-express", "start-server", "--port", "80", "--user", "www-data", "--group", "www-data", "--log-to-terminal", "wsgi.py" ] Install Apache Fix Unicode Problems Run mod_wsgi-express
- 19. Building the container $ docker build -t mypyapp . Sending build context to Docker daemon 3.584kB Step 1/9 : FROM python:3 ---> 968120d8cbe8 Step 2/9 : WORKDIR /opt/app-root ---> Using cache ---> 003096a40d39 Step 3/9 : .......
- 20. Starting the container $ docker run --rm -p 80:80 mypyapp
- 21. Friends don’t let friends run containers as root
- 22. Friends don’t let friends use Python without a Python virtual environment
- 23. A better container image #1 FROM python:3 RUN apt-get update && apt-get install -y --no-install-recommends apache2 apache2-dev locales && apt-get clean && rm -r /var/lib/apt/lists/* RUN adduser --disabled-password --gecos "Warp Drive" --uid 1001 --gid 0 --home /opt/app-root warpdrive && chmod g+w /etc/passwd RUN echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen && locale-gen ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 PATH=/opt/app-root/bin:$PATH HOME=/opt/app-root Create non root user
- 24. A better container image #2 RUN pip install --no-cache-dir virtualenv && virtualenv /opt/app-root && . /opt/app-root/bin/activate && pip install --no-cache-dir warpdrive && warpdrive fixup /opt/app-root WORKDIR /opt/app-root COPY . /opt/app-root/src RUN warpdrive fixup /opt/app-root/src USER 1001 RUN warpdrive build && warpdrive fixup /opt/app-root EXPOSE 8080 CMD [ "warpdrive", "start" ] Create a Python virtual environment Use warpdrive, it's magicUse non root user Non privileged port
- 25. Running as non root $ docker run mypyapp warpdrive exec id uid=1001(warpdrive) gid=0(root) groups=0(root) $ docker run -u 100001 mypyapp warpdrive exec id uid=100001(warpdrive) gid=0(root) groups=0(root) Default to assigned non root user Can be run as arbitrary high user ID
- 26. Same tools for development $ warpdrive project mypyapp (warpdrive+mypyapp) $ warpdrive build (warpdrive+mypyapp) $ warpdrive start https://pypi.python.org/pypi/warpdrive
- 27. Generate image with no Dockerfile $ warpdrive image mypyapp $ docker run --rm -p 80:8080 mypyapp
- 28. Source-to-Image $ s2i build https://github.com/GrahamDumpleton/warpdrive-django-modwsgi getwarped/warp0-debian8-python35 mypyapp https://github.com/openshift/source-to-image
- 29. OpenShift $ oc new-app --image-stream getwarped/warp0-debian8-python35 --code https://github.com/GrahamDumpleton/warpdrive-django-modwsgi --name mypyapp https://www.openshift.com
- 30. Manual Apache configuration $ mod_wsgi-express module-config LoadModule wsgi_module "/.../venv/.../mod_wsgi/server/mod_wsgi-py27.so" WSGIPythonHome "/.../venv"
- 31. Friends don’t let friends use embedded mode of mod_wsgi
- 32. Embedded mode
- 33. Daemon mode
- 34. Manual daemon mode configuration WSGIRestrictEmbedded On WSGIDaemonProcess mypyapp python-home=/.../env WSGIScriptAlias / /.../src/wsgi.py process-group=mypyapp application-group=%{GLOBAL} <Directory /.../src> <Files wsgi.py> Require all granted </Files> </Directory>
- 35. The missing options • display-name='%{GROUP}' • lang='en_US.UTF-8' • locale='en_US.UTF-8' • startup-timeout=15 • connect-timeout=15 • socket-timeout=60 • queue-timeout=45 • request-timeout=60 • inactivity-timeout=0 • restart-interval=0 • maximum-requests=0 • shutdown-timeout=5 • deadlock-timeout=60 • graceful-timeout=15 • eviction-timeout=0 http://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html
- 36. Failed application loading • startup-timeout=15 Defines the maximum number of seconds allowed to pass waiting to see if a WSGI script file can be loaded successfully by a daemon process. When the timeout is passed, the process will be restarted.
- 37. Connection timeouts • connect-timeout=15 Defines the maximum amount of time for an Apache child process to wait trying to get a successful connection to the mod_wsgi daemon processes. This defaults to 15 seconds. • socket-timeout=60 Defines the timeout on individual reads/writes on the socket connection between the Apache child processes and the mod_wsgi daemon processes. If not specified, the number of seconds specified by the Apache Timeout directive will be used instead. See also response-socket- timeout if need to control this only for writing back content of the response. • queue-timeout=45 Defines the timeout on how long to wait for a mod_wsgi daemon process to accept a request for processing. Not enabled by default.
- 38. Time triggered restart • request-timeout=60 Defines the maximum number of seconds that a request is allowed to run before the daemon process is restarted. • inactivity-timeout=0 Defines the maximum number of seconds allowed to pass before the daemon process is shutdown and restarted when the daemon process has entered an idle state. To restart on stuck requests use request-timeout instead.
- 39. Request Monitoring import mod_wsgi def event_handler(name, **kwargs): if name == 'request_started': ... elif name == 'request_finished': environ = kwargs['request_environ'] response_time = kwargs.get('response_time') cpu_user_time = kwargs.get('cpu_user_time') cpu_system_time = kwargs.get('cpu_system_time') ... elif name == 'request_exception': ... mod_wsgi.subscribe_events(event_handler)
- 40. Resources • mod_wsgi - http://modwsgi.readthedocs.io • warpdrive - http://warpdrive.readthedocs.io • Source-to-Image - https://github.com/openshift/source-to-image • OpenShift - https://www.openshift.com
- 41. Friends don't let friends use Windows for running Python web applications
- 42. Friends don't let friends use the mod_wsgi which comes packaged with the operating system
- 43. Friends don't let friends use those other WSGI servers
- 44. Friends don't let friends make things too complicated, simple is good