Celery no longer officially supports Windows since Celery version 4.x. And while Celery 3 does support Windows, it is not compatible with Celery 4. So if you have to resort to Windows for some (one) of your Celery tasks, you are stuck with a legacy Celery version across your infrastructure. Which is certainly not an acceptable situation.

However, even though Celery dropped Windows support, I’ll show you two simple workarounds to make Celery 4 play nicely on Windows.

The Celery concurrency pool

The first strategy to make Celery 4 run on Windows has to do with the concurrency pool. In a nutshell, the concurrency pool implementation determines how the Celery worker executes tasks in parallel. Celery defaults to the prefork implementation which spawns processes (and is limited to a handful of processes per CPU), whereas Eventlet spawns threads (hundreds of them, without breaking a sweat).

Celery comes with a number of concurrency pool types to choose from:

• prefork
• eventlet
• gevent
• solo

The Prefork pool is better suited for CPU-bound tasks while the eventlet pool works better if you’re I/O bound. Discussing the different options in-depth is another task for another blog post, in the meantime I recommend checking out the docs about concurrency and concurrency with Eventlet.

Strategy 1: Celery on Windows with eventlet, gevent or solo

What makes Celery 4 incompatible with Windows is actually just the default prefork concurrency pool implementation. In other words, if your Celery-job-to-be-done copes well with eventlet, gevent or solo (solo is a blocking single-threaded execution pool), you can run Celery 4 on Windows with any of these execution pools.

Clone the GitHub repository, create a virtual environment and install the pip requirements:

C:\Developer\celery-4-windows>virtualenv celery-4-windows
C:\Developer\celery-4-windows>activate celery-4-windows
(celery-4-windows) C:\Developer\celery-4-windows>pip install -r requirements.txt

You can start the Celery worker with any of these pool arguments:

(celery-windows) C:\Developer\celery-4-windows>celery worker --app=app.app --pool=eventlet --loglevel=INFO
(celery-windows) C:\Developer\celery-4-windows>celery worker --app=app.app --pool=gevent --loglevel=INFO
(celery-windows) C:\Developer\celery-4-windows>celery worker --app=app.app --pool=solo --loglevel=INFO

Open a new command line window to execute a task asynchronously and your Celery worker is back in Windows business:

C:\Developer\celery-4-windows>activate celery-4-windows
(celery-4-windows) C:\Developer\celery-4-windows>python app.py

Strategy 2: FORKED_BY_MULTIPROCESSING

If we dig a bit deeper, it turns out that the reason the default prefork concurrency pool implementation does no longer work on Windows, is because of the Celery billiard package. Billiard itself is a fork of the Python mulitprocessing package with some fixes and improvements.

Billiard used to set the not-so-well documented environment variable FORKED_BY_MULTIPROCESSING=1 by default. Celery in turn checks if FORKED_BY_MULTIPROCESSING is set to determine whether forking is disabled (it’s an OS thing). Since the billiard version Celery 4 depends on, billiard no longer sets FORKED_BY_MULTIPROCESSING which in turn causes the prefork pool to fail on Windows (have a look at the prefork source code and billiard change log).

To cut a long story short, you can work around the problem by setting a Windows environment variable. Go to: System Properties => Environment Variables => User or System variables => New…:

Variable name: FORKED_BY_MULTIPROCESSING
Variable value: 1

Open a new command prompt window to pick up the new environment variable. You can start the Celery worker without the pool argument:

C:\Developer\celery-4-windows>activate celery-4-windows
(celery-windows) C:\Developer\celery-4-windows>celery worker --app=app.app --loglevel=INFO

Open a new command line window to execute a task asynchronously and your Celery worker just works with the default prefork pool (which is actually forked by multiprocessing):

C:\Developer\celery-4-windows>activate celery-4-windows
(celery-4-windows) C:\Developer\celery-4-windows>python app.py