pycos: Concurrent, Asynchronous, Distributed, Communicating Tasks with Python

pycos is a Python framework for concurrent, asynchronous, network / distributed programming and distributed / cloud computing, using very light weight computational units called tasks. pycos tasks are created with generator functions similar to the way threads are created with functions using Python’s threading module. Programs developed with pycos have same logic and structure as programs with threads, except for a few syntactic changes - mostly using yield with asynchronous completions that give control to pycos’s scheduler, which interleaves executions of generators, similar to the way an operating system executes multiple processes.

Unlike threads, creating tasks with pycos is very efficient (see below). Moreover, with pycos task context switch occurs only when tasks use yield (typically with an asynchronous call), so there is no need for locking and there is no overhead of unnecessary context switches.

Unlike with other asynchronous frameworks, programming with pycos is rather straight forward; Asynchronous Concurrenct Programming (pycos) describes 4 simple steps to convert programming with threads to programming with pycos.

pycos stands for “Python concurrent tasks”, “pico os” (tiny operating system), “pico tasks” (very lightweight computation units).

Features

pycos features include:

  • No callbacks or event loops (required in other asyhcornous frameworks)! No need to lock critical sections either,

  • Efficient polling mechanisms epoll, kqueue, Windows I/O Completion Ports (IOCP) for high performance and scalability,

  • Asynchronous (non-blocking) sockets and pipes, for concurrent processing of I/O,

  • SSL for security,

  • Asynchronous locking primitives similar to Python threading module,

  • Asynchronous timers and timeouts,

  • Message passing for (local and remote) tasks to exchange messages one-to-one with Message Queue Pattern or through broadcasting channels with Publish-Subscribe Pattern,

  • Location transparency with naming and locating (local and remote) resources,

  • RPS (Remote Pico/Pycos Service) for defining services that remote clients can run as tasks.

  • Distributing computation components (code and data) to execute Distributed Communicating Tasks (dispycos), for wide range of use cases of distributed / parallel computing, covering SIMD, MISD, MIMD system architectures at the process level; in-memory processing, data streaming, real-time (live) analytics and Cloud Computing are supported as well,

  • Web interface to monitor cluster/application status/performance,

  • Monitoring and restarting of (local or remote) tasks, for fault detection and fault-tolerance,

  • Hot-swapping of task functions, for dynamic system reconfiguration,

  • Thread pools with asynchronous task completions, for executing synchronous tasks, e.g., external library calls, such as reading standard input.

pycos works with Python 2.7+ and Python 3.1+ and tested on Linux, Mac OS X and Windows; it may work on other platforms (e.g., FreeBSD and other BSD variants) too. pycos works with PyPy as well.

For reference purposes, pycos with Python 2.7 on Raspberry Pi Model B+ (compliment by James Philips of pyeq2 project) running the program:

import time
import pycos
def task_proc(task=None):
    yield task.suspend()

tasks = [pycos.Task(task_proc) for i in xrange(10000)]
time.sleep(5)  # wait for tasks to run (and suspend)
for task in tasks:
    task.resume()

for line in open('/proc/self/status'):
    if line.startswith('VmPeak'):
        print('VM Peak: %d MB' % (int(line.split()[1]) / 1024))
    elif line.startswith('VmHWM'):
        print('VM Max: %d MB' % (int(line.split()[1]) / 1024))
    elif line.startswith('VmStk'):
        print('VM Stack: %d MB' % (int(line.split()[1]) / 1024))

shows that running 10,000 tasks takes about 23 MB of memory (with about 10 MB taken by modules used in pycos). Memory and time scale linearly: For 100,000 tasks memory used is 116 MB, for 200,000 tasks it is 218 MB, for 300,000 tasks it is 315 MB.

Dependencies

pycos is implemented with standard modules in Python.

Node / Servers sends node availability status (availability of CPU as percent, memory in bytes and disk space in bytes) at pulse_interval frequency if psutil module is installed. This information may be useful to clients, for example, to analyze application performance, to filter nodes based on available resources etc.

Under Windows efficient polling notifier I/O Completion Ports is supported if pywin32 is available; otherwise, inefficient ‘select’ notifier is used.

For IPv6, netifaces module is required with OS X. netifaces module is strongly recommended with Linux and Windows as well (for both IPv4 and IPv6). With Windows and Python 2.7 win_inet_pton module is required for IPv4 with multicast UDP or for IPv6.

Download/Installation

  • pycos package is available in Python Package Index (PyPI) so it can be installed with:

    python -m pip install pycos

  • Docker Container describes how to build Docker image and run pycos modules in containers.

  • pycos can also be downloaded from Sourceforge Files. Starting with 4.6.6 release, the download file is PyPI package so it can be installed with:

    python -m pip install pycos-<version>.tar.gz

  • pycos development is hosted at github. If necessary to install from source, download entire source and build package with python setup.py sdist at top level directory, which generates pycos-<version>.tar.gz file in dist directory that can be installed with python -m pip install dist/pycos-<version>.tar.gz --upgrade.

Release Notes

Short summary of changes for each release can be found at News. Detailed logs / changes are at github commits.

Examples

Examples illustrating some of the features of pycos are installed in ‘examples’ directory under where pycos module is installed, which can be obtained with the program:

import os, pycos
print(os.path.join(os.path.dirname(pycos.__file__), 'examples'))

See README file in that directory for brief description of each of the files.

SupportNew

Post issues, discuss at forum.

Contents

Indices and tables