The concept of Flow-based programming (FBP) is to create a reusable software component that can be configured to get a task done. Flow-based programming is the best way to build scalable systems with ease. The easiest way to get into Flow-based programming is with Python. However, there are so many libraries and modules involved in this that the process of setting everything up can be a burden for some. Luckily we have done it for you and in this article, we will provide a list of not only the best libraries but also some excellent examples of what currently exists.

The Luigi library is a Python module that groups services and IPC connections into services.  Luigi helps you quickly and easily run a distributed application across multiple servers. You can use it to run your application across all your data centers, or even in the cloud.  Luigi also helps you debug and monitor your application, so you can quickly figure out when and why your application is misbehaving.

Luigi lets you focus on your automation needs in a single language. You can chain together Luigi tasks into scripts or tasks onto remote cores. This is done with no hassle of thinking about the plumbing, eg: how to connect to databases and naming/configuring all those pesky environment variables for each machine in some headless server farm. Instead, you can write your code once in Luigi and run it on any number of machines. 

One of the most challenging things in programming is to abstract away boilerplate and make it more readable. Pipelines are useful constructs in programming. They allow you to compose branching computations graphically and they help you model your computations as unidirectional data flows. Data science relies on the ability to acquire, manipulate and store data. The Pipeless library is capable of adapting to a data pipeline by providing a number of commonly-used methods. A major goal of the library is to prevent developers from having to write large amounts of boilerplate code that are often repeated when implementing data pipelines.

The papy package provides a robust implementation of the flow-based programming paradigm in Python. The papy framework enables the easy construction and deployment of distributed workflows that are capable of handling any variety of work unit data, including structured, semi-structured, and unstructured data. Work Units are handled by a reusable set of workflow operations called papywaf-* operations that have been carefully designed to handle any type of data. Furthermore, papy features a predefined set of operations that are accessed via an intuitive HTTP-based API that allows users to easily specify the input and output data of each operation, giving users the ability to deploy complex workflows.

Orkan is a pipeline parallelization library, written in Python. Making use of the multi-core capabilities of one’s machine in Python is often not as easy as it should be. Orkan aims to provide a plain API to utilize those underused CPUs of yours in case you need some extra horsepower for your computation. The library is built to be easily pluggable into different applications. A few examples, written with Orkan, can be found in the documentation and on the GitHub project page. While this library is somewhat of a niche, it is an easy and effective way to utilize your CPU cores. Orkan provides a plain, but efficient API to make use of your extra core power.

Pype is a simple, yet robust framework designed to handle data communication between multiple Ray processes. It is a lightweight library that has a simple API and is ideal for small- or large-scale applications. With Pype, you can create a multiprocess application without needing to spend time writing messages passing code. Pype allows you to easily define communication channels between multiple processes. Using the Pype framework means you can focus on more important aspects of your application such as your code rather than needing to write your own backend system for data communication.

Kamaelia is a framework that simplifies the way you build systems by providing you with a way to identify the right components, wire them up, and make them communicate. By doing this, you’re able to reduce complexity and build systems that are simpler to maintain and easier to understand. It does this by providing you with a simple ruleset for communicating between components and a way to identify components that you need to build your system. Kamaelia is open source software, which means you can use it free of charge and share it with the world.

The Ruffus python module provides a highly functional pipeline library that can be used to automate the process of implementing data analysis pipelines. The Ruffus library enables users to focus on their core data processing logic without having to worry about the gritty details of managing dependencies, cluster deployment, parallel job execution, re-starting from arbitrary points, and error management. Ruffus is designed to be lightweight and unobtrusive and to make the process of setting up a new workflow as easy as possible. At the same time, Ruffus does not try to impose any particular design on the workflow and is powerful enough to have been used for complex workflows involving more than 50 interdependent stages.

Pypeman is a library that frees you to focus on your use case. It does not require a specific database back-end or GUI toolkit. It is minimalist but pragmatic. It supports Python 2.7 and Python 3.4+. It has a single and coherent API that can be used to model any scenario that involves data transformations and data mapping. Pypeman has a permissive MIT license. Pypeman sources are hosted on Github.

Final Words

In this article, we got a basic introduction to Flow-based programming. We reviewed some of the best libraries that exist in Flow-based programming. With these libraries in hand, we are ready to continue on our journey into the world of Flow-based programming. We hope this article provided you with a great introduction to the topic of Flow-based programming and you have found some excellent resources that will help you as you continue on your journey into the world of Flow-based programming.