in ,

Opnieuw: A simple and intuitive retrying library for Python, Hacker News


In this post, we want to introduce our new, easy to use, intuitive and simple retry package: Opnieuw , which is responsible for all the network retries in Channable ‘s Python code.

Opnieuw is a general-purpose retrying library that simplifies the task of adding retry behavior to both synchronous as well as asynchronous tasks.

Channable is a feed processing tool that imports products from webshops and exports those to marketplaces, price comparison sites, and advertisement platforms. At Channable we consume a lot of external APIs for our daily operations. If you make enough API calls, even the most robust API will sometimes fail. By retrying our request we can usually solve this problem. Therefore, having a good retry mechanism is important for making our operations run smoothly.

Why we wrote Opnieuw

Before writing our own retry package, we tried several other retry packages. Unfortunately we fell into multiple pitfalls:

      Due to too many knobs, it is easy to configure a retry in a way that does not make sense.

    • For example, we had multiple places in our codebase where we used a combination of exponential backoff with a minimum and maximum delay, such that the effective delay would always be the minimum or the maximum; there was no exponential backoff going on. We solve this by providing just two parameters, where every combination of parameters is valid.
    • due to a modular approach, doing the right thing is hard . Most places in our codebase were not using jitter, simply because adding jitter is a separate step that was often forgotten. We solve this by providing only a single retry strategy: exponential backoff with jitter.

        Time units were unclear.

      When you see a line that reads

          @ retry  () wait_exponential_multiplier =

        does that mean that the base delay is 12 milliseconds, or seconds? Due to the nature of the APIs we interact with, reasonable wait times span several orders of magnitude, from milliseconds to minutes. We solve the unit problem by putting the unit in the parameter name. When we upgraded our codebase to our own retry decorator, we fixed multiple bugs where the original code confused seconds with milliseconds.

      • Parameters were unintuitive.

      • All retry libraries that we used, required configuring the base delay for exponential backoff. To determine the base delay, we always found ourselves working backwards from a total wait time and a maximum number of calls. In Opnieuw, we accept those more intuitive parameters directly, and we solve for the base delay.
          The meaning of parameters was unclear. When we checked the documentation of the libraries we used, we would find explanations like “Wait 2

    n base_delay between each retry. ”It is unclear here if that n starts counting at 0 or 1. With exponential growth, an off-by-one can make minutes difference after a few attempts. We solve this by providing more intuitive parameters, and by giving them a verbose but unambiguous name.

    Opnieuw is our take at avoiding those pitfalls, to make a retry library that is easy to use. Let’s look at an example in order to demonstrate the power of Opnieuw. Suppose we want to parse , and we want to retry on  ConnectionError  and 
     HTTPError . We can add Opnieuw to our network handler as follows:         import  requests   

    from requests.exceptions (import (ConnectionError) , HTTPError

    What do you think?

    Leave a Reply

    Your email address will not be published.

    GIPHY App Key not set. Please check settings

    Diabolical Fallout 76 Has a Make-or-Die Update Coming in April, Crypto Coins News

    Diabolical Fallout 76 Has a Make-or-Die Update Coming in April, Crypto Coins News

    Mosque Land 25 km From Ayodhya Ram Temple Complex – NDTV News,

    Mosque Land 25 km From Ayodhya Ram Temple Complex – NDTV News,