Uncategorized

Correct memory deallocation in C extension for Python


I implemented a function in C, that takes an array and creates vectors containing certain elements:

double** delay_vectors(const double* ts, int L, int tau, int p) {
    int i, j;

    // Allocate contiguous memory for v_i_list
    double **v_i_list = (double **)malloc((L-tau*(p-1)) * sizeof(double *));
    v_i_list[0] = (double *)malloc((L-tau*(p-1)) * p * sizeof(double));
    
    for (int i = 1; i < (L-tau*(p-1)); i++) {
        v_i_list[i] = v_i_list[i - 1] + p;
    }
    

    for (i = 0; i < L - tau * (p - 1); ++i) {
        for (j = 0; j < p; ++j) {
            v_i_list[i][j] = ts[i + j * tau];
        }
    }

    return v_i_list;
}

For this function, I created a python wrapper, which takes a numpy array and returns the 2d array to numpy again:

static PyObject* delay_vectors_wrapper(PyObject* self, PyObject* args) {
    PyObject *input_array;
    PyArrayObject *input_array_np;
    int tau, p;

    // Parse the input arguments
    if (!PyArg_ParseTuple(args, "Oii", &input_array, &tau, &p)) {
        return NULL;
    }

    // Convert the input object to a NumPy array
    input_array_np = (PyArrayObject *)PyArray_FROM_OTF(input_array, NPY_DOUBLE, NPY_ARRAY_IN_ARRAY);

    if (input_array_np == NULL) {
        return NULL;
    }


    int ndim = PyArray_NDIM(input_array_np);
    if (ndim != 1) {
        PyErr_SetString(PyExc_ValueError, "Input array must be 1-dimensional.");
        Py_DECREF(input_array_np);
        return NULL;
    }
    
    int L = PyArray_DIM(input_array_np, 0);

    // Get data pointer from the NumPy array
    double *ts = (double *)PyArray_DATA(input_array_np);

    double** result = delay_vectors(ts, L, tau, p);

    // Convert the C result to a np array
    npy_intp dims[2] = {L - tau * (p - 1), p};
    PyObject* output_array = PyArray_SimpleNewFromData(2, dims, NPY_DOUBLE, result);
    


    
    Py_DECREF(input_array_np);

    return output_array;
}

As you can see, I am not freeing any memory except for “input_array_np”. Freeing result would lead to an error, since, as I understand “PyArray_SimpleNewFromData” creates a wrapper around the data pointed to. I tested this function using valgrind and it shows no memory leaks. However, if I use this function in a loop e.g. in

ts = np.random.random(10**5)
for i in range(500):
   dv = test.delay_vectors(ts,2,3)

I can see, that the consumed memory increases. I suspect this is due to the fact that everytime the function is called, it allocates a new potion of memory to store result in, which does not get freed even if the array is overwritten in Python, but I sadly can´t think of another way to do it.

I tried giving ownership of output_array to numpy by using PyArray_ENABLEFLAGS(output_array, NPY_ARRAY_OWNDATA); before returning to get pythons garbage collector to free it, but I can`t seem to get it working. I am also very new to writing C-code, so if you have any further suggestions regarding memory allocation, I would appreciate it.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *