I am totally lost on this one and hoping someone can help. I am using Celery-Redis to execute a long running task for my Django project. The setup works perfectly on my local Windows setup. I have been working on porting this to AWS EC2 for last few days with Nginx/Gunicorn running on Ubuntu. And at this point, everything seems to be working except the background task execution. I have installed Redis and Celery on the same EC2 instance where I have the webserver running. Both (Redis/Celery) start without issues and they acknowledge each other (connection is established). Celery even shows the registered tasks. However, whatever I do, the task does not get executed when the request is made from Django. I have put logging everywhere in the code and can see the task (executed with .delay()) being called but after that everything stops. The call never returns. No error messages either in Nginx or Gunicorn logs. Celery and Redis-server both are running on the terminal with logging on so that I can see any messages. No messages in any of them as if they are not called at all.
I have also tried to execute the task without .delay just to check. And in that case, the task executes without issues. Nginx does timeout after 60 seconds, so my call never finishes but that is expected. The moment I add .delay(), the same behavior returns.
I have reviewed all of existing stackoverflow posts which seemed related but nothing helped. In one of the questions, I saw someone saying that this problem got resolved for them when they changed the redis url from localhost to 127.0.0.1. I tried that also but it didn’t make any difference.
I have also tried to play with EC2 permissions in case that is the issue. So I created an IAM role with administrative access and added that to the EC2 instance. That has not helped either.
# project init.py
from .celery_app import app as celery_app
__all__ = ("celery_app",)
# tasks.py
@celery_app.task()
def call_ai(user_request):
# Redis URL
REDIS_URL=redis://127.0.0.1:6379/0
CELERY_BROKER_URL=REDIS_URL
