I have my project file siting in the path /Workspace/folder1/subfolder1/myproject

this is how I have structured my project files

Here baseMLPipeline is my entry file where I am importing the file in the modules directory. I have .py files along with init.py file in modules folder.

When I run the baseMLPipeline notebook on databricks ,it works fine and is importing the custom package successfully. However, when I trigger this notebook from ADF(using job cluster), it throws “ModuleNotFound” error. Can you please suggest if there is something that I a missing or doing wrong? 

The code thows the error at line where I am importing the custom package

from modules import test_1

Below are the steps that I have already tried:

1. Moved the modules folder in same directory containing the main notebook i.e. /Workspace/folder1/subfolder1/myproject .
2. Created an empty file called init.py in the same directory as my .py files.
3. In the notebook, added the following code at the beginning to add the Workspace/folder1/subfolder1/myproject folder to the Python path:

import sys
sys.path.append("Workspace/folder1/subfolder1/myproject")