This documentation is related to the version 1 and 2 of the Cytomine platform. For the recent versions, please follow this related documentation.
List of the related main concepts :
- Software : An external script that can be launched by Cytomine for various purposes: AI model building, AI model exploitation for data inference, data mining, statistics, ...
- Job : A job is an instance (a run) of a software. A same software can have multiple runs with different parameters
- Parameter : There is two types of parameters : a software parameter is a declaration of a variable used by the software and a job parameter is related to a software parameter and contains the value of this variable.
- Software router : A component of Cytomine that will launch the softwares.
When you can use the Cytomine clients to interact with Cytomine and create various scripts, it can be useful to integrate these scripts into the Cytomine platform and thus allowing end-users to run them through the Web interface.
On this page, we will see how to integrate a script into the Cytomine platform.
To be able to plug your script into the Cytomine platform you will need to
- use a command-line terminal
- have an SSH (and SCP) access to the server hosting your Cytomine
- modify and compile a Dockerfile. You will find the Docker documentation here
In this article, we will use this sample, in Python, that will list the images into a project and create a file, that will be uploaded in Cytomine, containing the value of an integer input parameter.
Once the script is created, tested and validated, identify the wanted parameters that the users will be able to set.
To be able to allow end users to run your program, we need to add meta-information about it into the platform.
To do so, we are currently using an external client and declaring the information programmatically.
For our example, we add the Software and SoftwareParameter using the Python client and declaring the name of the software and its command line to launch our software and the name, type (integer of string or ...) and default values of each software parameter.
Note that the execution command will always reference your script in the following way : algo/DEDICATED_FOLDER_OF_YOUR_SCRIPT/YOUR_SCRIPT
As out example is in Python, the beginning of the command is python algo/test_soft/test.py
The script can be find here
We run this script with the command
As your scripts will need some dependancies, we need to add them to the Docker container that will run your scripts.
The easiest way is to start from this Dockerfile and to extend it to fit your needs.
Then you will need to compile the Docker image
sudo docker build -t cytomine/software_router:my_organisation .
As you're now using a custom software router, it is mandatory to modify the start_deploy.sh script and modify the line.
docker create --name software_router \ ... cytomine/software_router:v1.1.1 > /dev/null
docker create --name software_router \ ... cytomine/software_router:my_organisation > /dev/null
Now, Cytomine know how to launch your script and has the needed dependancies but the script itself is still missing.
We need to copy it on the server that hosts the software router container (classically with a SCP command).
Once the file on the server, we move it into the folder mapped between the container and the host (usually /data/algo).
In our example, and our previous declaration to the Cytomine platform, we will move the file into the /data/algo/test_soft folder after creating the test_soft folder.
Restart the container to apply the previous changes by running
sudo docker stop software_router sudo docker rm software_router sudo sh start_deploy.sh
You can now browse the Web UI of Cytomine and go to the analysis panel of a project space.
If your software is not yet associated with your project, you will be able to associate them in the Algorithms tab in the Configuration panel of your project space.
Click on the Launch new job button to run a job based on your software by selecting the wanted software
Each job will store, in the container, its logs in the /software_router/algo/logs/ folder in a file with a timestamp in the filename.
In our example, a file log could be /software_router/algo/logs/queueSoftwareTest_Software-6-1-2020_10-24-56-056.log
In the same way, the software_router Java program will log if it received the message from the core component.
So, to consult the two mains logs, use the following commands
#enter in the container sudo docker exec -it software_router bash #log of the software_router program cat /software_router/log4j.log #search for the log of your job ls /software_router/algo/logs/queueSoftwareTest_Software-*.log # then display the related log