Loading...
 

Create an external job for Cytomine

Remark

This documentation is related to the version 1 and 2 of the Cytomine platform. For the recent versions, please follow this related documentation.

Concepts

List of the related main concepts :

  • Software : An external script that can be launched by Cytomine for various purposes: AI model building, AI model exploitation for data inference, data mining, statistics, ...
  • Job : A job is an instance (a run) of a software. A same software can have multiple runs with different parameters
  • Parameter : There is two types of parameters : a software parameter is a declaration of a variable used by the software and a job parameter is related to a software parameter and contains the value of this variable.
  • Software router : A component of Cytomine that will launch the softwares.


When you can use the Cytomine clients to interact with Cytomine and create various scripts, it can be useful to integrate these scripts into the Cytomine platform and thus allowing end-users to run them through the Web interface.
On this page, we will see how to integrate a script into the Cytomine platform.

Prerequisites

To be able to plug your script into the Cytomine platform you will need to

  • use a command-line terminal
  • have an SSH (and SCP) access to the server hosting your Cytomine
  • modify and compile a Dockerfile. You will find the Docker documentation here

 

Create the script


Firstly, we need to create the script. As we have currently clients in Java, Javascript and Python, you can freely choose to program in one of these 3 language.

In this article, we will use this sample, in Python, that will list the images into a project and create a file, that will be uploaded in Cytomine, containing the value of an integer input parameter.

Once the script is created, tested and validated, identify the wanted parameters that the users will be able to set.

Declare the script at the platform


To be able to allow end users to run your program, we need to add meta-information about it into the platform.
To do so, we are currently using an external client and declaring the information programmatically.

For our example, we add the Software and SoftwareParameter using the Python client and declaring the name of the software and its command line to launch our software and the name, type (integer of string or ...) and default values of each software parameter.

Note that the execution command will always reference your script in the following way : algo/DEDICATED_FOLDER_OF_YOUR_SCRIPT/YOUR_SCRIPT
As out example is in Python, the beginning of the command is python algo/test_soft/test.py

The script can be find here

We run this script with the command

python add_software.py

 

Extend the software router Dockerfile:


As your scripts will need some dependancies, we need to add them to the Docker container that will run your scripts.
The easiest way is to start from this Dockerfile and to extend it to fit your needs.

Then you will need to compile the Docker image

sudo docker build -t cytomine/software_router:my_organisation .


As you're now using a custom software router, it is mandatory to modify the start_deploy.sh script and modify the line.

docker create --name software_router \
...
cytomine/software_router:v1.1.1 > /dev/null

by

docker create --name software_router \
...
cytomine/software_router:my_organisation > /dev/null

 

Add your script file on the Cytomine server

Now, Cytomine know how to launch your script and has the needed dependancies but the script itself is still missing.
We need to copy it on the server that hosts the software router container (classically with a SCP command).

Once the file on the server, we move it into the folder mapped between the container and the host (usually /data/algo).

In our example, and our previous declaration to the Cytomine platform, we will move the file into the /data/algo/test_soft folder after creating the test_soft folder.

 

Restart the software router container


Restart the container to apply the previous changes by running

sudo docker stop software_router
sudo docker rm software_router
sudo sh start_deploy.sh

 

Run and debug it


You can now browse the Web UI of Cytomine and go to the analysis panel of a project space.
If your software is not yet associated with your project, you will be able to associate them in the Algorithms tab in the Configuration panel of your project space.
Job panel

Click on the Launch new job button to run a job based on your software by selecting the wanted software

Launch a new job
and filling the parameter values. As you can see, the type is checked as you declared it previously

Fill job values
After the run, you will be able to have the detail of the job with its status and, if finished, its output.
Job panel with job detail

 

Get logs


Each job will store, in the container, its logs in the /software_router/algo/logs/ folder in a file with a timestamp in the filename.
In our example, a file log could be /software_router/algo/logs/queueSoftwareTest_Software-6-1-2020_10-24-56-056.log

In the same way, the software_router Java program will log if it received the message from the core component.
So, to consult the two mains logs, use the following commands

#enter in the container
sudo docker exec -it software_router bash

#log of the software_router program
cat /software_router/log4j.log
#search for the log of your job
ls /software_router/algo/logs/queueSoftwareTest_Software-*.log 
# then display the related log