Participants will install/deploy their retrieval models in dedicated TIRA virtual machines, so that their runs can be reproduced and so that they can be easily applied to different data (of same format) in the future.
Once retrieval models are ready, participants will upload them to the VM along with any other code
necessary. Please install your software to your TIRA virtual machine by connecting through SSH or RDP to your VM
(you can find host ports in the web interface, same login as to your VM).
If you cannot connect to your VM, please make sure it is powered on: you can check and
power on your machine in the web interface.
Your software is expected to accept two arguments:
$inputDatasetin TIRA). This input directory contains a
topics.xmlfile hat contains the topics for which documents should be retrieved.
$outputDirin TIRA). Your software should create a standard trec run file in
mySoftwarewhich expects an argument
-ispecifying the input directory, and an argument
-ospecifying the output directory. Click on "Add software" and specify the command
./mySoftware -i $inputDataset -o $outputDir. The other fields can stay with default settings.
Click on "Run" to execute your software in TIRA. Note that your VM
will not be accessible while your system is running – it will be “sandboxed”, detached from the
internet, and after the run the state of the VM before the run will be restored. Your run will be
reviewed and evaluated by the organizers.
NOTE: By submitting your software you retain full copyrights. You agree to grant us usage rights for evaluation of the corresponding data generated by your software. We agree not to share your model with a third party or use it for any purpose other than research.
Once the run of your system completes, please also run the evaluator on the output of your system to verify that your output is a valid submission. These are two separate actions and both should be invoked through the web interface of TIRA. You don’t have to install the evaluator in your VM. It is already prepared in TIRA. You should see it in the web interface, under your software, labeled “Evaluator”. Before clicking the “Run” button, you will use a drop-down menu to select the “Input run”, i.e. one of the completed runs of your system. The output files from the selected run will be evaluated.
You can see and download STDOUT and STDERR as well as the outputs of your system.
In the evaluator run you will see only STDOUT and STDERR, which will tell you if one or more of your output files is not valid.
If you think something went wrong with your run, send us an e-mail.
Additionally, we review your submissions and contact you on demand.
You can register more than one system (“software/ model”) per virtual machine using the web interface. TIRA gives systems automatic names “Software 1”, “Software 2” etc. You can perform several runs per system.