Submission to TIRA

Introduction

Participants will install/deploy their retrieval models in dedicated TIRA virtual machines, so that their runs can be reproduced and so that they can be easily applied to different data (of same format) in the future.

Quickstart

Once retrieval models are ready, participants will upload them to the VM along with any other code necessary. Please install your software to your TIRA virtual machine by connecting through SSH or RDP to your VM (you can find host ports in the web interface, same login as to your VM). If you cannot connect to your VM, please make sure it is powered on: you can check and power on your machine in the web interface.

Quickstart Task 2

Your software is expected to accept two arguments:

  • An input directory (named $inputDataset in TIRA). This input directory contains a topics.xml file hat contains the topics for which documents should be retrieved.
  • An output directory (named $outputDir in TIRA). Your software should create a standard trec run file in $outputDir/run.txt.
Your Software can use the API of the search engine ChatNoir to produce the run file.
As soon as your Software is installed in your VM, you can register it in TIRA. Assume that your software is started with a bash script in your home directory called mySoftware which expects an argument -i specifying the input directory, and an argument -o specifying the output directory. Click on "Add software" and specify the command ./mySoftware -i $inputDataset -o $outputDir. The other fields can stay with default settings.

Overview of the software configuration in TIRA.

Click on "Run" to execute your software in TIRA. Note that your VM will not be accessible while your system is running – it will be “sandboxed”, detached from the internet, and after the run the state of the VM before the run will be restored. Your run will be reviewed and evaluated by the organizers.

NOTE: By submitting your software you retain full copyrights. You agree to grant us usage rights for evaluation of the corresponding data generated by your software. We agree not to share your model with a third party or use it for any purpose other than research.

Scoring

Once the run of your system completes, please also run the evaluator on the output of your system to verify that your output is a valid submission. These are two separate actions and both should be invoked through the web interface of TIRA. You don’t have to install the evaluator in your VM. It is already prepared in TIRA. You should see it in the web interface, under your software, labeled “Evaluator”. Before clicking the “Run” button, you will use a drop-down menu to select the “Input run”, i.e. one of the completed runs of your system. The output files from the selected run will be evaluated.

Overview of the software evaluation in TIRA.

You can see and download STDOUT and STDERR as well as the outputs of your system. In the evaluator run you will see only STDOUT and STDERR, which will tell you if one or more of your output files is not valid. If you think something went wrong with your run, send us an e-mail. Additionally, we review your submissions and contact you on demand.

You can register more than one system (“software/ model”) per virtual machine using the web interface. TIRA gives systems automatic names “Software 1”, “Software 2” etc. You can perform several runs per system.