Mechanical Turk (Mturk) is a Web service where users, turkers, are paid small rewards (few cents) for short computational task called HITs (Human Intelligence Tasks). A contractor generates the HITs, post them on Mturk and later download all the result.

TurKit is a Java/JavaScript API (developed by the User Interface Design Group at MIT) for running iterative tasks on Mechanical Turk. As of today, TurKit represents the first example of iterative tasks framework for Mturk, as it allows users to perform incremental tasks by automatically generating HITs based on the results of previous HITs.

Many applications can benefit from this iterative paradigm: turkers can take turns improving a passage of text, verify each other’s work by voting on it or implement the comparison function of an iterative sorting algorithm. In the context of SeCo, turkers can be employed, for instance,  to evaluate the quality of a query response.

The developers tested the tool in different scenarios: from iterative text improvement (as image description and writing tasks – outline to prose, active voice, change grammatical tense, OCR) to human sorting experiments (O(logn) algorithm based on quicksort).

The tool includes API designed to make the system fault-tolerant, avoiding waste of time and money caused by re-publishing HITs and collecting responses.

The toolkit

The programmer is asked to write a set of JavaScript files which are executed by TurKit. TurKit stores information about the running program in the JavaScript database, so that it can restart if the system crashes.

TurKit is in charge of:

• Interpreting the JavaScript files that describe the page turkers will see
• Interact with Mturk for publishing the HITs
• Store information about the program trace of execution in the database

The tool allows recovering from crashes simply accessing the JavaScript database where traces are stored and using the information collected since that moment while HITs are still active.

TurKit relies on different APIs which enable users to execute voting, sorting and other common high level tasks increasing system fault-tolerance. APIs are divided into three sets:

• Mturk API: to support different programming languages, including java, using a JavaScript wrapper.
• Trace API: uses the JS database API to store information about program’s trace of execution in it
• Utility API: cover high level Mturk tasks

More information about the TurKit API are available in the project’s Web site, which also contains three tutorials and a paper describing the rationale behind the approach.

We would like to thank Michele Follo and Andrea Vaccarella for their analysis on the TurKit API.