Creating and Managing Assignments
A Worker Accepts a HIT
When your application creates a HIT using the CreateHITOperation operation, the HIT becomes available for Workers to find on the Amazon Mechanical Turk website. If a Worker has Qualifications that meet the HIT's Qualification requirements, the Worker can preview the HIT, then select the "Accept HIT" button to begin work.
When a Worker accepts a HIT, Amazon Mechanical Turk creates an assignment to track the work to completion. The assignment belongs exclusively to the Worker, and guarantees that the Worker will be allowed to submit results any time until the HIT's has elapsed, and still be eligible for the reward.
Multiple Assignments, HIT Lifetime
By default, a HIT has at most one assignment. When a Worker accepts the HIT, an assignment is created, and the HIT is no longer available for other Workers to accept. If the Worker returns or abandons the HIT, the assignment is removed, and the HIT becomes available again.
A HIT can be created to accept multiple assignments by specifying a parameter greater than 1 to the CreateHITOperation operation. Such a HIT will remain available for Workers to accept as long as the number of assignments, in progress or submitted, is less than .
A HIT is only available for Workers to accept until the HIT's elapses, from the time the HIT was created. Once this time elapses, the HIT expires. Such a HIT is no longer available, even if the number of assignments is less than . Workers with assignments in progress are allowed to continue working on the assignments as long as the has not elapsed, even after the HIT expires. If a Worker returns or abandons the HIT after the HIT has expired, the HIT is not made available to other Workers.
Seeing HITs In Progress
You can retrieve a list of all of your HITs at any time using ListHITs.
You can also retrieve the submitted assignments for a HIT at any time using ListAssignmentsForHIT. If your HIT has multiple assignments and has not expired, but some of the assignments have been submitted, ListAssignmentsForHIT will return the submitted assignments.
The Worker Submits, Returns or Abandons the Assignment
A Worker completes an assignment by entering values into the question form and selecting the "Submit HIT" button. The results are stored for later retrieval by your application. The Worker continues working on other HITs, or ends the session.
If a Worker decides not to complete a HIT after accepting it, the Worker may select the "Return HIT" button. Unless the HIT has expired-- that is, unless the HIT's has elapsed since the HIT was created-- the returned HIT becomes available for another qualified Worker to accept. The returned assignment ends, and is no longer accessible using the Requester API.
If a Worker does not submit results before the assignment's deadline-- that is, before the HIT's has elapsed since the Worker accepted the HIT-- the HIT is considered abandoned. The effect is similar to if the Worker explicitly returned the HIT, except that abandonments and returns are tracked as separate statistics in the Worker's profile.
A Worker's HIT return and abandonment rates are tracked by Amazon Mechanical Turk as system Qualifications. You can create HITs that use Qualification requirements based on these numbers. Abandonments and returns are tracked as separate Qualifications.
Forcing a HIT to Expire Early
Normally, a HIT will remain available for Workers to accept as long as results have not been submitted and the HIT has not expired (the HIT's has not elapsed).
You can cause a HIT to expire early by calling ForceExpireHIT. This has the same effect as the HIT expiring naturally: The HIT becomes no longer available for new Workers to accept. Assignments in progress are allowed to complete, either with the Worker submitting results, or the Worker returning or abandoning the assignment. After the HIT has expired, returned or abandoned assignments are not made available to other Workers.
UpdateExpirationForHIT is your "stop" button. If you have submitted incorrect data for a HIT, or otherwise have changed your mind about the HIT, you can force it to expire to prevent Workers from completing it. You will still be responsible for approving assignments that have already been submitted, but UpdateExpirationForHIT can minimize the damage.
Whether the HIT expired with a call to UpdateExpirationForHIT, or expired naturally, the HIT can be made available again with a call to CreateAdditionalAssignmentsForHIT.
Retrieving and Approving Results
When either all of a HIT's assignments have been submitted by Workers, or the HIT has expired and all assignments have either been submitted, returned or abandoned, the HIT is considered "reviewable." Once a HIT is reviewable, the Requester can retrieve and process the results.
ListReviewableHITs returns the HIT IDs for all of your HITs that are reviewable. ListAssignmentsForHIT takes the ID of a reviewable HIT and returns the completed assignments, with the answers submitted by Workers. Your application can use these operations to poll for and retrieve results for HITs.
Once results for an assignment have been retrieved, you approve or reject the assignment. Your application calls ApproveAssignment to approve the assignment, or RejectAssignment to reject the assignment.
A call to will return an error if the Requester's account does not have sufficient funds to pay the Worker and the listing fee at the time of the call.
An assignment should always be approved, unless it is clear the Worker did not follow the instructions in the HIT. The method your application uses to decide if an assignment should be approved will depend on the nature of your application. For example, you may request 3 assignments for each HIT then check to see if the results match, and if they do, approve all 3 assignments. If they don't match, have those results checked by a human operator. In some cases, it may be appropriate to automatically approve all assignments, then control the quality of answers using Qualification requirements.
Every HIT has an , an amount of time after an assignment is submitted at which point Amazon Mechanical Turk will automatically approve the assignment and pay the Worker if it has not already been explicitly approved or rejected. If not specified in the call to CreateHIT, this is set to the maximum, equivalent to 30 days. The maximum value assures that the Worker will get paid even if the Requester is unavailable. You may wish to specify a shorter auto-approval delay if the Workers are trusted (such as by having been vetted with Qualification requirements) and to assure Workers that they will be paid in a timely manner.
When you call ApproveAssignment, Amazon Mechanical Turk automatically processes payment of the reward to the Worker, and payment of the HIT listing fee, using the money in your Amazon.com account. You will not be charged for the reward, nor for the listing fee, if the results are rejected.
When you approve or reject an assignment, you can include a message, a string that the Worker can see on his or her HIT status screen. Including feedback message when rejecting a result may improve the quality of the results the Worker submits for your other HITs.
When a HIT becomes reviewable, its assignments are ready for your application to retrieve and process. Your application can approve or reject assignments, then either dispose of the HIT, or extend it to collect more assignments. ListReviewableHITs returns the IDs of your HITs that have the "reviewable" status.
If your application does not want to make an immediate decision about approving or rejecting assignments, or disposing of or extending the HIT, the application can promote the status of the HIT to the "reviewing" status. "Reviewing" HITs are not returned by a call to ListReviewableHITs by default, so your application can continue to poll for "reviewable" HITs while "reviewing" HITs are awaiting processing. You can promote a HIT to the "reviewing" status using UpdateHITReviewStatus.
The "reviewing" status is useful for answer validation techniques that require waiting for additional information to approve an answer. For example, you can create a "verification" HIT based on the answers for a completed HIT that asks a Worker to compare the submitted answers and verify that they meet the HIT's requirements. Your application can give the original completed HIT a status of "reviewing" while it waits for the verification HIT to be completed, and continue to poll for other HITs as they become "reviewable".
Since your task HITs and your verification HITs will have different HIT types, your application can poll for task and verification results separately by passing the appropriate HIT type IDs to ListReviewableHITs.
You can retrieve a list of IDs of HITs with the "reviewing" status by calling ListReviewableHITs with an appropriate value for the parameter. You can revert a "reviewing" HIT back to the "reviewable" status by calling with an appropriate value for the parameter.
There is no need to revert a "reviewing" HIT back to "reviewable" before disposing of or extending the HIT. If you have decided on the fate of the HIT, you can just call DeleteHIT or CreateAdditionalAssignmentsForHIT on the "reviewing" HIT.
Paying the Worker a Bonus
When you approve an assignment a Worker has submitted, Amazon Mechanical Turk automatically processes the payment of the HIT's reward from your account to the Worker's account. By approving the assignment, you pay the Worker the amount of money specified in the original HIT.
In addition to the HIT reward, you can pay the Worker a "bonus" amount of money by calling SendBonus. You might pay a bonus to reward Workers that submit high quality results. You could use the promise of a bonus payment to encourage Workers to complete your HITs and also to develop a following of Workers.
You can grant a bonus to any Worker who has submitted an assignment for one of your HITs any time after you approve or reject the assignment up until the HIT has been disposed.
Amazon Mechanical Turk collects a fee for bonus payments separate from (and in addition to) the HIT listing fee. The charge for awarding a bonus is 20% of the bonus with a minimum charge of $0.01. For example, if a Worker has a $0.01 assignment and is awarded a $0.01 bonus, the Requester pays the Worker $0.02 ($0.01 assignment+$0.01 bonus) and MTurk $0.02 ($0.01 + $0.01). Or, for example, if the Worker has a $1 assignment and is awarded a $1 bonus, the Requester pays the Worker $2 and MTurk $0.40.
For more information about Amazon Mechanical Turk pricing, see Amazon Mechanical Turk Pricing.
There is no way to pay a Worker for completing a HIT an amount less than the reward for the HIT. When you approve the assignment, the Worker is paid the full HIT reward. Rejecting an assignment impacts your account statistics as well as the statistics of the Worker, so you should always approve an assignment if the Worker completed the instructions in the HIT successfully.
If you want to offer Workers a reward within a range of amounts, you can post the HIT with a reward equal to the lowest amount in the range, then mention the offer of a bonus payment in the description of the HIT. For example, to offer a HIT with a reward between $1 and $5, post the HIT with a reward of $1, then mention in the description that you will pay a bonus of up to $4.
You can retrieve a list of bonuses paid for a particular HIT or assignment by calling the operation.
Deleting the HIT
Once all assignments have been approved or rejected, your application calls DeleteHIT. This removes the HIT from the list of HITs returned by ListReviewableHITs, and tells Amazon Mechanical Turk the data is no longer needed.
Once you delete a HIT, you can no longer approve the HIT's rejected assignments.
Creating additional assignments for a HIT
If a HIT has expired, or the maximum number of assignments has all been submitted, the HIT will no longer be available for Workers to accept. If the HIT has not gathered a satisfactory result, you may extend either the expiration date or the number of assignments, or both, using CreateAdditionalAssignmentsForHIT. With appropriate values, extending the HIT will make it available again.
Runbook execution in Azure Automation
When you start a runbook in Azure Automation, a job is created. A job is a single execution instance of a runbook. An Azure Automation worker is assigned to run each job. While workers are shared by multiple Azure accounts, jobs from different Automation accounts are isolated from one another. You do not have control over which worker services the request for your job. A single runbook can have multiple jobs running at one time. The execution environment for jobs from the same Automation Account may be reused. When you view the list of runbooks in the Azure portal, it lists the status of all jobs that were started for each runbook. You can view the list of jobs for each runbook in order to track the status of each. For a description of the different job statuses Job Statuses.
The following diagram shows the lifecycle of a runbook job for Graphical runbooks and PowerShell Workflow runbooks.
The following diagram shows the lifecycle of a runbook job for PowerShell runbooks.
Your jobs have access to your Azure resources by making a connection to your Azure subscription. They only have access to resources in your data center if those resources are accessible from the public cloud.
The following table describes the different statuses that are possible for a job.
|Completed||The job completed successfully.|
|Failed||For Graphical and PowerShell Workflow runbooks, the runbook failed to compile. For PowerShell Script runbooks, the runbook failed to start or the job encountered an exception.|
|Failed, waiting for resources||The job failed because it reached the fair share limit three times and started from the same checkpoint or from the start of the runbook each time.|
|Queued||The job is waiting for resources on an Automation worker to come available so that it can be started.|
|Starting||The job has been assigned to a worker, and the system is in the process of starting it.|
|Resuming||The system is in the process of resuming the job after it was suspended.|
|Running||The job is running.|
|Running, waiting for resources||The job has been unloaded because it reached the fair share limit. It resumes shortly from its last checkpoint.|
|Stopped||The job was stopped by the user before it was completed.|
|Stopping||The system is in the process of stopping the job.|
|Suspended||The job was suspended by the user, by the system, or by a command in the runbook. A job that is suspended can be started again and resume from its last checkpoint or from the beginning of the runbook if it doesn't have checkpoints. The runbook will only be suspended by the system when an exception occurs. By default, ErrorActionPreference is set to Continue, meaning that the job keeps running on an error. If this preference variable is set to Stop, then the job suspends on an error. Applies to Graphical and PowerShell Workflow runbooks only.|
|Suspending||The system is attempting to suspend the job at the request of the user. The runbook must reach its next checkpoint before it can be suspended. If it already passed its last checkpoint, then it completes before it can be suspended. Applies to Graphical and PowerShell Workflow runbooks only.|
Viewing job status from the Azure portal
You can view a summarized status of all runbook jobs or drill into details of a specific runbook job in the Azure portal or by configuring integration with your Microsoft Operations Management Suite (OMS) Log Analytics workspace to forward runbook job status and job streams. For more information about integrating with OMS Log Analytics, see Forward job status and job streams from Automation to Log Analytics (OMS).
Automation runbook jobs summary
On the right of your selected Automation account, you can see a summary of all of the runbook jobs for a selected Automation account under Job Statistics tile.
This tile displays a count and graphical representation of the job status for all jobs executed.
Clicking on the tile presents the Jobs blade, which includes a summarized list of all jobs executed, with status, job execution, and start and completion times.
You can filter the list of jobs by selecting Filter jobs and filter on a specific runbook, job status, or from the drop-down list, the date/time range to search within.
Alternatively, you can view job summary details for a specific runbook by selecting that runbook from the Runbooks blade in your Automation account, and then select the Jobs tile. This presents the Jobs blade, and from there you can click on the job record to view its detail and output.
You can view a list of all of the jobs that have been created for a particular runbook and their most recent status. You can filter this list by job status and the range of dates for the last change to the job. To view its detailed information and output, click on the name of a job. The detailed view of the job includes the values for the runbook parameters that were provided to that job.
You can use the following steps to view the jobs for a runbook.
- In the Azure portal, select Automation and then select the name of an Automation account.
- From the hub, select Runbooks and then on the Runbooks blade select a runbook from the list.
- On the blade for the selected runbook, click the Jobs tile.
- Click on one of the jobs in the list and on the runbook job details blade you can view its detail and output.
Retrieving job status using Windows PowerShell
You can use the Get-AzureRmAutomationJob to retrieve the jobs created for a runbook and the details of a particular job. If you start a runbook with Windows PowerShell using Start-AzureRmAutomationRunbook, then it returns the resulting job. Use Get-AzureRmAutomationJobOutput to get a job’s output.
The following sample commands retrieve the last job for a sample runbook and displays its status, the values provided for the runbook parameters, and the output from the job.
In order to share resources among all runbooks in the cloud, Azure Automation will temporarily unload any job after it has been running for three hours. During this time, jobs for PowerShell-based runbooks are stopped and will not be restarted. The job status shows Stopped. This type of runbook is always restarted from the beginning since they don't support checkpoints.
PowerShell-Workflow-based runbooks will be resumed from their last checkpoint. After running three hours, the runbook job will be suspended by the service and its status shows Running, waiting for resources. When a sandbox becomes available, the runbook will be automatically restarted by the Automation service and resumes from the last checkpoint. This is normal PowerShell-Workflow behavior for suspend/restart. If the runbook again exceeds three hours of runtime, the process repeats, up to three times. After the third restart, if the runbook still has not completed in three hours, then the runbook job is failed, and the job status shows Failed, waiting for resources. In this case, you receive the following exception with the failure.
The job cannot continue running because it was repeatedly evicted from the same checkpoint. Please make sure your Runbook does not perform lengthy operations without persisting its state.
This is to protect the service from runbooks running indefinitely without completing, as they are not able to make it to the next checkpoint without being unloaded again.
If the runbook has no checkpoints or the job had not reached the first checkpoint before being unloaded, then it restarts from the beginning.
When you create a runbook, you should ensure that the time to run any activities between two checkpoints does not exceed three hours. You may need to add checkpoints to your runbook to ensure that it does not reach this three hour limit or break up long running operations. For example, your runbook might perform a reindex on a large SQL database. If this single operation does not complete within the fair share limit, then the job are unloaded and restarted from the beginning. In this case, you should break up the reindex operation into multiple steps, such as reindexing one table at a time, and then insert a checkpoint after each operation so that the job could resume after the last operation to complete.