First Time User Guide

Appen Platform: First Time User Guide

Welcome to Appen! The purpose of this guide is to provide new users on the Appen platform
with a concise source of knowledge and best practices to successfully build, launch, monitor,
audit, and iterate on a job.

Throughout this guide, you will find links to relevant articles in the Appen Success Center. If
you’re unable to find what you’re looking for, please reach out to our Platform Support team
via chat (from the question mark in the bottom right of the platform) or by emailing
[email protected] with your request.

To understand the terminology throughout this guide, we recommend that you first review our
Glossary of Terms.

appen.com
First Time User Guide

Table of Contents

Home Page ........................................................................................................................................... 3

Job List Page ......................................................................................................................................... 4
Design Page .......................................................................................................................................... 8
Graphical Editor ................................................................................................................................... 9
Code Editor.........................................................................................................................................13
Quality Page (Test Questions) .........................................................................................................17
Settings ...............................................................................................................................................24
Launch Page .......................................................................................................................................25
Monitor Page .....................................................................................................................................27
Results.................................................................................................................................................29
Auditing Results .................................................................................................................................30
Automation ........................................................................................................................................32
Summary.............................................................................................................................................32

appen.com
First Time User Guide

Home Page

When you log into the platform for the first time, you will see our Home Page. This page can be
accessed by clicking the Appen logo in the top of the left Global Navigation bar.
When creating a job in the platform, you have the option to begin with one of our templates on
the Home Page or build a job from scratch. The templates are organized by use case on the
Home Page.

• Within each use case, there are various types of templates which you can “Preview” to
see their functionality or copy into your account by clicking “Use this Template”
• You’re encouraged to explore these templates and customize them to meet your needs.
If you do not see what you’re looking for, submit a request at the bottom of the page to
our Platform Support team.

To start from scratch, click “See More” and “Start from Scratch”.
For more information, visit the How to Use Job Templates Success Center article.

appen.com
First Time User Guide

Job List Page

Your Job List page shows the collection of jobs in your account. This page can be accessed by
clicking the list icon in the top of the left Global Navigation bar.
Use the search bar and filter dropdowns to find a specific job or narrow the list of jobs
displaying on the page. For more information on the Job List page, visit our Job Management
Success Center article.
In order to create a new job, click “Create Job” and you will be taken to the Home Page to
choose from the templates or start from scratch.

appen.com
First Time User Guide

1. Job ID - Unique identifier for that job

• When you reach out to our Platform Support team for help, include the job ID so
they can quickly begin to troubleshoot
2. Job Title - Short description of the job to attract contributors
3. Projects – Internal method of organizing jobs
4. Tags - Internal reference that can be used when searching for jobs.
5. Status
• Not Launched - no rows have been launched in the job

Running - the job is currently collecting judgments

• Paused - the job has been launched but is currently not collecting judgements
• Out of Funds - more crowd spend is required to finish annotating the launched
rows
• Canceled - the job cannot be launched from this state and all funds in progress
are returned to your account
• Deprecated - the job disappears from your Jobs list, but no data is deleted from
the platform (please see How to: Deprecate a Job for more information)
6. Cost – Amount that has been spent to pay the contributors that are labelling your data.
7. Funds In Progress - the money that is reserved for the completion of that job. If a job is
ever cancelled, any funds in progress that were tied to the job will be returned to your
account.
a. Any unused funds in progress will be returned after a job is finished or canceled.
8. Gear Icon – Copy, launch, cancel, deprecate, or resume a job by clicking the gear icon.
The options available depends on the current status of the job.
9. USERNAME/TEAM dropdown – view the jobs in your teammates’ accounts or all jobs
created by your team from the dropdown menu

appen.com
First Time User Guide

Data Page

In a new job, you will start on the Data Page where you upload source data to be displayed to
contributors. We recommend using a UTF-8 encoded CSV file, however other file formats are
supported.

Check out our Success Center article on How to: Add Data

Some important considerations for your data file:

• Your column headers will be used to render data in the platform, so make them
descriptive of the data
• Every column which contains data must have a header, and column headers must be all
lowercase text with underscores in place of spaces
• Each job has a limit of 250K rows, we recommend keeping volumes to 200K per job

You do not need to remove metadata/internal reference columns that will not be used in the
job prior to uploading the source data file.

• You can leave these columns with the rest of your columns that will be referenced in the
job.
• When you download your results after your job is complete, all the columns that were
originally uploaded will appear with the results columns.

Once source data is uploaded, it cannot be modified or batch deleted

• Unlaunched (or “new”) rows of data can be deleted one row at a time or one page of
data at a time
• If you uploaded the wrong data set or need to update the source data, create a copy of
the job with no rows, then upload the correct source data to the new version of the job

Click ‘Browse’ to upload your CSV into the platform or drag and drop your file.

appen.com
First Time User Guide

Once your data is uploaded, you will see your source data as well as some new columns:

Column Description
Unit ID Unique identifier for that row
State • New - the row has not been launched
• Judgable - the row has been launched but has not collected all
judgements
• Finalized - the row has collected all judgements
• Golden - the row is a test question
Judgments Number of judgments that the row has collected thus far
Agreement Score of the similarity between contributors’ judgements on the row - visit
How to: Calculate Overall Unit Agreement for more information on this score

appen.com
First Time User Guide

The next step is to design the job. Click on “Next: Design your job” or “Design” from the top
navigation bar to visit the Design Page.

Design Page

On the Design page, you will give your job a title, build the interface where contributors will
view and annotate the data in your job, and create a set of instructions to provide guidelines
and examples to the contributors working in your job.
Periodically when building your job design, remember to check the job “Preview” by clicking the
eye icon from the top right of your screen to ensure that everything displaying correctly. This is
exactly how contributors will view the task.

There are two options for building jobs in the Appen platform:

• Graphical Editor
• Create simple jobs using no code with our “plug-and-play” interface
• Great for people who are not familiar with CML or HTML
• Visit our Guide to: The Graphical Editor in the Success Center
• Job designs in the Graphical Editor are limited to one layer of conditional logic

• Code Editor
• Create more complex, customized job designs using our HTML-based Custom
Markup Language (CML)
• Visit our Guide to: The Code Editor in the Success Center
• The following beta tools are only available for the Code Editor Video Object Tracking,
Image Transcription, Image Segmentation (PLSS), Audio Transcription, Audio
Annotation, and Text Relationship

Regardless of the editor that you choose to use, we have a collection of Tips for Job Design and
Instructions Best Practices that we recommend you follow to receive the highest quality data
and create the best experience for contributors.

appen.com
First Time User Guide

Graphical Editor

To Design a job in the Graphical Editor:

• Click ‘Insert Data’ to display data to contributors in the task

o You will be shown all the column headers that exist in the source data uploaded
to the job.
o Choose the column(s) containing the data that you will show to contributors and
select whether each should be displayed as Text, as a Link, or as an Image.
If there are links to hosted images in your uploaded data, choose the
Image option to display it directly within the contributor’s interface.
Choosing link will require that contributors click the link to open the
image.

• Choose the question type that you’d like to add to the design (i.e. multiple choice, text
box, checkbox group, etc.)
o When a question type is selected, you will be prompted to configure the
question so it meets the needs of your design.
o Different question types will have different fields to configure.

appen.com
First Time User Guide

When you add a question, the side panel will appear for you to configure the attributes of that
particular question including:

General Options

Option Description
Question Text This is what contributors will see in the job
Label These are the answer options the contributors will see
Value These are the values stored in your results report for the respective
labels
• Some question types allow you to define a Default Value which
will be used as the answer if the contributor does not select a
different answer
Required Check this so that contributors must answer this question before
(Validator) submitting work
• There must be at least one required question in every job
Tips/Hints In-line instructions for that question
Results Header Column header in the results file for that question
• It is best practice to give each question a short, descriptive
results header

appen.com
First Time User Guide

• If a results header is not defined, the default is to use the

Question Text in all lowercase with underscores replacing spaces

A few notes on general options:

• Tips/Hints and Results Header are found under “Additional Options”

• Not all general options will need to be configured for every question type
• The Image Annotation and Text Annotation tools have different general options than the
standard options shown above

Conditional Logic

You can use Conditional Logic to show certain questions or information depending on the
answer to a previous question

• The Graphical Editor is limited to one layer of conditional logic

• For more complex conditional logic, you will need to use the CML Editor

appen.com
First Time User Guide

Basic, Smart Validation, and Clean Output

Text input question types including Text Box (Paragraph), Text Box (Single Line), and Website
(URL) allow you to control and normalize the input provided by contributors using validation.

Validation Type Description

Basic Validation Used to include specific text formatting rules based on the type of data
you are collecting with the text box.
• These validation rules require that the contributor input is in a
particular format
Smart Validation Includes our Text Utterance Smart Validation models leveraging
machine learning to detect language, coherence, and duplicates in the
provided utterances
• Learn more about the models by visiting Smart Validation Model
Details
Clean Output These validators clean and normalize the text format – contributors
never “fail” these validators since it is not validating, it’s cleaning the
text
• Visit our Guide to: Validators > Clean to learn how each cleaning
option works

At any time in the Graphical Editor, you can switch to the code editor by clicking ‘Switch to
Code Editor’ at the bottom of the page.

• Once changes are made in the CML Editor, you will not be able to return to the
Graphical Editor for that job
• The screenshot below shows the warning that you will see after clicking ‘Switch to Code
Editor’

Appen Expert Tip

If you are new to using CML and need to switch to the Code Editor to make custom updates,
we recommend creating a copy of the job that’s built in the Graphical Editor as a base
template to go back to if needed.

appen.com
First Time User Guide

Code Editor

To display data to contributors in the platform using, click ‘Insert Data’ and choose the column
header

• The column headers are referenced as liquid variables (e.g. {{column_header}})

o Check out our Guide to: Liquid and Liquid Logic for more information on how you
can control the way your data is formatted using liquid.
• Standard HTML tags can be used to add content to your job design such as images/GIFs,
audio/video, iframes and more!
• If you are using our Image Annotation tool, Text Annotation tool, or any of our Beta
tools including Video Object Tracking, Image Transcription, Image Segmentation (PLSS),
Audio Transcription, Audio Annotation, and Text Relationship, then you will display your
data within the tool.

To add questions to your job in the Code Editor, click ‘CML Reference’ and choose the question
type, or copy/paste from the Success Center articles for each question type

Check out these helpful Success Center articles on CML and the Code Editor

• CML Overview
• Guide to: CML Attributes
• Guide to: CML Logic
• Guide to: Validators
• Tips and Tricks for the Code Editor

appen.com
First Time User Guide

Within the opening tag of CML elements, you will have the following Attributes:

For parent (main question) tags:

Attribute Description
label The question text that contributors will see
name The column header in your results file
• The name can be referenced when using CML Logic to only
display content when a particular option is chosen in a previous
question
instructions Optional in-line instructions for that question
default Optional default response for the question
validates Optional attribute to reinforce validations on the contributors’
responses

appen.com
First Time User Guide

• validates = “required” requires that contributors answer the

question for every row before submitting a page of work
• Clean validators are recommended when using open text fields
to standardize your results formatting
• Check out our Guide to: Validators to see all of the possible
validation methods
only-if Optional attribute to include conditional logic
• The value of this attribute should be the name of the field a user
must complete before this field or group of fields will be
displayed
• Check out our Guide to: CML Logic for more detail of CML
Conditional Logic

For child (answer options) tags:

Attribute Description
label The text for the child option that contributors will see
value The value that will appear in the results when the child option is chosen

Some tools have additional CML attributes which are outlined in the tool-specific Success Center
articles:

• Text Annotation: Guide to: Text Annotation Job Design

• Image Annotation: cml:shapes - Bounding Box, Polygon, Dot Annotation, and Line
Annotation tool
• Beta Tools: Video Object Tracking, Image Transcription, Image Segmentation (PLSS),
Audio Transcription, Audio Annotation, and Text Relationship

In the CML Editor, you can click ‘Show Custom CSS/JS’ to add custom styling and JavaScript to
your job design

• If you’d like to add custom JavaScript, please reach out to our Platform Support team via
in-platform chat or by emailing [email protected] as this is a special configuration.

Visit some of our CSS and JavaScript articles in the Success Center

• CSS | Guide to: Collapsible Containers

• CSS | Guide to: Enlarge Images on Hover

appen.com
First Time User Guide

• JavaScript | Guide to: Twitter Bootstrap

• JavaScript | How to Add jQuery
• JavaScript | Guide to: jQuery Popovers
• JavaScript | Guide to: Customizing an Appen Validator

Instructions

The instructions module is where you will let contributors know how to be successful when
working in your job. We highly recommend that your instructions follow the template outlined
in the Instructions Best Practices Success Center article.

Important reminders when creating instructions:

• Use simple, concise language in your instructions

• You’re the expert, but contributors may be new to the task and may not have the
same background knowledge as you
• Cover all “edge cases” that you anticipate in your dataset in the instructions
• Provide guidance around how you’d like contributors to handle these cases
• Include examples that are directly from your source data
• This lets contributors know what to expect prior to entering the job
• Instructions may need to be revised, especially when creating test questions or identifying
areas of improvement after launching a test run

A few nice-to-haves:

• Provide context around the purpose of the task, if possible

• This “big picture” can be very helpful to contributors
• Put a little “Thank You” in your instructions and let the contributors know that you
appreciate their hard work!

appen.com
First Time User Guide

Appen Expert Tip

If you have a complex or difficult job, consider adding a short 1-2-minute instructional video
to the job instructions. In the video, explain how to work in the task successfully with a
screen/audio recording of you walking through the job preview or internal link. To maximize
the value of the video, keep it concise and highlight only the most important pieces of
information that you want to communicate to contributors.

Quality Page (Test Questions)

This section is extremely important, and we recommend reading it fully before launching a job on the
platform

Test Questions are the main quality control feature on the Appen platform. They are pre-
labeled data which contributors’ answers are compared against to ensure that they are
submitting high quality work in your job. Please see our Test Question Best Practices Success
Center article.

Note: test question functionality is not available for our beta tools Video Object Tracking, Image
Transcription, Image Segmentation (PLSS), Audio Transcription, Audio Annotation, and Text
Relationship

• To control quality in jobs using these tools, we recommend following a “peer-review”

process (using the “review-from” CML attribute) where the resulting annotations are
loaded into another job for contributors to validate and correct

Before launching a job, you make test questions because:

• Test questions allow us to verify a contributor is qualified to work on your job in Quiz
Mode and establishes an accuracy score
• They help us check in on how well a contributor is maintaining their accuracy in Work
Mode as they continue to label your data
• More test questions unlock faster job completion

appen.com
First Time User Guide

• They are a method of training the contributors working in your job

How are test questions used?

• In Quiz Mode, contributors are provided a page consisting entirely of test questions
• Contributors must correctly answer enough test questions to meet the job accuracy
threshold that you set and enter work mode
• Think of Quiz Mode as the “gatekeeper” to your task which qualifies high-
performing contributors and removes low-performing contributors
• You don’t pay for contributors’ judgments in Quiz mode, only those in Work Mode
• Please see Test Questions and Enabling Quiz Mode for more information on Quiz
Mode.

• In Work Mode, there are hidden test questions throughout the job which are used to
continue testing and training contributors
• Every page of work will contain a hidden test question and each contributor can see
each test question only once
• Each test question passed or failed continues to affect the contributors’ accuracy
scores in Work Mode

appen.com
First Time User Guide

Quiz Mode (All test questions) vs. Work Mode (One test question per page)

How do you create test questions?

Manually Create Test Questions via the Quality Page

• Click ‘Create Test Question’ and a random row will be pulled from your data to be
converted into a test question
• Enter answers to each question according to your instructions, and provide explanations
for why those are the right answers in the “Reason” text boxes
• You can select multiple correct answers for leniency if multiple answers are
acceptable
• When a contributor incorrectly answers a test question, they will see the correct answer
and your reasoning. They’re also given the opportunity to provide a contention to let
you know why they disagree
• If you do not provide a correct answer to a certain aspect of the test question, then that
aspect will not be considered when evaluating the contributor’s answer

appen.com
First Time User Guide

Upload Test Qustions in Bulk Using a CSV

• Here is an example of a prepared test question report

• The answer column headers are the same as your results headers for each question
except with “_gold” appended
• The “_gold_reason” columns are for including the explanation for each answer
• The column “_golden” with “TRUE” for all rows lets the platform know that these
are test questions
• Upload the Test Question Report to the Data Page, then click “Convert Uploaded Test
Questions”
• Go to the Quality Page to see your new test questions
• Bulk uploading test questions is not available for the Text Annotation tool
o Please see the Guide to: Text Annotation Test Questions for more information on
Text Annotation test questions

Convert Finalized Rows into Test Questions

• This process is best used on a job that has successfully launched 100 rows with quality
results. These 100 rows can then be converted into test questions. They can also be
downloaded in a test question report and uploaded to a new job with no test questions
via the Data page.
• Make sure the aggregation settings in the Options Page is set to "Best Answer('agg')" for
each question. This will ensure the test question conversion feature works properly.
• Using this feature helps create more high-quality test questions in significantly less time!

appen.com
First Time User Guide

How are contributors evaluated against test questions?

Test questions are PASS/ FAIL

• In order to pass a test question, a contributor must correctly answer every aspect of the
test question for which you’ve provided a correct answer.
• For jobs that ask a lot of questions and include logic, the questions that are answered
should vary. Varying the answer makes it easier for contributors to get the question
correct because they are tested on less fields. If contributors get one question wrong,
they get the entire test question wrong.
• Please see the Guide to: Text Annotation Test Questions if you are creating test
questions for a text annotation job.
o You can introduce leniency on a question-by-question basis for text annotation
test questions.
• For image annotation test questions, an intersection over union calculation is used to
determine whether the contributors’ answers are correct.
o While test questions can be used, image annotation use cases also support peer
review where contributors can validate and edit the results from a previously run
annotation job.

Establishing a baseline accuracy from Quiz Mode

• Contributors earn an accuracy score from Quiz Mode which must be higher than the
job’s accuracy threshold in order to enter the job
• If the contributor does not answer enough questions correctly in Quiz Mode, they will
not be allowed to enter Work Mode

appen.com
First Time User Guide

Contributor Passes Quiz Mode and Correctly Answers Work Mode TQ

Maintaining accuracy in Work Mode

• As contributors continue to answer test questions in Work Mode, their accuracy score is
constantly updated
• If a contributor answers a test question correctly, their accuracy score will increase (see
screenshot above)
• If a contributor answers a test question incorrectly, their accuracy score will decrease
(see screenshot below)
• If their accuracy falls below your minimum threshold:
o We thank them for their work, pay them but do not allow them to provide more
judgments in your job
o We remove the judgments they provided on your data (marked as “untrusted
judgments”) and replace their work with someone’s whose accuracy meets or
exceeds your requirements
o As you “clean” test questions while monitoring jobs, a person’s judgments can
switch between untrusted or trusted, but the end result is high quality data

appen.com
First Time User Guide

Contributor Passes Quiz Mode and Incorrectly Answers Work Mode TQ (They Fail Out!)

How many test questions do you need?

Quiz Mode and test questions are enabled when: Total Number of Test Questions >= (Rows
Per Page setting+ 3)

• The number of rows per page is set by you, so this may vary. By default, there are 5
rows per page, so you’d need to create 8 test questions to enable Quiz Mode.
• IF YOU HAVE FEWER THAN ONE PAGE’S WORTH + 3, no test questions will be used in
your job and contributors’ accuracy will not be evaluated
• Please see Test Questions and Enabling Quiz Mode for more information on Quiz Mode.

The more test questions you have, the faster your job will complete

• A contributor will “Max Out of Work” when they have seen every test question in your
job - each test question can only be seen once by each contributor
• Each additional test question that you make allows each contributor to do one
additional page of work in your job

appen.com
First Time User Guide

As contributors provide judgments in your job after it has been launched, you may find that
your test questions need adjustments. If so, you should update your test questions as the job is
running.

• This will retroactively update the contributor trust scores based on the cleaned test
questions.
• Check out the Success Center article How to: Monitor Test Questions

Appen Expert Tip

We advise against reusing test questions to keep your evaluation metrics consistent with
new batches of data. Data may vary from batch to batch and refreshed test questions allow
contributors to be continually tested on new material and prevent contributors from
recognizing past test cases.

Settings

Before launching your job, go to the Settings page by clicking the gear icon on the top right-
hand side of the platform to make sure that your job settings are configured appropriately. In
this guide, we will touch on the basics of Contributor and Pay settings, which are most
important when launching a job for the first time.
Please see the following Success Center articles to learn about the rest of the settings that you
can configure for your job:

• Quality Control – Test Questions

• Quality Control – Quality Control Settings
• Quality Control – Dynamic Judgments
• Sharing/Visibility
• API

appen.com
First Time User Guide

Contributors Channels Page

From the Contributor Channels Page within your job settings, you can select to launch your job
externally, internally, or both.

• When launching to the External channel, you must have test questions to control the
quality of your results. Make sure that you have enough test questions to enable Quiz
Mode!
• The Levels indicate the experience level of the contributors that can work in your job
Level 1 is the largest pool of contributors and Level 3 is the smallest
• We always recommend first launching to Level 1, then increasing the
contributor level later if necessary
o When launching to the external crowd, you can limit who can work in your job
via geotargeting and language targeting.
• Before updating the contributor targeting, check with your Customer
Success Manager for recommendations.
o You can choose to launch to a Custom Channel of preferred or high-performing
contributors from a previous job.
• When launching internally, you and your colleagues can work in the job via the job’s
internal link.
o Contributors working through the internal channel link are not paid through the
platform.

Pay Page

From the Pay Page within your job settings, you can control the number of rows that will
appear on each page, as well as the amount that you pay each contributor for submitting each
page of work.

Launch Page

On the Launch Page page, you can configure:

appen.com
First Time User Guide

The number of judgments that you would like to collect for each row

• We recommend 3 judgments per row for aggregating your results

• If you are using Dynamic Judgments, the job will collect a variable number of judgments
dependent upon contributors’ agreement
The price that you will pay per judgment

• This reflects the settings on the Settings > Pay Page

• Changing the pay from the Launch Page will update the Pay Page

The number of rows that you would like to order

• Start with 100 rows as a test before launching a large batch of rows

Expected Cost

• Contributor Judgments - estimation of contributor costs

o Calculated as:

(Judgments per row * (Pages of work * Price per page))

+ estimated buffer for untrusted judgments

• Cost Buffer - additional fees to reach $10 minimum order

• Transaction Fee - percent mark-up
• See the Job Cost FAQ in the Success Center

Due Now

• This is the total amount of money due for the current settings
• If you have already launched the job, the estimator will be based on the settings in place
at the time of the first job launch

Available Funds

• This is the amount of money currently available for your team

• Add Funds - Click ‘Add Funds’ to increase the available funds in your account

appen.com
First Time User Guide

Visit the Guide to: Launch Page for more detail and check out the Job Launch Checklist if you
are preparing to launch a new job on the Appen platform.

Monitor Page

Dashboard Page

Once your job has been launched, you will be able to view statistics on your running job from
the Dashboard Page. See the Guide to: Dashboard Page for details on each aspect of the page.

At the top of the Dashboard Page, there are high level statistics about your job. Hover over
each number with your cursor to see more granular information.
1. Percent Completion - This is (Rows Finalized / Total Rows Uploaded to Job) * 100
• Hover over to see the number of finalized rows
2. Job Cost - The amount of money that has already been spent on contributor judgments
• Hover over to see the cost down to the cent
3. Active Test Questions - The number of active test questions in the job
4. Rows - Total number of units in a job. This count does not include Test Questions.
• Hover over to see how many units there are in total, how many have been
launched and how many are new units
5. Judgments By Hour - Average number of judgments collected per hour (over the last 24
hours)

appen.com
First Time User Guide

6. Trusted Judgments - Total number of Trusted Judgments the job has collected (not
including judgments on test questions)
• This may be rounded, hover over to see the exact number
7. Untrusted Judgments - Total number of Untrusted Judgments the job has collected (not
including judgments on test questions)
• The goal is to minimize the number of untrusted judgments in your job
8. Pending Judgments - The number of judgments for launched rows that are awaiting
judgment by the crowd
• The job state will change to ‘Finished’ when this reaches 0

Advanced Analytics

This page of the platform is used to take a deeper dive into performance data while job is live
or after a job has run. Each section of the Advanced Analytics page is described in the Guide to:
Advanced Analytics Page.
Some of the most important metrics for first time users are within the ‘Contributors’,
‘Distributions’ and ‘Times’ dropdown options:

• Contributors will show you some high level information about the contributors working
in your job
• If the bar chart shows that a group of people have submitted the same amount of
work, that is an indicator that contributors are “Maxing out” of work and you may
want to add more test questions to allow them to continue working in the job

• Distributions will show you the distribution of answers that contributors are providing
in your overall dataset for each individual question
• If you have an idea of what to expect in your data, check whether the contributors’
answers are aligned with your expectations

• Times, specifically “IQM Trusted Judgment Time” will show you approximately how long
it takes for a contributor to submit a judgment in your job
• Are contributors taking much longer than expected?
• If so, consider clarifying instructions
• Are contributors going much quicker than expected?
• If so, do an audit to check on your results to make sure they’re meeting
quality expectations

appen.com
First Time User Guide

Results

Reports
The Reports page can be accessed by clicking on ‘Results’ in the top navigation. From the
Reports Page, you can download the following report types:

Report Type Description

Full Report • Each row represents a judgment and includes information on
the contributor who submitted the judgment
• The setting ‘judgments per unit’ determines the number of
judgments you will have in the source report for each row in
your source data

Aggregated Report • This report includes aggregates all the responses for each
individual row in the job
• The aggregation settings can be adjusted from the CML attribute
“aggregation” or from the Reports > Options Page

Source Report This report includes the original, unprocessed data that was uploaded
to the job before it was run

Test Question This report includes only data on the test question rows in the job
Report
Contributor Report This contains information on all contributors that worked in your job

JSON Report This report contains JSON formatted data for each judgment and row

Download All This report type is only available for Text Annotation jobs
Annotations

Check out our Guide to: Reports Page and Options Page for a detailed breakdown of the
contents of each report type

appen.com
First Time User Guide

Options
From the Results Options Page, you can adjust the aggregation method for each question in
your job and customize the data included in the reports.

For open text fields

• The default aggregation is aggregation=“all” which will include all judgments separated by a
newline
• If you’re accepting multiple answers from each contributor with the multiple=”true”
attribute in the CML, we recommend using aggregation=“cagg_0.6”
• If you are collecting a single option, we recommend updating to aggregation=”agg”

For multiple checkbox groups

• The default aggregation is aggregation=“all” which will include all judgments separated by a
newline
• We recommend using aggregation=“cagg_0.6” to ensure only high confidence answers are
returned

For Image Annotation

• With class-agg, you can aggregate on the class if using an ontology

• With box-agg / polygon-agg, you can control the minimum IoU required to aggregate the
shapes
• With dot-agg / line-agg, you can control the maximum acceptable distance between the
dots/lines to aggregate

Auditing Results

appen.com
First Time User Guide

We recommend that you audit your results to see how contributors are performing in your
task. Please read our Success Center article on How to Audit your Results for step-by-step
instructions on Auditing.

• Audit your test run results to identify areas of improvement prior to launching
additional rows
• Audit your job results periodically when running large jobs to ensure that the quality is
being maintained

The Spotcheck Beta feature can be used for auditing the results of jobs within the platform.

• If you are interested in using this feature, please reach out to our Platform Support
team via chat (from the question mark in the bottom right of the platform) or by
emailing [email protected] with your request.
• Spotcheck is not available for the Image Annotation, Image Transcription, Image
Segmentation (PLSS), Video Object Tracking, Audio Annotation, Audio Transcription, and
Text Annotation, and Text Relationship
o These tools support “peer-review” where results from previous jobs can be
loaded back into the platform to be viewed and corrected.

Appen Expert Tip

Use your audit results as test questions in a future launch. When auditing, you decide the
correct answers for a subset rows from your job. Format the audit results in a CSV as
described in How to: Create Test Questions > Manual Test Questions.

Iteration

Iterating is key to building a new job or making changes to existing jobs. We recommend
launching a 100-row test in your new job to identify any areas of improvement within your
instructions, test questions, design, and/or settings for the larger batches. If your full dataset is
lower volume, consider a test batch smaller than 100 rows.

appen.com
First Time User Guide

• As your 100-row test is running, clean your test questions

• Highly missed and contested test questions indicate that contributors may not be as
confident in those cases, and you could address it in the instructions/examples

• When your 100-row test completes, do an audit of the results

• Identify areas of lower accuracy and address any common mistakes by adding more
examples, in-line instructions and tips, and creating test questions out of rows that were
missed in the test run.
• If you would like a job review consultation from our team, please reach out to our Platform
Support team via chat (from the question mark in the bottom right of the platform) or by
emailing [email protected]. We’re happy to review and offer recommendations!

Automation

Once you are happy with the job you’ve designed and the results you’re receiving, consider
automating the job launch process. There are a few automation options in the Appen platform:

Automation Description
Option
Workflows • Automatically route your unlabeled data across multiple jobs in
the platform.
• Visit the Workflows API documentation.
Appen Platform Programmatically create, edit, and launch Appen jobs, download
API results, and more!

Webhook Create robust applications that interact with Appen and take actions
Integration based on the real-time results that the platform provides.

Summary

By following the processes and best practices outlined in this guide, you should be able to
successfully design, launch, monitor, audit, and iterate upon your Appen job, and receive high-
quality data as a result.

appen.com
First Time User Guide

Data & Design

• Data should be in a CSV file that is UTF-8 Encoded

• Use the Graphical Editor for simple tasks and the CML editor for more complex jobs
• For two or more layers of Conditional Logic, you’ll use CML in the Code Editor
• Use CML attributes, validators, and liquid logic to improve your job design
• Follow these Job Design Best Practices when building a job

Quality & Settings

• Test questions are the main form of quality control

• Filter for quality contributors
• Provide feedback to contributors
• Primary quality control method
• You must have one page worth + 3 to enable quiz mode
• How to create test questions
• Consider your answer distribution
• Allow ALL acceptable answers
• Do not test on all questions if you have many in your design
• Review contentions
• Convert high confidence rows into TQs
• Do not reuse test questions
• Basic Settings and Settings Best Practices
• Start with contributor Level 1 for your first iteration
• Increase the level after if necessary
• We recommend 3 judgments per unit
• Rows per page
• Pay per page

Launch, Monitor, Audit, & Iterate

• Launch a small test before launching large batches of rows

• Set aside some time after launching to monitor your job and clean test questions
• Audit the results of your test run

appen.com
First Time User Guide

• Iterate on your instructions and examples, test questions, and design based on
insights gathered from the audit and missed test questions
• Continue launching small batches and iterating until you're confident in the results
• Launch larger batches and add more test questions to increase throughput
• Remember to monitor your job and clean your test questions periodically

Automate
Once you are happy with the job you’ve designed and the results you’re receiving, consider
automating the job launch process. There are a few automation options in the Appen platform:

• Workflows - Automatically route your unlabeled data across multiple jobs in Appen.
Check out the Workflows API documentation.
• Appen Platform API - Programmatically create, edit, and launch Appen jobs, download
results, and more!
• Webhook Integration - Create robust applications that interact with Appen and take
actions based on the real-time results that the platform provides.

appen.com