Scribd
Upload
274 views6 pages

Outlier

Uploaded by

Nicole Salas Muentes
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
274 views6 pages

Outlier

Uploaded by

Nicole Salas Muentes
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 6

Languages: Preference Ranking and Rewrites - Next Steps NS Nicole Salas

Transitioning from Onboarding to Assessment Tasks

Introduction

Having completed the onboarding courses, you're now prepared to take on assessment
tasks known as Benchmarks. These tasks will look different from what you

encountered during onboarding. While onboarding tasks were structured to guide you
through learning, Benchmark tasks are designed to assess your ability to apply those
skills independently.

The goal of this guide is to help you recognize the differences between onboarding and
Benchmark tasks and to remind you that the principles and skills you learned during

onboarding are essential tools for success in these assessments.

Understanding Production/Project Task Format


During onboarding, you became familiar with a specific task format designed to
introduce you to our systems and processes. These tasks typically included:

1. Write a Prompt to Test the Model: Start by crafting a prompt that aligns with specific
requirements to observe how the model responds, aiming to create a scenario where
one model might not perform as expected.
2. Rate Responses and Identify Issues: Evaluate both model responses, noting any

issues or areas where they fall short. This step helps you practice critical

assessment skills.
3. Side-by-Side Ranking: Rank the original and improved responses in a side-by-side

comparison, providing justifications for why one response is better. This encourages

analytical thinking and understanding of quality standards.


4. Rewrite to Improve: Take the stronger model response and enhance it further. This

exercise builds skills in constructive feedback and optimization.


5. Build on the Conversation: Using the previous responses as a foundation, write

another prompt that extends or deepens the conversation, practicing continuity and

context management.

Always reference the Project Instructions if anything is confusing about

Production/Project Tasks.
Languages: Preference Ranking and Rewrites - Next Steps
Key Differences Between Production/Project and

Assessment Tasks

Assesment tasks will look different. Here are some of the differences you’ll encounter:

For more details, please refer to the Assessment Task Instructions

Assessment Task Walkthrough

Step 1: Assess the Prefilled Prompt and Response


Description

Assessment Tasks contain prefilled prompts and responses. Review these carefully to
understand how they relate to the task requirements.

Key Actions
• Read the prefilled prompt to ensure it aligns with the task's instructions.
• Review the prefilled response for accuracy, tone, and completeness.
Languages: Preference Ranking and Rewrites - Next Steps

Step 2: Evaluate and Rate Task Dimensions


Description
Rate the provided response based on specific dimensions such as clarity, accuracy, and
instruction following. Remember, Assessment Tasks use a 1-3 rating scale with

generalized fields.

Key Actions

• Use the 1-3 scale to rate each dimension objectively.


• Focus on whether the response meets the expectations set by the task prompt.

Step 3: Provide a Justification for Your Rating


Description

In this step, explain why you gave each rating. This is your opportunity to clarify your
thought process and ensure the reviewer understands your assessment.

Key Actions
• Describe the reasons behind each rating.

• Point out any issues or positive aspects that influenced your decision.
Languages: Preference Ranking and Rewrites - Next Steps

Step 4: Complete the Preference Ranking


Description

You now need to rank the 2 responses based on specific characteristics: Instruction

Following, Writing Quality, and Truthfulness. Use a preference ranking score and follow
these guidelines: if the difference between the responses is mainly subjective or varies

based on opinion, choose one of the middle three preference scores, as these reflect a

more neutral stance.

Key Actions

• Compare responses based on the specified characteristics.


• Select a preference score, leaning towards a middle score if the differences are

subjective or minor.

Step 5: Provide a Likert Justification for Preference


Ranking
Description
Languages:
In thisPreference Ranking
field, write and Rewrites
a justification - Next
for your Steps ranking. Clearly explain your thought
preference

process and why you believe the selected response is better than the other. Be detailed
and concrete in your reasoning.

Key Actions
• Provide specific examples from each response to support your ranking.

• Write your justification in English, ensuring clarity and coherence in your explanation.

Step 6: Task Response Rewrite (If Applicable)


Description
In this step, the task requires a response rewrite to improve the existing response.

Key Actions
• Make necessary adjustments to the response, focusing on any areas that didn’t meet

the prompt requirements.

• Confirm that the rewrite aligns closely with the task’s objectives and expectations.
Languages: Preference Ranking and Rewrites - Next Steps

You might also like