Aptitude Test Contest - Phase 2

The task in this contest is to create a set of three Quizzes that can be used to test a person's knowledge of the English Language, their Mathematical Ability and their capacity for Spatial & Logical Thinking. For more general details, see the original contest announcement.

Contest Outline

Phase 1 - Quiz Set, Cover Letter, and Review and Submission.
(📍) Phase 2 - Peer Review and Rebuttal.
Phase 3 - Final Review by Telegram.

About Phase 2

In this Phase, you will peer review 3 Quiz Sets (9 Quizzes) from other participants by testing and evaluating them against the guidelines below. Once you're done, you'll post your findings as a comment and rate the Quizzes. Finally, you'll spend some time in rebuttal.

Your work in both Peer Review and Rebuttal will be rated alongside your Quiz Set, and influence the final prize that you receive. Therefore, this is an essential part of the contest - it will not be possible to reward those who choose not to participate in Phase 2.

💡 Please subscribe to @AptitudeTestContest. We'll send reminders, new FAQ entries, and any other important news about the contest there instead of in the main @contest channel.

2.1 - Peer Review

⏰ ~7 days will be spent here. From 2021-02-14 to 2021-02-22 at 23:50:00 (Dubai time, check local time here).

You will have received a message from @ContestBot with links to 3 Quiz Sets which we have chosen for you to peer review.

When you receive Peer Review comments on your Quizzes, do not reply yet. We will provide instructions on how to do that, as part of Rebuttal.

2.1.1 Peer Review Steps

This section outlines in detail how you should carry out your peer reviews for each Quiz Set.

Before you start, you should read this entire document through at least once.

Step	Task
1	Complete each Quiz in the Set. Try your best to get a good score on the first try as we will use this data when we assess your reviews. It's a good idea to take notes as you're doing each Quiz, of potential issues and exceptional elements.
2	After you've completed a Quiz, press the Review Quiz button to see which answers were marked as correct, all of the explanations and the original question order. You must use this view when writing your comment and when making question tag references. Otherwise, your review may refer to the wrong questions when posted. After completing every Quiz in the Set, you will be able to see the Cover Letter.
3	Now, check each Quiz against the Issues List below, both based on your experience and by looking through the Quizzes again. It may be useful to read the Cover Letter and focus on potentially weak areas.
4	Look out for any Exceptional Elements that are present.
5	Compose and post your full Peer Review comment on each Quiz, detailing the issues and any exceptional elements you found. You must follow the Commenting Guide and add full details where necessary.
6	Rate each Quiz based on the Rating Guide.

2.1.2 Issues List

Note: Major issues are marked with a ⚠️.

Type	Issue	Description
QUESTIONS	Questions which are too simple or too hard. ⚠️	Quizzes should aim to target mid-to-high-intellect adults. Use this for questions which test concepts which are too easy or heavily depend on the knowledge of non-basic concepts. Also includes cases where the question timer is too long or too short for certain questions. ISSUE TEMPLATE: remember to follow the commenting guide `X. These questions are too simple or too hard: #q1, #q2: Too hard, requires knowledge of nuclear physics. #q3, #q4, #q27: Too hard to complete in the given time (2m), the problems are complex and include an element of trial and error, so it seems like 5m would be required.`
QUESTIONS	Questions which are not Google-proof.	Where answers to questions are easily obtainable from Google or other search engines. ISSUE TEMPLATE: remember to follow the commenting guide `X. These questions are not Google-proof: #q1: https://www.google.com/search?q=answer%20can%20be%20obtained%20from%20this%20search.`
QUESTIONS	Misleading or trick questions and questions which are confusing due to the omission of key data. ⚠️	For example, questions which ask you to find the number of triangles in a shape like this, but fail to clarify if larger triangles which are made up of multiple smaller triangles should be counted or not. ISSUE TEMPLATE: remember to follow the commenting guide `X. These questions are misleading: #q1: It is not made clear enough that the shape is able to be rotated. #q2: Media shows example with letters, but question asks for numbers, and there is no way to infer what should be done.`
ANSWERS	More than one correct answer is present.	For cases where you suspect a question actually has multiple answer options which could be correct. ISSUE TEMPLATE: remember to follow the commenting guide `X. There is more than one correct answer for these questions: #q1: Following the steps in the explanation, both of the top options are correct. #q2: "12" and "24" are both valid here.`
ANSWERS	Answer is incorrect.	For cases where you suspect the answer is not correct. ISSUE TEMPLATE: remember to follow the commenting guide `X. The answers for these questions are not correct: #q1: Following the steps in the explanation, the first option is correct, not the second. #q2: "13" doesn't seem correct here. "16" seems correct and is listed, but as an incorrect answer.`
ANSWERS	Most alternative answer options are obviously incorrect. ⚠️	For cases where most alternative answer options are obviously incorrect or implausible, making it easier to guess the correct option. ISSUE TEMPLATE: remember to follow the commenting guide `X. Most of the answer options for these questions are obviously incorrect or implausible: #q1, #q2, #q3: Options here contain made up words. #q4: The question is clearly asking for a yes/no answer, but many other options are also present aside from "Yes" and "No".`
QUESTIONS / ANSWERS	Unoriginal questions and answers. ⚠️	Where you are able to demonstrate that questions and answers have been taken from elsewhere and are not original. ISSUE TEMPLATE: remember to follow the commenting guide `X. These questions are unoriginal: #q1: Taken from https://example.org/quiz/question-11. #q2: Taken from https://example.org/quiz2/question-9.`
EXPLANATIONS	Insufficient explanations.	When explanations do not clearly expain why the correct answer is right, or why the incorrect answers are wrong. ISSUE TEMPLATE: remember to follow the commenting guide `X. The explanations for these questions do not clearly explain why the correct answer is right, or why the incorrect answers are wrong: #q1, #q2`
MEDIA	Media used as question text.	Where media is used as the question text (for example, a screenshot of text). In cases where the text would not fit in the question field, it is expected that pre-poll text is used instead. Also includes cases where media exactly repeats the question text, which may distract and confuse quiz-takers - even if it has other content besides the question text. ISSUE TEMPLATE: remember to follow the commenting guide `X. The media in these questions repeats, or is used as, the question text: #q1, #q2`
MEDIA	Low quality media. ⚠️	Media which is low quality or blurry, to the point where it is difficult to understand or see crucial details. Also includes cases where the media looks bad or low-effort. ISSUE TEMPLATE: remember to follow the commenting guide `X. The media in these questions is low quality or blurry: #q1, #q2`
MEDIA	Errors in media.	Media which contains either graphical/textual errors. ISSUE TEMPLATE: remember to follow the commenting guide `X. The media in these questions contains errors: #q1: Some lines on the diagram are not straight, this introduces some confusion when trying to work out the answer. #q2: The text "purple" is not fully contained in its square.`
MEDIA	Inconsistent media style. ⚠️	Where the style of media is not consistent and there is no clear reason for this, whether that be in font, size, color scheme, etc. ISSUE TEMPLATE: remember to follow the commenting guide `X. The media in these questions is not consistent with the style of the rest: #q1, #q2`
GENERAL	Spelling and/or grammar issues.	Mistakes in spelling and/or grammar. This includes cases where our guidelines about use of American English have been ignored. ISSUE TEMPLATE: remember to follow the commenting guide `X. These questions contain spelling and/or grammar mistakes: #q1: "... cannot be build by tomorrow." should be "cannot be built by tomorrow.". #q2: "defiantly" should be "definitely".`
GENERAL	Requires local or overly-niche knowledge.	Where questions require local knowledge or knowledge of overly-niche topics. This includes cases where our guidelines about local knowledge, currency and measurements have been ignored. ISSUE TEMPLATE: remember to follow the commenting guide `X. These questions require local or overly-niche knowledge: #q1: Requires knowledge of how the highways in Scotland work. #q2: Requires knowledge of who the "Axis powers" were. #q3: Euros (€) and imperial measurements are used.`
GENERAL	Too much reliance on the same mechanics. ⚠️	Where questions rely on the same mechanics too much. See Q: Is it encouraged to be inventive with my questions?. ISSUE TEMPLATE: remember to follow the commenting guide `X. These questions rely too much on the same mechanics: #q1, #q2, #q3, #q4, #q5, #q6, #q7, #q8, #q9, #q10: All use the same “which shape is out of place here” mechanic. #q11, #q12, #q13, #q14, #q15, #q16, #q17: All use the same "find the missing number" mechanic.`
GENERAL	Inconsistent style or formatting.	Where style or formatting changes between questions without any obvious reason. ISSUE TEMPLATE: remember to follow the commenting guide `X. These questions contain inconsistent style or formatting: #q1, #q2: "i, ii, iii" is used even though the rest of the Quiz used "a, b, c". #q3: 3 answer options have spaces between each sub-option, but 2 don't. #q4: "( )" is used to indicate a gap, but the rest of the quiz uses "___".`

2.1.3 Exceptional Elements

While you're reviewing the Quizzes, you should keep an eye out for exceptional elements - special, inventive or otherwise extraordinary elements of a Quiz which are impressive.

It is possible for a Quiz with many issues to still have exceptional elements.

Some examples of exceptional elements:

Unique and inventive question types which have clever and thoughtful design.
Consistently excellent, high-quality and relevant media which has 0 issues.
Innovative and varied testing approaches which have real value, throughout.

You are encouraged to use your judgement and note other elements or features of Quizzes which stand out as exceptional.

If you find exceptional elements, make note of them in your comment and add a short explanation of why you think they are exceptional. See the sample comments below for an example of how to format this.

2.1.4 Commenting Guide

1. Use Templates. For the sake of consistency, always use the same format as the Issue Templates provided here. Your comment must start with PEER REVIEW.

2. Adapt templates. Adapt the templates you use to the particular Quiz you're dealing with. Obviously, you should replace the example question tags and reasoning, etc.

Where our templates indicate that more information is required than just a list of question tags, you must add your own reasoning.

3. Be objective. Be fair and objective with your peer review, listing fake issues will not be tolerated.

4. Send ONE comment. Others will get a notification when you add a comment to their Quiz. You are required to collate all issues and exceptional elements together, and write one comment per Quiz. See the sample comments below.

5. Use numbered lists. Always place issues in a numbered list with line breaks. We don't expect to see any comments that still have the placeholder X. numbering…

6. When referring to questions, use question tags. You must use the correct formatting (e.g. #q1 for Question 1) in your comment. Once your comment is posted, question tags written like this will create a link that opens the relevant question on a separate page.

Note: It is crucial that you refer to the original question order shown when clicking the Review Quiz button. Otherwise, your peer review may refer to the wrong questions when posted.

While in this review page, clicking the “Question 1 / 30” text will automatically add a correctly formatted question tag to your comment.

Sample Comments

A. Quiz with two issues and no exceptional elements:

PEER REVIEW

1. The answers for these questions are not correct:
#q1: following the steps in the explanation, the first option is correct, not the second.

#q2: "13" doesn't seem correct here. "16" seems correct and is listed, but as an incorrect answer.

2. The media in these questions is low quality or blurry:
#q3, #q4

B. Quiz with two issues and two exceptional elements:

PEER REVIEW

1. The answers for these questions are not correct:
#q1: following the steps in the explanation, the first option is correct, not the second.

#q2: "13" doesn't seem correct here. "16" seems correct and is listed, but as an incorrect answer.

2. These questions are not Google-proof:
#q9: https://www.google.com/search?q=answer%20can%20be%20obtained%20from%20this%20search.

Exceptional elements:
1. #q3, #q4, #q5, #q6, #q7 and #q21, #q24, #q25 demonstrate completely unique and inventive question types, which are a great way to assess linguistic ability.

2. The entire Quiz uses excellent, high-quality and relevant media which has 0 issues.

2.1.5 Rating Guide

You are required to rate each Quiz with 1 Star, 3 Stars, or 5 Stars. Follow the guide below to decide which rating you should use.

Rating	Description
1 Star	• The Quiz has many major issues which fundamentally impact the quality or usefulness. • Fixing the issues in this Quiz would require replacing the majority of the content. • If you received this Quiz (without any fixes) from an employer, you would question their judgement.
3 Stars	• The Quiz has some issues which impact the quality or usefulness. • Fixing the issues in this Quiz would be possible, and not require huge changes. • If you received this Quiz (after the fixes) from an employer, you would NOT leave with a negative impression.
5 Stars	• The Quiz has 0 issues which impact the quality or usefulness. • Fixing the other issues in this Quiz would be possible within a day or two, and not require big changes. • If you received this Quiz (after the fixes) from an employer, you would be impressed by the quality.

2.2 - Rebuttal

⏰ ~3 days will be spent here. From 2021-02-26 to 2021-03-01 at 23:50:00 (Dubai time, check local time here).

You now have some time to check all of the Peer Review comments you received. You'll post rebuttals to each, outlining which issues you agree should be fixed (and how, if it differs from a suggested approach), and which you disagree with.

You are more likely to get a reward by acknowledging valid issues, as opposed to trying to frame every reported issue as completely invalid.

2.2.1 Rebuttal Steps

This section outlines in detail how you should carry out rebuttal for each of your Quizzes.

Before you start, you should read this section and all of the peer review comments that have been left on your Quizzes at least once.

Step	Task
1	Look through all of the peer review comments on the Quiz.
2	Use your judgement to figure out which issues need to be fixed, and which are not issues in your case - you must be able to provide very strong justification for the issues which you mark as invalid.
3	Reply to each comment, following the Rebuttal Comment Templates below.

Rebuttal Comment Templates

A. Replying to a comment where all listed issues seem valid and should be fixed:

REBUTTAL

Valid issues which should be fixed:
1, 2, 3, 5, 6.

4. Extra notes can be added in this format, if needed.

7. This is a valid issue, but will be fixed by lorem veniam quia instead of ipsum accusamus drunt.

B. Replying to a comment where some listed issues are invalid:

REBUTTAL

Valid issues which should be fixed:
1, 2, 3, 5.

4. Extra notes can be added in this format, if needed.

6. Agree about #q1 and #q2, but not the rest. See below.

7. This is a valid issue, but will be fixed by lorem veniam quia instead of ipsum accusamus drunt.

Invalid issues:
6. #q3, #q4 and #q5 do not have this issue because lorem ipsum, which means veniam accusamus.

8. Mandatory text fully and clearly explaining a strong and convincing reason why the issue is not valid.