Claim-Matching Data Annotation

The University of Texas at Arlington (UTA)
Informed Consent for Studies with Adults

TITLE OF RESEARCH PROJECT

Data Annotation for Claim-Matching

RESEARCH TEAM
PRINCIPAL INVESTIGATOR

Chengkai Li
Professor and Associate Chair
Department of Computer Science and Engineering
(817) 272-0162
cli@uta.edu

STUDENT PERSONNEL

Theodora Toutountzi
Ph.D. Candidate Student
Department of Computer Science and Engineering
theodora.toutountzi@mavs.uta.edu

IMPORTANT INFORMATION ABOUT THIS RESEARCH PROJECT

The research team above is conducting a research study that aims to collect annotated data (often called labeled data in the field of artificial intelligence and machine learning) that would be used to train a machine learning model for measuring how helpful an existing fact-check is in vetting a factual claim that was not fact-checked before. Specifically, you will be provided with pairs of sentences to annotate. Each pair consists of a fact-check published by fact-checking outlets such as PolitiFact, FactCheck.org, Washington Post Fact Checker, and Snopes and an unchecked claim derived from political debates, speeches, interviews, etc. For each pair, you will be asked to decide how helpful the fact-check is in vetting the unchecked political claim. You will continue this same process for as many available pairs as you wish.

For that, you choose one of the five options: Not at all helpful, Slightly helpful, Somewhat helpful, Very helpful, and Extremely helpful. The machine learning model can become useful in automating fact-checking. Imagine an audience of a presidential debate. When a presidential candidate repeats a factual statement that has been marked as false by professional fact-checkers, the machine learning model can help identify previous relevant fact-checks and present them to the audience. This can help mitigate the spreading of misinformation, which is a fundamental threat to the modern society.

You can choose to participate in this research study if you are over the age of 18 and fluent in English. Please note that you are not eligible for this study if you are not an English fluent speaker or are younger than 18.

You might want to participate in this study if you would like to get a taste of how technology might help vet factual statements and familiarize yourself with the fact-checking process. However, you might not want to participate in this study if you do not have the time or are not interested in the fact-checking process.

This study has been reviewed and approved by an Institutional Review Board (IRB). An IRB is an ethics committee that reviews research with the goal of protecting the rights and welfare of human research subjects. Your most important right as a human subject is informed consent. You should take your time to consider the information provided by this form and the research team and ask questions about anything you do not fully understand before making your decision about participating.

TIME COMMITMENT

If you decide to participate in this study, you should know that you have the full freedom to decide the duration of your participation, from a couple of minutes to as long as you prefer, for as many times as you want until the end of the study.

RESEARCH PROCEDURES

If you decide to participate in this research study, this is the list of activities that we will ask you to perform as part of the research:

Go to https://idir.uta.edu/claimpairs_annotation/ and click on the “Sign Up” button at the top right corner.
This will open this consent form that you need to accept if you want to participate in the study. If you decide that you do not wish to participate, click the “Decline” button or close your web browser.
If you click the “Accept” button you will be presented with the account registration form to create an account by providing a username, an email address, and a password. Note that only the first three letters of your username will be visible to other participants in this study. We will use your email address only to communicate with you regarding this study. We will delete your username, email and password once we have concluded the study.
We will send you a verification email. Once you click the verification link in the email, your account will be activated. Check your junk/spam email box if you haven't received it.
Using the registered account, you will log into our data annotation website to the instructions page. Read the instructions carefully and continue to the annotation page. On the top right corner of the webpage, there is a link to the instructions page if you need to reread the instructions at any time.
On the annotation page, you will be asked to read a pair of sentences (a fact-check and a political claim). Your task is to decide how helpful the fact-check is in vetting the political claim. For that, you should choose one of the five options: Not at all helpful, Slightly helpful, Somewhat helpful, Very helpful, and Extremely helpful, or the “Skip this Pair” option. The data annotation website will record your choices as well as the timestamp of each interaction with the website.
After submitting your choice, a new pair of sentences will appear for annotation. This process will continue until you decide to stop by clicking the “Log Out” button or by closing the web browser. If you click the “Log Out” button, the system will not ask to verify your choice to quit your session. Although, If you decide to close your web browser, it may ask you to confirm your intention of terminating your session. Of course, you can always come back at a later time with your existing account and continue to annotate pairs.
If you want to modify the choice you made on a previous pair, click “Modify my previous responses”, where you can see all pairs on which you provided annotations, ordered by the timestamps. After your modification, your work quality score may become different.
After the first 50 annotations, the “Leaderboard” button will be activated. By clicking it, a pop-up window will show your work quality score, total points, and where you stand among other participants. Only the first three letters of usernames (instead of full names) will be displayed on the leaderboard. The leaderboard calculation is updated every 15-30 pairs reflecting your new work quality score and total points.

To gauge the quality of your choices, we use several “gold standard” sentence pairs, for which we have “correct choices” selected by research experts on the subject. Your choices will be compared with the “gold standard” choices to estimate your work quality level. There is no visual distinction between a pair from the “gold standard” set or a pair outside that set.
Below is a list of tips for improving your work quality score. You can find a copy of this list under each annotation pair.

(1) Carefully examine each pair of fact-check and claim.
(2) Contextual information (such as the fact-check summary, claimant information, and claim context) may help you form answers.
(3) Review the "Instructions" to understand the examples.
(4) Don't guess. Skip the pairs that you are not sure about.
(5) Modify previous responses if necessary.
(6) You may be tempted to pick easy/short claims to work on by clicking "Skip this pair". Keep in mind that our work quality score calculation formula has a component that accounts for the length/complexity of claims as well as how many pairs are skipped. We discourage excessive skipping. Nevertheless, if you are not confident on a question, it is still better to skip, because every single mistake will lower your work quality score.

Whenever you make one mistake, our algorithm lowers your work quality score, which means you get less points for every pair you have annotated. It takes multiple correct answers to make up for every single mistake and get the work quality score back to the previous value. If your current work quality score is 0 or very low, it is because our algorithm detected many mistakes in your answers. The best thing to do is to review your answers and modify them if necessary. If your work quality score is 0, it might actually be negative internally. If you continue to answer new questions, it will take MANY questions before you can see positive and improving work quality score.

If you have only annotated 50-150 pairs, the work quality score based on the small sample may not reflect your true work quality. It will become more robust once you have annotated more pairs.

We may email you about our optional data annotation training workshops in our Lab (ERB - Room 414) or online through MS Teams, which are available to all participants. The purpose of the workshops is to review the information provided on the instructions page, discuss any questions you might have, and annotate some pairs. We expect this activity to be helpful in improving annotation quality. Note that other participants may also be present in the data annotation workshops.

POSSIBLE BENEFITS

You will get a taste of how technology might help vet factual statements. Furthermore, the research outcome might lead to advancement in data-driven fact-checking, which tackles an important sciential challenge.

POSSIBLE RISKS/DISCOMFORTS

This research study is not expected to pose any additional risks beyond what you would normally experience in your regular everyday life. However, if you experience discomfort, please inform the research team, and quit the study without any consequence to you.

To minimize the risk of your personal information being exposed, we are storing your personal data on UTA servers, limiting its access to the research team only.

Your decision to participate or not participate will not influence your grades or future research opportunities in any way.

COMPENSATION

You will not be compensated for your participation.

ALTERNATIVE OPTIONS

There are no alternative procedures offered for this study. However, you can elect not to participate in the study or quit at any time at no consequence.

CONFIDENTIALITY

The research team is committed to protecting your rights and privacy as a research participant. All paper and electronic data collected from this study will be stored in a secure location on the UTA campus and/or a secure UTA server for at least three (3) years after the end of this research.

The results of this study may be published and/or presented without naming you as a participant. The data collected about you for this study may be used for future research studies that are not described in this consent form. If that occurs, an IRB would first evaluate the use of any information that is identifiable to you, and confidentiality protection would be maintained.

While absolute confidentiality cannot be guaranteed, the research team will make every effort to protect the confidentiality of your records as described here and to the extent permitted by law. In addition to the research team, the following entities may have access to your records, but only on a need-to-know basis: the U.S. Department of Health and Human Services and the FDA (federal regulating agencies), the reviewing IRB, and sponsors of the study.

CONTACT FOR QUESTIONS

Questions about this research study may be directed to Dr. Chengkai Li at (817) 272-0162 or cli@uta.edu and Theodora Toutountzi at theodora.toutountzi@mavs.uta.edu. Any questions you may have about your rights as a research subject or complaints about the research may be directed to the Office of Research Administration; Regulatory Services at 817-272-3723 or regulatoryservices@uta.edu.

CONSENT

By clicking “Accept”, you are confirming that you understand the study’s purpose, procedures, potential risks, and your rights as a research subject. By agreeing to participate, you are not waiving any of your legal rights. You can refuse to participate or discontinue participation at any time, with no penalty or loss of benefits that you would ordinarily have. Please click “Accept” if you are at least 18 years of age and voluntarily agree to participate in this study.

Thank you for participating in Claim-Matching data annotation!

Leaderboard