Quantifier disagreement from round 6
Quantifiers need to determine if a praise is to be quantified, dismissed, or marked as duplicate of another praise. For many cases we seem to be lacking community agreement on how to decide the status of a praise. Here I summarize some example disagreements on dismissal and duplication from round 6 of quantification. I also propose some solutions for cases but of course this is meant to be an open discussion! Hopefully we can add some more specified quantifier guidance and in general praise ettiques for the community.
The data collection process is described at the end of the post for those data-inclined readers.
Dismissal disagreement
Right now the established rules for dismissal, according to this ārules of quantificationā post, include:
- If itās the same person giving the same praise, dismiss.
- If the praise is about forum posting or github contribution, dismiss (because sourcecred will be able to capture that).
Below are the other cases that quantifiers have disagreement on whether they count as dismissal or not.
-
Twitter/ social media contribution:
- Example praise: āfor mentioning or retweeting TE Commons on socials the past week! Thank you for helping us grow the Token Engineering Commons community and spreading the message!
š¼ā - Some people give it a low score or even 0, others dismiss it right away
- Suggestion: Clarify if we have a way to automatically capture this ā seems like it is not happening now. So probably should treat it as meeting attendence ā a lower score below 5.
- Example praise: āfor mentioning or retweeting TE Commons on socials the past week! Thank you for helping us grow the Token Engineering Commons community and spreading the message!
-
Incomplete message with unclear meaning:
- Example praise: āfor tsā/āfot itā/āforā
- Suggestion: Make it clear to quantifiers that should all dismiss that.
-
General mention of the project but not the actual contribution:
- Example praise: āfor the analysis dashboardā/āfor their work on the Rewards WGā
- Most people would still give a score for this but some would dismiss. Yet for those who give a score, because of the vagueness of praise, the quantified score will vary a lot.
- Suggestion: First of all this roots from praise givers not specifying the action of praise, so more education/hints would be needed. Then thereās big difference of how much context a quantifier could have. We may suggest for quantifiers with less context to give less score, but donāt dismiss it, then the average will be bumped up by more knowledgeable quantifiers if the contribution is actually big.
-
Action related to other TE related organizations but not TE:
- Example praise:āfor his work in Giveth and for supporting ETHColombia"/"for great participation at the TE Academy Team Sync meeting yesterday and to guide us
very excited with this team to grow and growā - Many quantifiers may not realize this is an event related with Giveth/TE Academy, not TE.
- Suggestion: emphasize this policy to quantifiers
- Example praise:āfor his work in Giveth and for supporting ETHColombia"/"for great participation at the TE Academy Team Sync meeting yesterday and to guide us
-
Action seems unrelated to TE (some personal interaction?):
- Example praise: āfor being man enough to know how to change a tireā.
- Suggestion: needs discussion to agree on a policy?
Duplication disagreement
Right now the agreed rule for duplication is: different praise giver, same contribution praised and the same week = duplicate.
Whatās vague is whether itās the same contribution depending on the phrasing of a praise. This is definitely a tricky problem but letās see if we can identify some typical categories of confusion.
Below are some examples where some quantifiers would mark one praise as a separate praise yet others see it as a duplicate.
One important thing I get from this data is that, this is not only a discussion for quantifiers, but really, for praise givers: how to phrase your praise so that the action has been done is clear, and the impact is understandable?
- Same event, more action description:
- Example praise 1: āfor engaging and participating on the Orientation call! Amazing to have you here!ā v.s. āfor joining the orientation callā
- Example praise 2: āfor recording ALL the calls
ā v.s. āfor recording and uploading all the calls behind the scenesā
- Example praise 3: āfor joining the meeting that discussed extending the deadline to debate about proposalsā v.s. for asking questions and participating in the Stewards debate call"
- Suggestion: additional action and quality of action should not be a duplicate, but a new praise but only evaluated with the additional part.
- Praise the outcome v.s. the action:
- Example praise: āfor a great AMA on bonding curvesā v.s. āfor hosting the ABC AMAā
- Example praise: āfor his work on the params and partiesā v.s. āfor all the love you build at Param Partiesā
- Suggestion: for the praise giver side, encourage more praise giving with action description. for the quantifier, adding more description of action and effort should be counted as an additional part to evaluate the score. Vice versa: the additional praise for impact/outcome should be counted too.
- Adding personal expression for the same action
- Example praise: āfor the param parties poap. Such a pleasant surprise
ā v.s. āfor the param poap so cuteā:smiling_face_with_three_hearts: im honored having itā
- Suggestion: similiar as above ā only evaluate the additional part.
- Example praise: āfor the param parties poap. Such a pleasant surprise
- Vague similiar expression:
- Example praise 1: āfor all the work that they did on Commons Swarm this weekā v.s. āfor carrying the Commons Swarm forwardā
- Suggestion: i think it could be marked as duplicateā¦needs discussion?
Technical part: data processing
In the updated RAD dashboard analysis pipeline, we are able to generate 2 new tables: one table with all the praises that quantifiers have disagreement on whether to dismiss or not; another table with all the praises that quantifiers disagree on duplicate, with the āsupposed duplication messageā versus original message side by side so the reviewer can easily look at them.
Then we need to figure out the categories of disagreement by our own judgmentā¦if you have any thought on how to do more automated analysis on this, let us know!