Quantifier disagreement from round 6
Quantifiers need to determine if a praise is to be quantified, dismissed, or marked as duplicate of another praise. For many cases we seem to be lacking community agreement on how to decide the status of a praise. Here I summarize some example disagreements on dismissal and duplication from round 6 of quantification. I also propose some solutions for cases but of course this is meant to be an open discussion! Hopefully we can add some more specified quantifier guidance and in general praise ettiques for the community.
The data collection process is described at the end of the post for those data-inclined readers.
Dismissal disagreement
Right now the established rules for dismissal, according to this “rules of quantification” post, include:
- If it’s the same person giving the same praise, dismiss.
- If the praise is about forum posting or github contribution, dismiss (because sourcecred will be able to capture that).
Below are the other cases that quantifiers have disagreement on whether they count as dismissal or not.
-
Twitter/ social media contribution:
- Example praise: “for mentioning or retweeting TE Commons on socials the past week! Thank you for helping us grow the Token Engineering Commons community and spreading the message!
🏼” - Some people give it a low score or even 0, others dismiss it right away
- Suggestion: Clarify if we have a way to automatically capture this – seems like it is not happening now. So probably should treat it as meeting attendence – a lower score below 5.
- Example praise: “for mentioning or retweeting TE Commons on socials the past week! Thank you for helping us grow the Token Engineering Commons community and spreading the message!
-
Incomplete message with unclear meaning:
- Example praise: “for ts”/“fot it”/“for”
- Suggestion: Make it clear to quantifiers that should all dismiss that.
-
General mention of the project but not the actual contribution:
- Example praise: “for the analysis dashboard”/“for their work on the Rewards WG”
- Most people would still give a score for this but some would dismiss. Yet for those who give a score, because of the vagueness of praise, the quantified score will vary a lot.
- Suggestion: First of all this roots from praise givers not specifying the action of praise, so more education/hints would be needed. Then there’s big difference of how much context a quantifier could have. We may suggest for quantifiers with less context to give less score, but don’t dismiss it, then the average will be bumped up by more knowledgeable quantifiers if the contribution is actually big.
-
Action related to other TE related organizations but not TE:
- Example praise:“for his work in Giveth and for supporting ETHColombia"/"for great participation at the TE Academy Team Sync meeting yesterday and to guide us
very excited with this team to grow and grow” - Many quantifiers may not realize this is an event related with Giveth/TE Academy, not TE.
- Suggestion: emphasize this policy to quantifiers
- Example praise:“for his work in Giveth and for supporting ETHColombia"/"for great participation at the TE Academy Team Sync meeting yesterday and to guide us
-
Action seems unrelated to TE (some personal interaction?):
- Example praise: “for being man enough to know how to change a tire”.
- Suggestion: needs discussion to agree on a policy?
Duplication disagreement
Right now the agreed rule for duplication is: different praise giver, same contribution praised and the same week = duplicate.
What’s vague is whether it’s the same contribution depending on the phrasing of a praise. This is definitely a tricky problem but let’s see if we can identify some typical categories of confusion.
Below are some examples where some quantifiers would mark one praise as a separate praise yet others see it as a duplicate.
One important thing I get from this data is that, this is not only a discussion for quantifiers, but really, for praise givers: how to phrase your praise so that the action has been done is clear, and the impact is understandable?
- Same event, more action description:
- Example praise 1: “for engaging and participating on the Orientation call! Amazing to have you here!” v.s. “for joining the orientation call”
- Example praise 2: “for recording ALL the calls
” v.s. “for recording and uploading all the calls behind the scenes”
- Example praise 3: “for joining the meeting that discussed extending the deadline to debate about proposals” v.s. for asking questions and participating in the Stewards debate call"
- Suggestion: additional action and quality of action should not be a duplicate, but a new praise but only evaluated with the additional part.
- Praise the outcome v.s. the action:
- Example praise: “for a great AMA on bonding curves” v.s. “for hosting the ABC AMA”
- Example praise: “for his work on the params and parties” v.s. “for all the love you build at Param Parties”
- Suggestion: for the praise giver side, encourage more praise giving with action description. for the quantifier, adding more description of action and effort should be counted as an additional part to evaluate the score. Vice versa: the additional praise for impact/outcome should be counted too.
- Adding personal expression for the same action
- Example praise: “for the param parties poap. Such a pleasant surprise
” v.s. “for the param poap so cute​:smiling_face_with_three_hearts: im honored having it”
- Suggestion: similiar as above – only evaluate the additional part.
- Example praise: “for the param parties poap. Such a pleasant surprise
- Vague similiar expression:
- Example praise 1: “for all the work that they did on Commons Swarm this week” v.s. “for carrying the Commons Swarm forward”
- Suggestion: i think it could be marked as duplicate…needs discussion?
Technical part: data processing
In the updated RAD dashboard analysis pipeline, we are able to generate 2 new tables: one table with all the praises that quantifiers have disagreement on whether to dismiss or not; another table with all the praises that quantifiers disagree on duplicate, with the “supposed duplication message” versus original message side by side so the reviewer can easily look at them.
Then we need to figure out the categories of disagreement by our own judgment…if you have any thought on how to do more automated analysis on this, let us know!