Quantifier disagreement on duplication and dismissal -- analysis from round 6

Quantifier disagreement from round 6

Quantifiers need to determine if a praise is to be quantified, dismissed, or marked as duplicate of another praise. For many cases we seem to be lacking community agreement on how to decide the status of a praise. Here I summarize some example disagreements on dismissal and duplication from round 6 of quantification. I also propose some solutions for cases but of course this is meant to be an open discussion! Hopefully we can add some more specified quantifier guidance and in general praise ettiques for the community.

The data collection process is described at the end of the post for those data-inclined readers.

Dismissal disagreement

Right now the established rules for dismissal, according to this “rules of quantification” post, include:

  1. If it’s the same person giving the same praise, dismiss.
  2. If the praise is about forum posting or github contribution, dismiss (because sourcecred will be able to capture that).

Below are the other cases that quantifiers have disagreement on whether they count as dismissal or not.

  • Twitter/ social media contribution:

    • Example praise: “for mentioning or retweeting TE Commons on socials the past week! Thank you for helping us grow the Token Engineering Commons community and spreading the message! :pray:🏼”
    • Some people give it a low score or even 0, others dismiss it right away
    • Suggestion: Clarify if we have a way to automatically capture this – seems like it is not happening now. So probably should treat it as meeting attendence – a lower score below 5.
  • Incomplete message with unclear meaning:

    • Example praise: “for ts”/“fot it”/“for”
    • Suggestion: Make it clear to quantifiers that should all dismiss that.
  • General mention of the project but not the actual contribution:

    • Example praise: “for the analysis dashboard”/“for their work on the Rewards WG”
    • Most people would still give a score for this but some would dismiss. Yet for those who give a score, because of the vagueness of praise, the quantified score will vary a lot.
    • Suggestion: First of all this roots from praise givers not specifying the action of praise, so more education/hints would be needed. Then there’s big difference of how much context a quantifier could have. We may suggest for quantifiers with less context to give less score, but don’t dismiss it, then the average will be bumped up by more knowledgeable quantifiers if the contribution is actually big.
  • Action related to other TE related organizations but not TE:

    • Example praise:“for his work in Giveth and for supporting ETHColombia"/"for great participation at the TE Academy Team Sync meeting yesterday and to guide us :raised_hands:t3::raised_hands:t3::raised_hands:t3::raised_hands:t3: very excited with this team to grow and grow
    • Many quantifiers may not realize this is an event related with Giveth/TE Academy, not TE.
    • Suggestion: emphasize this policy to quantifiers
  • Action seems unrelated to TE (some personal interaction?):

    • Example praise: “for being man enough to know how to change a tire”.
    • Suggestion: needs discussion to agree on a policy?

Duplication disagreement

Right now the agreed rule for duplication is: different praise giver, same contribution praised and the same week = duplicate.
What’s vague is whether it’s the same contribution depending on the phrasing of a praise. This is definitely a tricky problem but let’s see if we can identify some typical categories of confusion.
Below are some examples where some quantifiers would mark one praise as a separate praise yet others see it as a duplicate.
One important thing I get from this data is that, this is not only a discussion for quantifiers, but really, for praise givers: how to phrase your praise so that the action has been done is clear, and the impact is understandable?

  • Same event, more action description:
    • Example praise 1: “for engaging and participating on the Orientation call! Amazing to have you here!” v.s. “for joining the orientation call”
    • Example praise 2: “for recording ALL the calls :mechanical_arm:” v.s. “for recording and uploading all the calls behind the scenes”
    • Example praise 3: “for joining the meeting that discussed extending the deadline to debate about proposals” v.s. for asking questions and participating in the Stewards debate call"
    • Suggestion: additional action and quality of action should not be a duplicate, but a new praise but only evaluated with the additional part.
  • Praise the outcome v.s. the action:
    • Example praise: “for a great AMA on bonding curves” v.s. “for hosting the ABC AMA”
    • Example praise: “for his work on the params and parties” v.s. “for all the love you build at Param Parties”
    • Suggestion: for the praise giver side, encourage more praise giving with action description. for the quantifier, adding more description of action and effort should be counted as an additional part to evaluate the score. Vice versa: the additional praise for impact/outcome should be counted too.
  • Adding personal expression for the same action
    • Example praise: “for the param parties poap. Such a pleasant surprise :slightly_smiling_face:” v.s. “for the param poap so cute​:smiling_face_with_three_hearts: im honored having it”
    • Suggestion: similiar as above – only evaluate the additional part.
  • Vague similiar expression:
    • Example praise 1: “for all the work that they did on Commons Swarm this week” v.s. “for carrying the Commons Swarm forward”
    • Suggestion: i think it could be marked as duplicate…needs discussion?

Technical part: data processing

In the updated RAD dashboard analysis pipeline, we are able to generate 2 new tables: one table with all the praises that quantifiers have disagreement on whether to dismiss or not; another table with all the praises that quantifiers disagree on duplicate, with the “supposed duplication message” versus original message side by side so the reviewer can easily look at them.

Then we need to figure out the categories of disagreement by our own judgment…if you have any thought on how to do more automated analysis on this, let us know!

5 Likes

This is awesome analysis, Zhiwei. I was just thinking today about how we work you all to build praise training into the overall TEC community orientation process. This kind of analysis will be crucial in pointing us towards the clarity for good training. I can see this work playing a big role in a Praise FAQ too.

This is partially incorrect!

We should dismiss praise only if it’s from the same person, on the same day and if the praise is written verbatim, exactly the same.

1 Like

Great breakdown! An additional suggestion/potential discussion point. If we’re unsure, let’s use the #quantifier-support channel to ask/discuss. Better to get feedback and maybe consensus prior to the review call (which is usually pressed for time).

Also, personally I would rather err on the side of giving more credit vs dismissing (or quantifying separately vs marking duplicate). I would like to be part of a community that risks giving people more credit than they deserve (via a higher quant), rather than risk taking away credit from valuable contributions (via giving a duplicate/dismissal).