Praise Quantification Automation

đź’ˇ Ideas
1

After being involved in the praise quantification process, I think the process could be automated a bit… I’ve read the Rules of Praise and Quantification post and if we try to follow the formulation proposal on that post, we could definitely set specific words / phrases / binomials / trinomials assigned to points…

  1. We as quantifiers could meet first to define words / phrases / activities and pre-assign points to them i.e. Attending a meeting has a different point score than participating or hosting it. Additional meetings could be done to add / remove / modify words / activities and their points.

  2. I could develop a Python script that can read the log or Google Docs and based on that assign points… I could add NLP to that process to make it a bit more accurate and allow the script to add the score that determined the points assigned ( so that it does not become a black box assigning scores ).

  3. Our quantification meeting could be used to validate the script and get feedback and make it better and probably manually assign points on the praise that could not be assigned automatically by the script.

Is this something that we would be interested in doing? Is anybody else already working on something like this?

4 Likes
3

Hi @bustavo

I’m new to the TEC, but I’ve been working on the praise app, so I hope I can clarify the thinking around it.

Below is a diagram showing the current plan for the praise quantification system:

image
image780Ă—1281 88.8 KB

And a written overview: Outlining the Rewards System Process.. v2!

The idea is to split incoming praise into two buckets:

  1. Objectively quantifiable praise: praise for an explicitly / automatically identifiable activity, which receives a fixed quantification.
  2. Inter-subjectively quantifiable praise: praise for a subjectively identifiable activity that receives an inter-subjective quantification.

For 1 we’ll be using SourceCred, and for 2 we’ll be using the praise app.

Just a note that if you were a quantifier this week or last you would have been reviewing the praise from way back – in July I think? There was likely a lot of praise in there which in the future will fall under bucket 1, and be automatically quantified by SourceCred.

I think what you’re suggesting is a third bucket: praise which is subjectively identified but objectively quantified. And with NLP we could analyse praise instances and assign objective quantification based on the words / phrases used.

I appreciate the goal of trying to reduce the overhead of the quantification process.

I guess I’d I have a few concerns:

  • Risk of gaming: The praise giver can tune their language to trigger a desired objective quantification. Effectively the praise giver becomes the quantifier.
  • Flattening impact: It will cause objective quantification of praise which may contain similar language, but have very different impact, or different subjectively understood impact.

In general though, I’m not sure how to think about the contrast between systems of objective quantification (where the value of different contributions is defined via governance), versus inter-subjective quantification (where the value of different contributions is defined by some composite of subjective peer quantification).

1 Like
4

Hello @mattyjee ! Thanks for the very nice and clear diagrams and explanation.

Regarding the 2 concerns, I’d like to add to the discussion:

Risk of gaming… isn’t that a risk already present now with human quantification? We could use the data sets being generated manually to try to train or do baseline comparisons with what the nlp determines ( I know it’s not that much data ). We could also run scenarios where we generate “tuned” praise and see if we can make the nlp find it and flag it ( could be an interesting experiment ).

Flattening impact: I think we can add rules to consider this and flag things that seem very similar or that are new ( compared with past praise ) and those elements could be manually evaluated and be part of the learning process ( with much less time spent on that ). I assume that as time goes by, praise will be changing according to new activities, events, etc.

I am also not sure on the contrast between systems of objective quantification vs inter-subjective quantification.

I’d love to continue the discussion :slight_smile: I am not tied to making this idea happen so if we decide it is something that would be a waste of time, I would totally understand.

1 Like