Measuring TEC’s ecosystem health

Measuring TEC’s ecosystem health

How to measure the health of a DAO ecosystem was a central question of the Governauts research group on reward systems. The teams focused on TEC Praise put a high emphasis on Ostrom’s Principles as an indicator of health in the first phase of the program. Inspired by their progress, soft gov working group has been mapping out Ostrom’s implementations in the TEC as well as possible metrics to track their success.

This work evolved into the idea of a dashboard to track ecosystem health when @sidcode and @mmurthy from Karma joined the conversation. The first step they suggested was for us to understand the data we have and what can be measured. @bends, @Vyvy-vi, @vegayp, @gideonro and myself have started to organize this miro board and this spreadsheet which are still work in progress.

The idea is create metrics for each one of ostrom’s principles, these metrics are based on the implementations we have for each principle. From principle 1 to 8 we’ll have a collection of metrics that can be analyzed individually and also grouped by principle to give a broader perspective of how that principle is doing in the TEC. Let’s say we have 10 metrics to evaluate “clearly define boundaries” and we identify in each one what threshold indicates a desired level. If one or more of those 10 metrics are below a desired level, we can understand that “clearly defined boundaries” is suboptimal and how that might be affecting the health of the whole system.

Metrics can’t be set in stone, but mapping them all out and having clarity of what is an optimal outcome and a suboptimal one will also help us understand which metrics aren’t working and what are the changes that need to be done along the way.

The main challenge is to identify our measurement apparatus. So, what can be measured and how? Also, how is this going to be displayed? Could it be part of RAD dashboard, or should we use Karma, or another option? @zhiwei, @0xNuggan, @mattyjee, @octopus, @ygg_anderson, @kristoferlund @akrtws I would love to have your advice here.

The benefits of having a complex analytics like this plus RAD (reward analysis dashborad) in the TEC could be:

  • Comprehensive feedback loop of our activities
  • granularity to identify problems and propose solutions
  • The system could become more independent and fluid (if core collaborators have to leave, there is a structure where others can understand what’s happening, plug in and address issues)
  • We would be the first Commons, as far as I know, to have analytics based on Ostrom’s principles
  • We could have scores to individuals based on these metrics, sum with the reward system score and create a wholesome reputation system.
  • We would have more insight on the health of individuals and the organization.

This article from TalentDAO shared by @AL0YSI0US is exploring similar things and I’m excited to try the questionary they developed as complimentary data input.

This is the list of Ostrom’s 8 principles and some of their implementation in the TEC

the metrics we are exploring are in the miro board that was also shared above

1. Clearly defined boundaries

  • TEC token governance

  • Access rights for communication platforms

  • Roles

  • Community covenant and code of conduct

  • Discord onboarding journey

  • Polycentric governance

  • Trusted Seed for economy initialization

  • Reward system

  • Token vesting period

2. Congruence between provision and appropriation

  • Entry and exit tribute

  • Value provided by TE proposals

  • TEC products

  • Working group services

  • Conviction Voting

  • Reward system analysis

  • Initial buy and LASERTAG

  • MVV and community covenant

  • TEC token liquidity

3. Collective Choice Arrangements

  • Do-ocracy

  • Individual and wg agency

  • Onboarding and education

  • Collaborative economics/ params dashboard

  • Polycentric governance framework

  • MVV

  • Sprint planning

  • Sprint retrospective

  • Praise quant

  • Sourcecred params

  • Inclusive culture

  • Templates

4. Monitoring

  • Sprint planning

  • Sprint retrospective

  • Advice process

  • Recording of meetings available on youtube

  • Proposals on forum

  • Public calendar

  • Github issues

  • Community and working group calls

  • Praise and praise quants

  • Reward system analysis

  • Transparency audits

5. Graduated sanctions (more work needed)

  • Celeste collateral tokens

  • Graduated bannings

  • 1 on 1 reach out

  • Warning plus admin power to delete harmful content

6. Conflict resolution mechanisms

  • Accessibility to get in touch with Gravitons at no cost

  • Gravity typeform

  • Graviton trainings

  • Graviton roles displayed on discord

  • TE lounge

  • NVC education

  • Psychology initiative

  • Community rituals

  • Celeste

7. Minimal recognition of rights to organize

  • Trusted Seed legal entity

  • Covenant

  • Legal working group

8. Nested enterprises

  • Sourcecred integration

  • Snapshot integration

  • Param parties in other communities

  • Tech build from 1hive

  • TE Academy partnership

  • Working groups becoming squad DAOs

  • Proposal inverter in partnership with PrimeDAO

  • Blockchain for good community

  • Commons Stack partnership

  • Block Science relationship

  • General Magic partnership

  • Longtail financial relationship

8 Likes

Thanks for the writeup @liviade
Looking at the spec and scope of this project I see it will require UX and UI design help.
There will be a lot of information and data shown and we should present it so it makes sense to the user balancing different use cases.

5 Likes

Thank you so much for putting this together @liviade.

It’s so nice to see the examples of the Ostrom’s principles. It looks, by itself, a product. Like a template of things DAO’s could have. I would love a visualization of that or a dashboard where you can point and click get data out of all this points.

For example, Do-Cracy: a person clicks on that example of Ostrom’s principles, and immediately it gets shown how it works in the TEC, how we are accomplishing it and the data that backs it up.

It would bring such a comprehensive way to understand the principles out of the conceptual phase. I would say we could, as a moving forward step (and I may be speaking nonsense, (since this probably has been agreed already) to identify who and which of this “examples” are easier to pick and build upon.

3 Likes

This is amazing work.

I see this work as eventually becoming an essential tool for the governance feedback loops of not just future commons but really any mission-driven and community-supported DAO. And I agree with @vegayp that this could be a product (software as service). With consulting services, there’s probably potential to generate some decent revenue streams too.

Once the TEC gets this base-level tracking in place, I think there’s also room to build impact evaluation systems on top of it with tools for creating, tracking, and managing “theories of change” and “logic models.” These can be super helpful for measuring progress in effecting change in the world, and it’s an area where we can actually learn a lot from the non-profit and philanthropic sectors.

2 Likes

Hi Marko! thanks for your feedback, yes that’s a good point. Would you like to stay in the loop for when that time comes? :slight_smile:

Yes, please :slight_smile: Thanks Livi.

2 Likes

Thanks for sharing! Really interesting application of Ostrom’s principles.

There are some parallels with the work we’re doing and it would be great to collaborate.

With RnDAO, we have started a research project on Community Health Analytics, and we’re primarily leveraging Organisational Network Analysis (ONA) as a tool to derive insights into a community.

ONA is a different approach than what’s being proposed here (less focused on custom metrics for each principle and more geared towards an overview of the network’s structure). So it would be really interesting to develop both views in parallel and then cross-check the data to explore whether a predictive model could be developed.

Additionally, we’re also building pulse survey capabilities into the tool. (As a methodology, pulse surveys increase the response rate, enable continuous data collection, and reduce the burden on contributors and community managers alike), and we’re doing a research project to triangular the research of TalentDAO in the topic with other sources (less McKinsey driven and more oriented towards decentralised communities, networks, and self-management). This last research piece could also be something we collaborate on and then with some financial support, we’d have the tool ready to load them and requests the community to participate.

We’re looking to apply for a few grants to get this going, so I’m wondering if the TEC could be one of the sponsors and a community for us to test the ONA and pulse survey tool.
That way, we can also explore the other bits (the research on the questions/metrics and the data crossing after both approaches are functioning).

How does that sound?

2 Likes