Measuring TEC’s ecosystem health

[liviade]

Measuring TEC’s ecosystem health

How to measure the health of a DAO ecosystem was a central question of the Governauts research group on reward systems. The teams focused on TEC Praise put a high emphasis on Ostrom’s Principles as an indicator of health in the first phase of the program. Inspired by their progress, soft gov working group has been mapping out Ostrom’s implementations in the TEC as well as possible metrics to track their success.

This work evolved into the idea of a dashboard to track ecosystem health when @sidcode and @mmurthy from Karma joined the conversation. The first step they suggested was for us to understand the data we have and what can be measured. @bends, @Vyvy-vi, @vegayp, @gideonro and myself have started to organize this miro board and this spreadsheet which are still work in progress.

The idea is create metrics for each one of ostrom’s principles, these metrics are based on the implementations we have for each principle. From principle 1 to 8 we’ll have a collection of metrics that can be analyzed individually and also grouped by principle to give a broader perspective of how that principle is doing in the TEC. Let’s say we have 10 metrics to evaluate “clearly define boundaries” and we identify in each one what threshold indicates a desired level. If one or more of those 10 metrics are below a desired level, we can understand that “clearly defined boundaries” is suboptimal and how that might be affecting the health of the whole system.

Metrics can’t be set in stone, but mapping them all out and having clarity of what is an optimal outcome and a suboptimal one will also help us understand which metrics aren’t working and what are the changes that need to be done along the way.

The main challenge is to identify our measurement apparatus. So, what can be measured and how? Also, how is this going to be displayed? Could it be part of RAD dashboard, or should we use Karma, or another option? @zhiwei, @0xNuggan, @mattyjee, @octopus, @ygg_anderson, @kristoferlund @akrtws I would love to have your advice here.

The benefits of having a complex analytics like this plus RAD (reward analysis dashborad) in the TEC could be:

• Comprehensive feedback loop of our activities
• granularity to identify problems and propose solutions
• The system could become more independent and fluid (if core collaborators have to leave, there is a structure where others can understand what’s happening, plug in and address issues)
• We would be the first Commons, as far as I know, to have analytics based on Ostrom’s principles
• We could have scores to individuals based on these metrics, sum with the reward system score and create a wholesome reputation system.
• We would have more insight on the health of individuals and the organization.

This article from TalentDAO shared by @AL0YSI0US is exploring similar things and I’m excited to try the questionary they developed as complimentary data input.

This is the list of Ostrom’s 8 principles and some of their implementation in the TEC

the metrics we are exploring are in the miro board that was also shared above

1. Clearly defined boundaries

• TEC token governance

• Access rights for communication platforms

• Roles

• Community covenant and code of conduct

• Discord onboarding journey

• Polycentric governance

• Trusted Seed for economy initialization

• Reward system

• Token vesting period

2. Congruence between provision and appropriation

• Entry and exit tribute

• Value provided by TE proposals

• TEC products

• Working group services

• Conviction Voting

• Reward system analysis

• Initial buy and LASERTAG

• MVV and community covenant

• TEC token liquidity

3. Collective Choice Arrangements

• Do-ocracy

• Individual and wg agency

• Onboarding and education

• Collaborative economics/ params dashboard

• Polycentric governance framework

• MVV

• Sprint planning

• Sprint retrospective

• Praise quant

• Sourcecred params

• Inclusive culture

• Templates

4. Monitoring

• Sprint planning

• Sprint retrospective

• Advice process

• Recording of meetings available on youtube

• Proposals on forum

• Public calendar

• Github issues

• Community and working group calls

• Praise and praise quants

• Reward system analysis

• Transparency audits

5. Graduated sanctions (more work needed)

• Celeste collateral tokens

• Graduated bannings

• 1 on 1 reach out

• Warning plus admin power to delete harmful content

6. Conflict resolution mechanisms

• Accessibility to get in touch with Gravitons at no cost

• Gravity typeform

• Graviton trainings

• Graviton roles displayed on discord

• TE lounge

• NVC education

• Psychology initiative

• Community rituals

• Celeste

7. Minimal recognition of rights to organize

• Trusted Seed legal entity

• Covenant

• Legal working group

8. Nested enterprises

• Sourcecred integration

• Snapshot integration

• Param parties in other communities

• Tech build from 1hive

• TE Academy partnership

• Working groups becoming squad DAOs

• Proposal inverter in partnership with PrimeDAO

• Blockchain for good community

• Commons Stack partnership

• Block Science relationship

• General Magic partnership

• Longtail financial relationship

[markop]

Thanks for the writeup @liviade
Looking at the spec and scope of this project I see it will require UX and UI design help.
There will be a lot of information and data shown and we should present it so it makes sense to the user balancing different use cases.

[vegayp]

Thank you so much for putting this together @liviade.

It’s so nice to see the examples of the Ostrom’s principles. It looks, by itself, a product. Like a template of things DAO’s could have. I would love a visualization of that or a dashboard where you can point and click get data out of all this points.

For example, Do-Cracy: a person clicks on that example of Ostrom’s principles, and immediately it gets shown how it works in the TEC, how we are accomplishing it and the data that backs it up.

It would bring such a comprehensive way to understand the principles out of the conceptual phase. I would say we could, as a moving forward step (and I may be speaking nonsense, (since this probably has been agreed already) to identify who and which of this “examples” are easier to pick and build upon.

[gideonro]

This is amazing work.

I see this work as eventually becoming an essential tool for the governance feedback loops of not just future commons but really any mission-driven and community-supported DAO. And I agree with @vegayp that this could be a product (software as service). With consulting services, there’s probably potential to generate some decent revenue streams too.

Once the TEC gets this base-level tracking in place, I think there’s also room to build impact evaluation systems on top of it with tools for creating, tracking, and managing “theories of change” and “logic models.” These can be super helpful for measuring progress in effecting change in the world, and it’s an area where we can actually learn a lot from the non-profit and philanthropic sectors.

[liviade]

Hi Marko! thanks for your feedback, yes that’s a good point. Would you like to stay in the loop for when that time comes?

[markop]

Yes, please Thanks Livi.

[danielo]

Thanks for sharing! Really interesting application of Ostrom’s principles.

There are some parallels with the work we’re doing and it would be great to collaborate.

With RnDAO, we have started a research project on Community Health Analytics, and we’re primarily leveraging Organisational Network Analysis (ONA) as a tool to derive insights into a community.

ONA is a different approach than what’s being proposed here (less focused on custom metrics for each principle and more geared towards an overview of the network’s structure). So it would be really interesting to develop both views in parallel and then cross-check the data to explore whether a predictive model could be developed.

Additionally, we’re also building pulse survey capabilities into the tool. (As a methodology, pulse surveys increase the response rate, enable continuous data collection, and reduce the burden on contributors and community managers alike), and we’re doing a research project to triangular the research of TalentDAO in the topic with other sources (less McKinsey driven and more oriented towards decentralised communities, networks, and self-management). This last research piece could also be something we collaborate on and then with some financial support, we’d have the tool ready to load them and requests the community to participate.

We’re looking to apply for a few grants to get this going, so I’m wondering if the TEC could be one of the sponsors and a community for us to test the ONA and pulse survey tool.
That way, we can also explore the other bits (the research on the questions/metrics and the data crossing after both approaches are functioning).

How does that sound?

[kristoferlund]

Great initiative @liviade! Adding to your ideas: I see our commons being comprised of four main domains:

• The community
• The common pool of funds
• The TEC token
• The subject matter - The emerging engineering discipline of “token engineering”

If any of these domains are unhealthy the commons will likely fail to contribute to its intended outcome - the furthering of token engineering as an vital and vibrant engineering discipline.

We should apply the lense of the Ostrom principles on each one of these domains separately as well as together.

To clarify what I mean by the four domains, let’s make an analogy. What if the TEC was a commons managing a forest?

• The community - represents the people doing the actual work of tending to the forest as well as other “stakeholders” - residents nearby, perhaps even visitors and so on.
• The common pool of funds - The money we use to support the forest - to buy seedlings, axes, to build bird watching towers and whatnot.
• The TEC token - A forest commons perhaps don’t have an actual token. But, this could be said to represent the mechanism by which the common pool of funds is replenished. Could be government money, fees from visitors, etc. Ideally not project money but some sort of stable long term stream of funding.
• The subject matter - The forest!

Looking at the forest example it becomes obvious that the first three domains need to be healthy to support the fourth. I have a sense that the TEC places way more emphasis on number one - TEC, the Community. The common pool, the TEC token and TEC, the Commons are less prioritised. Looking at each domain in isolation as well as together could more clearly highlight what parts of our commons that are healthy and what parts that need more love and attention.

Some metrics to look at, some questions we should attempt to answer:

• The Community
  • Is the community growing or shrinking?
  • What “churn rate” do we have for members, how long do they tend to stick around?
• The common pool of funds
  • Do we manage the funds responsibly?
  • How large percentage of the funds goes toward furthering the commons, how big percentage goes towards internal community work - the costs of running the community?
  • How does the burn rate look? How long will the funds last?
• The TEC token
  • Here we would like to present measures that attempt to answer the question: Have we succeeded in establishing this: Buying the TEC token is the premier way to support token engineering?
  • Is the trade volume increasing or decreasing?
  • How much does TEC trade contribute to the common pool?
• The subject matter - The emerging engineering discipline of “token engineering”
  • Is the engineering discipline of token engineering gaining traction?
  • Does the work we do contribute to that? How to measure and visualise this, I don’t know - this one is difficult.

[Letty]

Hey @kristoferlund! I find this reply very insightful and highly applicable to what Sampo WG proposal is currently doing, ie boosting financial health by “finding” TEC utility.

TEC utility is and will be pegged to the points you raise above, especially under “the TEC token” and “the subject matter”. So I’m tagging @gideonro here to make sure he has the chance to go over your contribution.

[AL0YSI0US]

Some time has passed since this post on Ecosystem Health was articulated and I’m curious to know how the resources in the image below, if at all, (screenshot from the miro board in this post) have influenced or informed the comprehension of Boundary work (technologically speaking and within the social layer) for the Token Engineering Commons ecosystem.