V2 - Should Gnosis DAO fund the continuation of the validator Monitoring Telegram bot?

[nicosampler]

GIP-Draft: Should Gnosis DAO fund the continuation of the validator Monitoring Telegram bot?

• In Favour
• Against

GIP: <to be assigned>
title: Should Gnosis DAO fund the continuation of the validator Monitoring Telegram bot?
author: Nicolás Domínguez (nicolas@bootnode.dev), Manuel García (manu@bootnode.dev)
status: Draft
type: Funding
created: 2024-04-13
duration: 12 months
funding: 50,000 USDC + 250 GNO (exclusively dedicated to running validators for at least 1 year).

TL;DR

This is a proposal by bootnode.dev to continue the conversation about the Gnosis Chain validator monitoring bot that we developed earlier this year.

Introduction

The Telegram bot functions as a monitoring tool, notifying users of any issues and providing continuous updates on validators’ performance.

Featuring

• Setup: requires just withdrawal addresses, and retrieves all their validators.
• Dashboard: gives all important info in a small message automatically updated in real-time.
• Stats Updates
• Automated Notifications: for performance < 90%, status change (active/inactive), slashing
• Subsidized Claim

Link to set up the bot: Telegram: Contact @gbc_validators_bot

We have shipped the first version. KPIs (as of April 9, 2024):

• Users: 67
• Total registered validators: 56056 (~ 30% of the total Gnosis Chain 199512 active validators)
• Validators under active monitoring: 6038 (we can only monitor 200 validators per user due to resource constraints)
• Claimed Rewards: 100+ subsidized claims through our bot using a very convenient ‘/claim’ Telegram command (https://gnosisscan.io/txs?a=0x9683693f8164b4f8c6d804e690ac2c26f8fe56ed)

We are seeking funding to invest in building improvements. Additionally, we are applying for retroactive funding as significant work and infrastructure resources have already been invested in the solution that benefited the Gnosis community.

Motivation

We were not quite satisfied with the existing validator monitoring solutions for the following reasons.

1. Gnosischa.in Free plan only allows monitoring up to 100 validators per account.
2. Gnosischa.in Max paid plan (21 euro) allows monitoring up to 280 validators per account, and we (along with other community members) have more.
3. Existing onboarding experience, including registering validators and configuring monitoring, was too complex and may deter less sophisticated users from trying.
4. We wanted a simple dashboard with the most essential parameters, relevant notifications, and a simple way to claim rewards.
5. We wanted to receive notifications per set of validators instead of individual validator identifiers.
6. We didn’t want to use a native app or email. We wanted to start with a straightforward Telegram bot.
7. We wished we had this for us, so we built the existing Prototype.
8. We want to Open Source it.

Current monitoring tools

beaconcha.in is an excellent tool that works well for monitoring validators on the Ethereum chain. However, for Gnosis Chain, this tool is less optimal than for Ethereum. For instance, missing one attestation in Gnosis Chain, where blocks occur much faster than in Ethereum, may be less critical. We notify the operator if the validator only misses 4 or 5 attestations in a row (a feature validated with our beta testers). Another example is the absence of a method to load all validators from a withdrawal key. This limitation arises because, generally, users only have a few validators on Ethereum, which is NOT the case for Gnosis Chain.

Lastly, the most significant issue that Gnosis Chain validators face is the price of monitoring tools such as gnosischa.in service. For validating more than 100 validators, up to 280, the price is 21€ per month. This pricing structure appears unreasonable as it costs approximately 4% to 10% of a validator’s revenue to monitor 100 to 280 validators at USD300/GNO. Not even considering the operational expenses (infra, devops, bandwidth, etc).

Our solution

A Telegram bot that functions as a monitoring tool, notifying the user of any issue and providing continuous updates on the performance of their validators.

Here is the X Post announcing its beta access launch.

We have been testing the bot over the past month, and it has performed way better than expected. The bot currently utilizes the beaconcha.in API. It periodically polls requests for all validators added by users and delivers relevant statistics.

You can already test the bot: Telegram: Contact @gbc_validators_bot. The following is a screenshot of the stats provided by the bot a few days ago.

Features

Users/Operators will:

• Connect the Telegram bot and execute the command /load_validators followed by a withdrawal address. This command can be invoked multiple times if the user has more than one withdrawal address.
• Receive notifications when a validator misses several attestations in a row, or their overall performance falls below 90%.
• Get updates on validator status changes such as being down, up, slashed, or exited.
• Convenient Dashboard that showcases Total balance, Rewards stats: 1d, 7d, and 31d, Claimable rewards, as shown above.
• Claim rewards directly from TG (subsidized).

Roadmap

After testing the bot with real validators, we received valuable feedback from the support channel Telegram: Contact @GBC_validators_bot_support. Consequently, we plan to add and improve features and increase infrastructure resources to open the bot for all Gnosis Chain validators (~200k).

For this reason, we have devised a plan to make this bot free and accessible to everyone. Below is the Roadmap for how we will make it happen:

• Milestone 0: Already delivered

  • The bot has already been deployed and is open for anyone to join. The only temporary limitation is the number of validators that can be loaded per Telegram account (200).

• Milestone 1: In-progress

  • The bot has been running in a testing phase for the last two months. During this time, we have collected the following feedback from different users that we plan to address:
    • Reduce alerting frequency when something goes wrong with a validator.
    • Enable tracking of xDai rewards.
    • Calculate effective APY.
    • Improve performance.
    • Improve UX for interactions such as loading validators, creating a command to check if new validators have been added for an account, specifying AM or PM for some dates, and others.
  • Open Source it.

  Timeline: 1 month.

• Milestone 2: not-started

  • API adjustments: We will adjust the number of validators users can track while avoiding exceeding the monthly request quota.
  • Validators onboarding: Step-by-step instructions, documentation, generate a bot ”brand” identity including website with basic information about the bot’s purpose and how to interact with it.
  • Notifications (under evaluation): allow the community manager (e.g.: https://twitter.com/0xarmagan) to push notifications through the bot, like important upcoming node upgrades, links to the weekly report, etc.
  • Historical rewards per month. We can save them to the DB to facilitate tax returns.
  • Make the source code open source.

  Timeline: 1 month.

• Milestone 3: not-started
  Our current API provider might not be the best solution due to its cost and restrictions, preventing us from scaling and minimizing the cost of monitoring/validator. We are exploring different options to offer an unrestricted service to all Gnosis validators and to provide real-time notifications. We’ve discussed this with community members like @dapplion, who recommended looking into the Dora project (a simplified version of a fork of beaconcha.in). After a thorough review, we found Dora to be a good starting point but only partially suited to our needs. To gather all the information our bot needs, we’re considering adding specific features from the beaconcha.in codebase. Other ideas involve hosting gnosischa.in Open Source stack in our infrastructure or finding an infrastructure partner (eg, those running RPCs/Nodes for Gnosis Chain). This topic requires more research.
  The result of this research will be our team’s analysis, outlining each strategy’s advantages and disadvantages. We plan to develop a proposal for the DAO aimed at securing a long-term commitment that will enable us to keep the bot open and unlimited for the entire Gnosis chain validators community.

  Timeline: 1 month.

Note:

We are aware that Pectra is optimistically aiming to arrive at the end of the year. EIP-7251 will allow stakers to consolidate way more than 1 GNO into each index, up to 64 GNO. It will allow small stakers to reduce costs and make the network participation rates higher and more stable. We will have a better assessment much earlier than that on what the best solution would be from our point of view.

Benefits for the community/DAO

1. Reduces monitoring costs for all Gnosis Chain validators, increasing performance and profitability.
2. Simplifies validator monitoring setup, improving operator satisfaction.
3. Built by an active validator, hence providing the most relevant features, stats, and user experience.
4. It will be continuously supported by BootNode going forward.
5. It is a fundamental tool for anyone running validators, and it is free to use. One of the goals of the Gnosis Chain is to be as decentralized as possible, and that won’t be possible without the appropriate tooling and low expenses.

Funding

• Retroactive

  • Prototype Research and Development: a team has dedicated several weeks to perform the required R&D to spec out and build the initial version.
  • The initial version has already been used by 67 operators, even though there is a cap of 200 validators per account, which proves its relevance and need.
  • Official Node Management documentation was updated to include bot setup instructions (Monitoring Validators | Gnosis Chain)
  • Infrastructure (USDC 1288.72):
    • Digital Ocean Services.
    • Beaconcha.in API (Diamond Plan: 399 euros/month. 3 months so far): 1288.72 USDC (1,197 EURe swap to USDC - CoW Swap | The smartest way to trade cryptocurrencies)

• Future Development

  • Detailed in milestones 1 to 3.

• Operation and Maintenance: 6 months, including the following costs and tasks:

  1. Gnosis Beacon Chain API plan.
  2. Infrastructure costs, monitoring, and operation.
  3. Support to validators by our team in the bot Support Telegram channel.
  4. Bug fixing, development, and maintenance.
  5. Pull Requests review and merge pipeline.

Funds requested:

• USDC: 50,000 that already includes retroactive funding for the delivered prototype
• GNO: 250 (exclusively dedicated to running validators for at least 1 year).

Progress status

• Monthly progress reports in the forum.
• Participation in the community calls.
• Frequent posts in X.

Gnosis Snapshot

Phase 2 Proposals: Please ignore this section, and leave as is. It is used for Phase 3 proposals.
Phase 3 Proposals: Add a link to the corresponding Gnosis Snapshot poll you’ve created.

[diegodefi]

Thank you very much for submitting a new proposal, as I was one of those who voted against.
Because my English is very poor, I am consulting about this in the Gnosis in Spanish Telegram Group (Telegram: Contact @gnosisespanol) with people from the team to clear up my doubts.

[diegodefi]

My biggest doubt is what happens and what the 250 GNO are needed for, how do those tokens help the operation of the Bot with that money?

I also don’t understand the infrastructure expenses indicated to run the BOT.

They are asking for 50k USDC, if the proposal were accepted, I wouldn’t give them all at once, it would be good to see the results first and especially the number of people who use it.

Sorry if my English is wrong, but I used ChatGPT

[mgarciap]

@diegodefi

They are asking for 50k USDC, if the proposal were accepted, I wouldn’t give them all at once, it would be good to see the results first and especially the number of people who use it.

Hi Diego, Thanks for bringing your concern from the Spanish TG channel to the forum so that more people can contribute.

50k given at once: I guess the DAO can determine the best way to deliver the stables funding (from $4.15K/month or all of it together).

As for “see the results first,” please read the proposal part We have shipped the first version. KPIs (as of April 9, 2024):. Feel free to join the support channel and ask real users about its benefits, needs, expected features, etc.

@diegodefi

My biggest doubt is what happens and what the 250 GNO are needed for, how do those tokens help the operation of the Bot with that money?

The monitoring infrastructure, backend, and bot does not require GNO. The telgram bot could be a webpage, Discord bot, API, etc. What I mean is that this development is not just a bot. The bot is the UI.

So, what are the GNOs for then? To create long-term incentives. The team has exchange a part of the compensation for a stake in the project.

Compared to paying a fixed amount of money, this approach incentivizes teams to work as hard as possible, adding value and positively impacting GNO holders. The teams will have skin in the game. Also, as they will be able to realize any gain in the future (who knows what the market will look like in a year), they also run a risk. Therefore, they share the risks and the rewards, as most of protocols and DAOs alignment mechanisms work.

On the other hand, many teams are just incentivized with locked GNO for a year or so (read previous GIPs) and that’s all. We commit to work adding value running validators to increase network decentralization and security. Of course, it provides rewards, but also additional fixed costs (infra + devops+ monitoring + ops&fin).

@diegodefi

I also don’t understand the infrastructure expenses indicated for running the BOT.

The infra cost includes Digital Ocean VMs and other services as well as a Beacon Chain top-layer API (at least for a few months until the team, along with some other community members who shared interest and ideas, finds a way to reduce these costs). Of course, you are welcome to provide improvement ideas.

I hope I was able to answer your questions.

[mfw78]

1. I’m not sure if this is reducing monitoring costs for all Gnosis Chain validators, given that there are those out there that are able to adequately monitor their performance without this, yet by tapping the GNO treasury, this actively causes increased costs (via subsidy) to those that are monitoring their nodes, by subsidising those that are not.
2. I’d generally disagree with any payout of funds up-front in lieu of streams. There needs to be value at risk to ensure continuation with development (if this were to pass).
3. There seems to be no outline on how costs will be contained and/or reduced.

Overall, this seems very costly, with seemingly little to no potential for return on investment for the DAO.

[filoozom]

I’m generally against this, as the costs seem very high in comparison to other tools for the use cases it covers. For example, https://gnosischa.in/ is asking for less than double what you’re asking for, while providing many more features and requiring much more computing power.

Your own KPIs mention that you have 67 users for 56 056 registered validators. This means that in average, each of your users owns 836 GNO, so more than $250k at current prices. We also ran a survey stating that in average, node operators have around 20 validators, which is perfectly monitorable with existing tools. I don’t think that the DAO should fund tools for big validators, who should be able to pay for the services themselves in case they require it.

While I agree that tooling might be lacking for larger providers, I would also argue that the most useful part of this tool is downtime notifications, which can already be monitored for a huge number of validators, as operators always run many keys on a single client / server, and it is thus generally sufficient to monitor one key per server rather than each of them individually.

A few more points:

• As you mentioned, Pectra is going to alleviate these issues almost entirely, albeit only in around one year
• Free automatic claims are already available with smart contract calls today, and will soon be configurable directly on https://deposit.gnosischain.com/
• Researching how this tool can be optimized for an entire month sounds like a lot of expenses to save 399€ / month for up to one year (until Pectra)

I could be convinced in case smaller node operators describe relevant use cases in this thread, but as it stands the costs are way too high in my opinion.

[armog]

Thanks for this proposal. The bot is useful and convenient. Thanks you for valuable contributions to the Bootnode team.

There are some questions I’ve been wondering. I think this project will be useful in a within period of time and depending on technical developments, but with Pectra planned to be implemented at the end of the year, the situation will change considerably and I am not sure whether the product will be needed.

Firstly, I would like to do some mental gymnastics for the target audience and users.

How do we determine that the users of this product are unique? Let’s say I am not a validator and I choose a random address from the deposit leaderboard list and use the product, will I be counted as a user? So when I do this, is it added to the number of users and monitored validators? In this case, if I arbitrarily add all addresses 200 validators and below to this list, would it be correct to say that the product is in high demand?

You have determined the target audience of this bot as the target audience that finds it costly to monitor 100-280 validators. Let’s examine who they are.

• 62 addresses in the Gnosis Beacon Chain with validator operations above 500 validators the majority are teams and mid-size initiatives.
• 21 addresses in the Gnosis Beacon Chain with validator operations around 500-280 validators. The majority of them are individuals.
• There are a total of 120 addresses between 280-100 validators. And most of this group is composed of individuals.

62 addresses consisting of professional teams and dapps (Stakewise, Chorus One, Consenys, Gateway), DAOs (Kleros, Aavechan, Karpatkey) and curated node operators (SenseiNode, Twinstake). We know that they mostly run their operations with beaconchain. Because of the many features it offers and it is a battle tested product. I think this group of people would prefer not to work with a telegram bot. Maybe prefer but they need a strong reason to change their current preferences

We see that there are approximately 141 addresses, less than 500 and more than 100. For 100 validator and under, beaconchain’s offer is 5 euro, which is quite reasonable.

In this case, your target audience is currently about 141 addresses. Is the target audience of the fund you request for 141 addresses? How do you aim to reach them?

One issue that catches my attention here are the commitments in Milestone 3. Obviously API diversity is important and should be supported. Is this proposal requested here a fund to develop API for beacon chain? or a retroactive payment for bot service and extra GNO? I think there is a confusion here. Which idea will the DAO vote on?

It is a very good idea to contribute to the decentralisation of Gnosis Chain, but I believe you need to provide more details about this proposal especially for using GNOs.

In the first proposal, 400 GNO was requested, now I see that there are 250. What is the reason for this change? What was the rationale for requesting 400 GNOs and why are you requesting 250 GNOs now?

You are requesting 250 GNO to run a validator for 1 year. Will it be returned to DAO after this period or will it be continued or sold at your choice after 1 year?

Thanks!

• edited some typos

[butta]

Hi, butta here from beaconcha.in!

As a general note, we have a webhook feature that you can use to forward notifications to Telegram. (Login - Open Source Gnosis (GNO) Mainnet Explorer - beaconcha.in - 2024)

Gnosischa.in Max paid plan (21 euro) allows monitoring up to 280 validators per account, and we (along with other community members) have more.

We are working on a v2 which will support up to 250,000 validators (stay tuned ). Note that we do not have a limit for notifications validators in v1. Theoretically, you could add hundreds of validators to your notifications today.

We notify the operator if the validator only misses 4 or 5 attestations in a row (a feature validated with our beta testers).

We already support this – Simply use the “validator offline” notification type.

The bot currently utilizes the beaconcha.in API

Thank you for using our API! I’m concerned that you might encounter API limits as more users, especially those with hundreds of validators, start using the bot. While I appreciate your use of our API, I find it difficult to justify the grant amount. I would prefer to see you develop an independent tool that does not rely on third-party APIs.

[mrtdlgc]

Even though I liked the bot and have heard pretty good feedback from node runners to monitor their nodes. I have to echo both armog and butta here a bit.

From what I know, this is a tool that relies on beaconchain team’s APIs.

What is to stop the node runners to crowdsource payment for beaconchain’s APIs and rely on this bot? Or what is to stop people to come up with their own version of the bot as I know there are a few node runners who have developed their own tools for monitoring their node health as well.

I have to also admit that these tools are necessary for the general node running/staker community. However, I have never been a proponent of SaaS in crypto and see these kinds of software as public goods. And there are better ways to fund these initiatives than a GIP from the Gnosis DAO.

I have to also acknowledge the fact that the current state of affairs in Gnosis DAO’s governance is not well-fitted and a bit obsolete in terms of funding small initiatives like this. And I would have also hate to see such a great initiative to help out node runners to go to waste due to lack of funding.

I wish to believe that with GIP-99 could improve the way in terms of encouraging relatively small initiatives like this in this proposal to be properly evaluated and funded though.

Hence, considering the discussion under GIP-99 and this proposal, I would like to direct @auryn_macmillan and the Guild’s attention to consider this proposal as part of immediate deliverables expected from the Guild’s proposal to brainstorm a solution to find a proper way to fund this incredible initiative (let it be through RPGF, conviction voting, or quadratic funding, in any novel way possible) which has already proven immense assistance to our current node runners who have already expressed their appreciation of the tool (check validators channel on Discord for testimonies).

[nicosampler]

Hello frens, thanks for your feedback! It seems the focus has shifted primarily to the funding aspect, rather than the significant benefits and improved user experience our project offers to node operators for monitoring their validators. As @mrtdlgc points out, I invite you to look at the positive feedback we have received.

@armog, I must admit I was quite surprised by your comment. Considering your engagement on Discord, you seem to have missed the many community validators who have shared positive feedback about the bot. I don’t see a reason why there would be an intention to manipulate the metrics as suggested. You’ve previously recognized the bot during beta testing as a beneficial tool for validators and mentioned it twice on the weekly Gnosis announcements. Additionally, it’s important to note that we are developing a tool based on a service that receives substantial annual funding from the DAO. Most importantly, I see this not merely as a cost to the DAO but as a valuable resource for all validators, not just those managing large numbers. cc: @mfw78, @filoozom.

@butta, that’s fantastic news, and I am eager to try out the promising v2. We understand the features you already support, and we believe we’ve found a way to simplify the data summary and enhance the UX, which many users find helpful. Offering an alternative is something we need to evaluate thoughtfully. Fortunately, you’re here giving feedback, and perhaps you could support our initiative by providing us with an API at a significant discount, or even better, for free. This would allow us to substantially reduce the costs requested from the DAO and help us evolve the bot into a public good.

I would like to invite you all to provide feedback on whether the bot is beneficial for the Gnosis community validators and whether you would like to see it become a public good or prefer it to be sunsetted. If we agree that it is beneficial for the community, we will happily revisit the requested amount and find a sum that makes sense for all of us.

[armog]

@nicosampler I am in favor of using the bot and I’ll will continue to protomote it within the community. All comments are on your fund request and the product you want to develop. I think the product will be better and more functional with comments and criticism.

We have a validator meeting on April 18. I think this is a good time to explain the features of the bot. I invited Manu last week. Hope you will join as well.

[diegodefi]

After the passing year, would it be necessary to finance again?

@nicosampler

Hello frens, thanks for your feedback! It seems the focus has shifted primarily to the funding aspect

But financing is precisely what they are asking for, that’s why that topic is being discussed.

[mgarciap]

@mrtdlgc

I wish to believe that with GIP-99 could improve the way in terms of encouraging relatively small initiatives like this in this proposal to be properly evaluated and funded though.

Thank you for pointing GIP-99 up.
Its commercial terms could be ~USDC 1.5m and ~5k GNO a year (not easy to estimate). The GNO is locked for 1 year. It doesn’t say if it will provide services to the network. It has only a few comments and votes (9). It is a high-impact proposal. I propose that everyone reading this comment check it out. There is an interesting comment from the Gnosis CEO.

Gnosis Guild is a great team. This proposal’s team should definitively chat with them (@auryn_macmillan)

[CryptosNF]

Thank you for this new proposal with lower amounts, and these interesting discussions.

But I think you’re missing the point of the questions you’re being asked about financing.

(You say we’re not talking about the product, but people who are very present at Gnosis say that your product is interesting, important and useful, so it’s not really a question of whether your product is what it really is.)

The subject is financing, and only that

And you haven’t answered these questions?

What are you going to do with the 250 GNO after 1 lock in stacking? Are you going to return them to the DAO? Are you going to sell them on the market? You’re going to keep running validators ?

Because the argument of decentralizing the gnosis ecosystem via these validators can be heard, but if it’s just to sell them in 1 year’s time, I don’t think this argument should be used.