Should GnosisDAO support ProbeLab to produce network performance and security metrics to ensure the Gnosis Chain P2P network is performant and resilient?

GIP: <to be assigned>
title: Should GnosisDAO support ProbeLab to produce network performance and security metrics to ensure the Gnosis Chain P2P network is performant and resilient?
author: Yiannis Psaras (GH: @yiannisbot), Dennis Trautwein (GH: @dennis-tra), Mikel Cortes (GH: @cortze)
status: Draft
type: Funding
created: 2025-09-26
duration: 3 months from project kick-off date
funding: $105k

Executive Summary

This project focuses on enhancing the stability, resilience and performance of the Gnosis Chain P2P layer network through an enhanced set of metrics and the corresponding dashboards. In particular, the project focuses on four primary areas of improvement, each of which is later composed into a milestone with its own objective and timeline. In summary, these four areas are:

  1. Metrics for the Execution Layer P2P Network. We will develop a detailed set of metrics for the Execution Layer DHT network (discv4), similar to the ones produced for discv5, see: https://probelab.io/gnosis/dht/2025-34/ for Week 34, 2025.
  2. Protocols that Consensus clients are using. We will extend the Nebula crawler to capture “Status and Metadata Exchange” and provide more visibility into the protocols that Consensus Client nodes are running.
  3. Actual size of the Gnosis network. We will adapt our Ants architecture to support the discv5 DHT and thus Gnosis peers. This will allow us to detect nodes that are not part of the core of the network but instead, e.g., reside behind NATs.
  4. Detect Eclipse Attacks. We will adapt our tooling for the Gnosis Chain discv5 DHT, deploy the tool to monitor the network 24/7 and alert the Gnosis engineering teams when density goes above a certain level.

After more than 3 years in operation with no downtime for our 24/7 monitoring infrastructure, ProbeLab is trusted by top firms to secure their networks and guarantee top performance at all times.

Specification

Details of the specification for each milestone are given in each of the Milestones.

Rationale

We provide the rationale for the work proposed here in terms of impact that will be created for each of the four areas that we outlined in the Executive Summary.

  • Produce Metrics for the Execution Layer P2P Network. Develop a detailed set of metrics for the Execution Layer DHT network (discv4), similar to the ones produced for discv5.
    • Why this is important: There is currently no information regarding the size, structure, or geo-distribution of the Gnosis Chain Execution Layer network. With this work we will extend and enhance the Nebula functionality to track EL (discv4) DHT nodes and produce metrics similar to those presented at probelab . io for the Consensus Layer.
  • Identify all protocols that Consensus clients are using. Extend Nebula crawler to capture “Status and Metadata Exchange” and provide more visibility into the protocols that Consensus Client nodes are running.
    • Why this is important: Through the “Status and Metadata Exchange”, we will be able to find out details regarding the following performance- and security-critical items:
      • Distribution of nodes up to date with the canonical chain and the nodes that are still syncing it
      • Attestation Subnets overlap
      • Sync Committee Subnets overlap
      • Monitoring how many tools (or attackers) are there not able to help syncing
      • Identify nodes with open publicly Beacon-API ports
  • Identify the actual size of the Gnosis network. Adapt our Ants architecture to support the discv5 DHT and thus Gnosis peers. This will allow us to detect nodes that are not part of the core of the network but instead, e.g., reside behind NATs.
    • Why this is important: Today, we do not have an accurate view of the size of the Gnosis discv5 DHT network. it is not clear how many DHT client nodes interact with the discv5 DHT, how many peers are behind a NAT, or the geo-distribution of those nodes. Among other things, this makes it difficult to have a view of the structure and geo-distribution of the network, as well as get accurate numbers on the energy footprint of the Gnosis network. Together with the existing discv5 reports/data, this will give a comprehensive view of the network.
  • Build tooling to detect Eclipse Attacks. ProbeLab has built a sophisticated tool to monitor how many peers reside at particular spots around the DHT key space. Higher than normal density of peers can reveal that there is a potential eclipse attack going on in the DHT network, affecting the performance of the overall system. We will adapt our tooling for the Gnosis Chain discv5 DHT, deploy the tool to monitor the network 24/7 and alert the Gnosis engineering teams when density goes above a certain level.
    • Why this is important: Eclipsing peers in the DHT network can be catastrophic for individual validators. Although this tool does not mitigate the attack, it is very effective in identifying it.

Who benefits and how: Having an accurate view of the structure and performance of the P2P network benefits the community at large, given that detailed metrics and relevant alerts increase confidence in correct operation of the network. At the same time, and perhaps more importantly, validators, client teams, infrastructure providers and core protocol developer teams will benefit the most from the outcome of this project. Network, infrastructure and protocol upgrades won’t fly blind, but instead will have an accurate view of the behaviour of the network and will be able to act accordingly.

Budget

The breakdown of the costs per milestone are given below. A more detailed breakdown of expenses is as follows: 60% of the budget will go towards development, 20% towards software and infrastructure maintenance and 20% towards infrastructure cost.

  • Milestone 1: $35k
  • Milestone 2: $20k
  • Milestone 3: $30k
  • Milestone 4: $20k
  • Total Cost: $105k

Milestones:

Proposed timeline: 3 months from project kick off date
Milestone dependencies: There is no dependency between any of the Milestones and therefore work can be carried out in parallel.
Final deliverable: Enhanced metrics for the Gnosis Chain DHT network at the Consensus and Execution Layers at probelab . io. See each milestone for individual deliverables.
Dashboard Access: Public (at probelab . io)
Data Access: Available at a fee.


Milestone 1: Produce Metrics for the Execution Layer P2P Network
Deliverables: Weekly Network Health reports for the Execution Layer
Time and Price Estimate: 1 month from project kick off, $35k
Dependencies: None.

Detailed Description: This Milestone comprises hardening the Nebula discv4 DHT (Execution Layer) support for Gnosis, as well as creating weekly reports for the Execution Layer, similar to discv5 DHT (for the Consensus Layer). While the general discv4 logic support in Nebula exists already, the execution/crawling stops for unknown reasons. This leads to incomplete data which in turn may lead to wrong conclusions when analysing the data.

Notes:

  • Raw data can be made available to the Gnosis team, or wider community, at a separate fee.
  • This milestone includes maintaining tooling and producing results for 1 year.

Scope: New development/extension of existing tooling to support Gnosis. Maintenance of infrastructure, tooling and software to produce results for 1 year.

Access: All results and reports, as well as the tooling (i.e., Nebula) will be public and available to the community. Raw data can be made available through ProbeLab’s API at a separate fee.


Milestone 2: Identify all protocols that Consensus clients are using
Deliverables: Enhance current metrics on probelab . io with more information on the protocols that Consensus clients are running
Time and Price Estimate: 1.5 months from project kick off, $20k
Dependencies: None. Although this Milestone involves the Nebula crawler, similar to Milestone 1, it targets a different layer (the Consensus layer, as opposed to the Execution Layer in Milestone 1).

Detailed Description: Currently, Nebula tries to establish a libp2p connection to every node in the discv5 network. However, it doesn’t make use of that connection yet beyond waiting for the libp2p identify exchange to complete. The identify exchange gives us information about the agent version, supported protocols and more. However, nodes running Consensus Client software like lighthouse and teku, support additional protocols which are usually exercised after a connection has been established and comprise an additional handshake. These additional protocols are the Metadata and Status exchanges. The data that’s exchanged there is the following:

message MetaDataV3 {
  uint64 seq_number = 1;
  bytes attnets = 2;
  bytes syncnets = 3;
  uint64 custody_group_count = 4;
}

message Status {
  bytes fork_digest = 1;
  bytes finalized_root = 2;
  uint64 finalized_epoch = 3;
  bytes head_root = 4;
  uint64 head_slot = 5;
  uint64 earliest_available_slot = 6; // part of StatusV2 - available soon
}

This milestone comprises extending Nebula to perform these requests and store all of the above information alongside the existing ones per visit row. Some of the data overlaps with the ones already available in the ENRs, however, the ENRs are comparably inert when it comes to providing accurate timely information.

As mentioned further up, there are critical insights that can be drawn from this data:

  • Distribution of nodes up to date with the canonical chain and the nodes that are still syncing it
  • Attestation Subnets overlap
  • Sync Committee Subnets overlap
  • Monitoring how many tools (or attackers) are there not able to help syncing
  • Identify nodes with open publicly Beacon-API ports

By requesting the subnet subscription that each node shares over the requests/responses, we will be able to assess the network’s coverage of each independent subnet. This could help visualize any over- or under-populated subnets, preventing edge cases where validators can struggle to find enough peers to publish attestations to.

Scope: New development.

Access: Open source.


Milestone 3: Identify the actual size of the Gnosis network
Deliverables: Enhance current metrics on probelab . io with an accurate number for the Gnosis Chain network size.
Time and Price Estimate: 2 months from project kick off, $30k
Dependencies: None. This Milestone does not involve the Nebula crawler and therefore does not depend on the progress/completion of the previous Milestones.

Detailed Description: This milestone focuses on adapting our Ants architecture to support the discv5 DHT and thus, Gnosis DHT client peers, or peers behind NATs. This will allow us to detect nodes that are not part of the core of the network and are therefore not visible by simply crawling the DHT network. The service is able to log the activity of all peers (also NAT’d peers) in a DHT network by carefully placing ants in the DHT keyspace.

The ants methodology achieves its goal because when nodes utilize the DHT they perform routing table maintenance tasks. These tasks consist of sending requests to several other nodes close to themselves in the DHT keyspace. The ants service ensures that at least one of these requests will always hit one of the deployed ants. When a request hits an ant, we record all available information in that exchange. For libp2p-based networks, for instance, this consists of things like agent version, supported protocols, IP addresses, and more.

This Milestone includes the infrastructure cost required to run the ants toolset and produce results for 1 year.

Scope: New development/extension of existing tooling to support Gnosis. Maintenance of infrastructure, tooling and software to produce results for 1 year.

Access: All results and reports, as well as the tooling will be public and available to the community. Raw data can be made available through ProbeLab’s API at a separate fee.


Milestone 4: Build tooling to detect Eclipse Attacks
Deliverables: Plot and alerting system to detect eclipse attacks on the Gnosis Chain discv5 DHT network
Time and Price Estimate: 2.5 months from project kick off, $20k
Dependencies: None. This Milestone involves a different tool to the ones used in previous Milestones and therefore does not depend on the progress/completion of those.

Detailed Description: Monitoring of the DHT keyspace density can reveal potential security vulnerabilities in the network. The density of the keyspace is not expected to be completely uniform, but large variations can reveal malicious behaviour, i.e., when a large number of nodes join the network with forged PeerIDs close to the node(s) that they want to eclipse and basically eclipsing the node. The attack can be executed in minutes from a laptop and has been demonstrated recently for the IPFS DHT.

Note that:

  • although this vulnerability is not expected to bring down the network as a whole, it can remain unnoticed and have an impact on particular nodes.
  • the methodology, tool and plot does not include a mitigation, or defence against an attack. It is a detection tool.
  • this Milestone includes the infrastructure cost required to run the Eclipse Attack detection tool and produce results and alerts for 1 year.

Scope: New development. Maintenance of infrastructure, tooling and software to produce results for 1 year.

Access: All results and reports, as well as the tooling will be public and available to the community. Raw data can be made available through ProbeLab’s API at a separate fee.


Evaluation

Success for ProbeLab’s work comes when networks face no surprises. We strive to build resilient software that runs 24/7 and produces metrics and alerts for engineering teams. After more than 3 years in operation, we have never experienced outages longer than a few minutes, which makes us confident that any incident will be captured by our specialised tooling.

On the technical front, some evaluation metrics are as follows:

  • Eclipse detection time target: Equal to the routing table update interval, normally set to 10mins.
  • Node coverage accuracy: Less than 3% discrepancy to actual network size. Discrepancy is due to node churn and routing table update timings.
  • Node geolocation accuracy: N/A. Depends on the accuracy of our database provider. We haven’t received any complaints from our monitored networks so far.

Team/ Organization

ProbeLab is developing measurement and monitoring tooling for the P2P layer of blockchain networks. ProbeLab is the leading team in P2P network metrics and analytics with extensive experience working on libp2p- and devp2p-based stacks. Running specialised infra 24/7, the team is producing both executive and tailor-made dashboards on metrics that are most valuable to engineering teams. Tailor-made dashboards, metrics and studies are normally published in forums, such as ethresear . ch. In some cases ProbeLab supports internal engineering teams with data that is not made publicly available.

GitHub repositories maintained by the ProbeLab team:

GitHub profiles of ProbeLab team members:

  • Dennis Trautwein (DevOps and Protocol Engineer): GH: @ dennis-tra
  • Steph Samson (DevOps and Infrastructure Engineer)
  • Mikel Cortes (Protocol Engineer): GH: @ cortze
  • Yiannis Psaras (Team Lead): GH: @ yiannisbot

Conclusion

The P2P layer of any blockchain network is the basis of everything that is built on top. It is imperative to keep the P2P network healthy and avoid collapse at this layer, because in that case, everything on top collapses as well. We argue that this is possible through detailed insights and metrics on core protocol functionality.

This GIP proposes critical metrics for the Gnosis Chain P2P layer that will keep the network resilient, secure and performant.

2 Likes

Links relevant to the post, but not allowed by discourse policy:

Thanks for your draft proposal @yiannisbot.

A brief question for you: has the content of your proposal been raised with any relevant core contributors working on Gnosis Chain infrastructure/monitoring as part of Gnosis Limited? And, if so, does it have any supporters/patrons from Gnosis Ltd willing to explain these discussions?

As it stands, a lot of the action items, risks and benefits described in your proposal are of a quite technical nature. It’s hard, at first glance, to discern the relative significance and value of this work. Though we appreciate your explanation of the benefits, we’d appreciate a second opinion.

Thanks again, and in advance :pray:

2 Likes

Thanks for the explanation and thoughts! Yes, we have worked with @armog , Hugo and their team in the past and we’ve discussed the items in this proposal with them already.

2 Likes

For the past few months, we’ve collaborated closely with the ProbeLab team, led by @hdser, Head of Data at Gnosis. They added Gnosis support to Nebula, one of their flagship tools, and we now publish weekly DiscV5 network reports. Their early work also enabled us to deploy our own P2P monitoring crawler, allowing continuous collection of high-value data on network peers.

We’re about to launch a new dashboard—the successor to https://metrics.gnosischain.com/ —where
this data will feature prominently. We now monitor the P2P network layer 24/7, and these insights are a key input to our ESG estimates.

ProbeLab maintains several additional monitoring tools and has a clear path to improving the current stack, which would significantly benefit the broader community.

For this next phase, we would like to invite everyone to talk about this proposal and find out how to allocate resources best.

5 Likes

Thanks @yiannisbot and @armog for clarifying - that’s very helpful.

So, as we understand it, this is an existing and valued data partnership, and this proposal is a question of extending that partnership with additional monitoring tools and analytics.

Perhaps we can also rely on an assumption that this is not an expense that Gnosis Ltd has budgeted for or plans to cover, so presumably isn’t a necessary component to our existing data and network contributors, hence why it’s a question for the DAO. Please do correct if this is wrong.

We can then turn to assess value for money. Our initial sense is that the subject of this proposal’s milestones are quite granular and specific monitoring and analytics, which will be of most value to a smaller number of users focused on network health, so namely (as the proposal says) validators, client teams, infrastructure providers and core protocol developer. It may be useful if some more feedback or testimonials from more potential users could be gathered to corroborate the proposal’s conclusion that this would benefit the community at large. We count @armog’s comments as one example:

Beyond that, we do approve of the milestone structure of this proposal, which would allow for changes or even cancellation if issues arise along the way. We also think the total cost does not seem totally unreasonable, though we find it difficult to gauge the amount of work involved.

We would however appreciate some clarification as to whether the additional tools and metrics introduced by this work would be subject to any future maintenance costs (whether ongoing for things like hosting or one-off costs for updates). We wouldn’t want to get 3 months down the line with all the work complete to then find a surprise proposal for a significant amount of necessary maintenance costs.

Thank you again

1 Like

Thanks for publishing this. View: Support with conditions.

Rationale: P2P‑layer visibility is foundational for validator reliability and incident response. The EL discv4 coverage, CL protocol probing (status/metadata, subnet health), Ants‑based network sizing, and eclipse‑attack detection meaningfully reduce blind spots. Noting Gnosis Data’s endorsement and planned dashboard integration, this looks like a pragmatic extension of current work.

Before Snapshot, could we lock in:

  1. Acceptance criteria per milestone, e.g., eclipse detection ≈10 minutes, network‑size accuracy <3%, and publish weekly EL+CL reports on a public dashboard.
  2. Integration plan with the upcoming Gnosis analytics dashboard (alert routing, owners, cadence), to avoid duplication with Gnosis Ltd monitoring.
  3. Data openness: dashboards and derived aggregates public. What stays paywalled (raw data/API) vs. open?
  4. Maintenance clarity: several items include 1‑year ops. Please state whether any follow‑on budget will be requested and when.
  5. Discv4 hardening plan: today’s crawler “stops for unknown reasons”. What fixes and success test will close this risk?

With these in place, I’ll vote For. Happy to review a revised draft and proposed milestones/payment gates.

3 Likes

Hi Yiannis - Feel free to move this temp check to phase-2 where the community can vote here on the forum about this proposal. You’re proposal will receive a GIP # when you update this post to phase-2.

Our DAO Docs are here, for your reference: Gnosis DAO | Gnosis DAO Docs

2 Likes

Great to have the metrics, but what I really miss is number of nodes, which seems much more important as number of validators.

Reading this makes it even more valuable…a total node number, even behind NATs would be great.

If this will be included in

I will for sure support this.

1 Like

Thanks everyone for the constructive feedback, the support and suggestions! Let me address the comments raised to make sure everyone is happy and on the same page, before moving to phase-2.

@staworth:

Very good point. The maintenance and infrastructure costs amount to approximately 40% of the grant. We plan to submit a separate proposal for ~$35k for maintenance and infra costs (for all metrics and dashboards) 1 year after the completion of this project. So, if, say, this project gets approved and work starts on Oct 1st 2025 and finishes on Dec 31st 2025, then the current grant covers maintenance and infra costs for 1 year, i.e, up to Dec 31st 2026. We would submit a separate proposal then to cover costs for 2027. I hope that sounds reasonable.

The subsequent grant will cover all metrics and dashboards, the tooling running 24/7 and tooling adjustments for all network upgrades that will happen during this period.


@uncannyowl:

Yes to all, apart from the eclipse detection period, which was an oversimplification on my end. In fact the detection time cannot be smaller than the interval at which we run crawls on the network, which is every 2hrs. The theoretical minimum is the routing table update interval, but the practical minimum is the time it takes for a crawl to finish - in our case this is 20mins. However, from past experience from the IPFS Amino DHT (and other networks), we find it an overkill to crawl the network every 20 mins, hence we have reduced the frequency. We plan to explore the option of having alerts in place, though to increase frequency if keyspace density goes above a certain threshold.

IIUC, @hdser and team is already working on this and we will coordinate to avoid double work.

All plots and dashboards will be public. Data (raw or aggregated) will be available to Gnosis Ltd. free of charge (already the case with data we’re producing). The data will be available to the community for a small monthly or yearly fee.

See comment further up. We will submit a grant proposal for maintenance and infra costs 1 year after work on this project has been completed to cover the year after. The expected grant amount won’t exceed $35k.

We will run the crawler in our infrastructure every 2 hours, monitor closely its operation in the discv4 network and debug any problematic behaviour.
If we see seven consecutive days of uninterrupted service (84 crawls) we will consider that things work as expected and mark this as complete.


@refri:

The total number of nodes will definitely be live at probelab.io. I believe this is the plan for metrics.gnosischain.com, but will work with @hdser to make sure it happens.


Unless any extra clarification is needed, I can move the proposal to phase-2.

1 Like

Very clear - thank you @yiannisbot. We agree that your maintenance costs sound reasonable, and are happy with the idea of a further proposal being raised after the initial maintenance period is complete (i.e. 1 year after work is completed).

We support moving this to Phase 2 in any event, and are leaning towards supporting the proposal in general.

Thank you! @staworth @john_szczepaniak I created this: Should GnosisDAO support ProbeLab to produce network performance and security for Gnosis Chain P2P network? and added the feedback from this thread as part of the post. I wasn’t able to edit the current post to change the phase - not sure if that’s the preferred workflow :slight_smile:

1 Like

Hey Refri, sorry for confusion. As I described on my post our new analytics tool will be the successor to https://metrics.gnosischain.com/ —where
this old website data and new datas such as number of nodes, geographical disribution and client usage details will feature prominently. It’s not ready to publis. just still some UI design and content.

3 Likes

I will support this proposal

1 Like

Thank you! It has moved to phase-2 here: Should GnosisDAO support ProbeLab to produce network performance and security for Gnosis Chain P2P network? if you want to for the proposal.