Balancer hack: hard fork

General
1

At the start of November the majority of Gnosis validators adopted a soft fork in response to the Balancer exploit affecting Balancer‑managed contracts on Gnosis Chain.

Since then contributors, validators and other stakeholders have been working on a hard fork intended to create a technical path for the potential return of stolen funds. Gnosis Chain rules are baked into the code the validators run. The last step is the crucial one: a majority of validators must adopt the new binaries in order for the hard fork to succeed…

The releases are now available and node operators have 10 days to update in order to remain in sync as determined by validator consensus. The hard fork is set to activate at 16:11:40 UTC on December 22nd. Nodes that do not follow the chain with a majority of stake will get penalized.

The execution client versions to run are the following:

  • Geth: v1.16.7-gc.9
  • Reth: v1.0.0 (requires a full database resync)
  • Nethermind: v1.35.8
  • Erigon: v3.3.2

The consensus layer client versinos to run are the following:

  • Nimbus: v25.11.1
  • Lighthouse: v8.0.1
  • Teku: v25.11.1
  • Lodestar: v1.37.0

There is still a live community discussion around how people will be able to claim back their funds, as well as how contributors involved in the rescue mission may be recognized or compensated. Right now we’re focused on enabling funds to be recovered by Christmas. Once they sit safely in a DAO controlled wallet we will figure out everything else.

We’ve also begun coordinating with all of the involved stakeholders on a technical post-mortem that we’ll share ASAP. It should make for interesting reading.

Finally, this whole process has delayed the Fusaka hard fork. The updated timeline is to execute on Chiado before Christmas and then schedule it for mainnet in the new year.

We believe that in due course, validators should not be able to censor transactions and the underlying network infrastructure should be actually blind. We commit to work towards this future, but in the meantime encourage a community discussion on how and when the community should wield this power it still has when acting in concert. look forward to continued discussions about how Gnosis Chain can maintain credible neutrality, while also clearly defining how / when action should be taken while still technically feasible.

In the meantime, we want to thank everyone whose work has made it possible to right a wrong and recover funds even while the path was not always clear or comfortable. We are very grateful to you all, with a particular mention of the client teams who worked tirelessly to enable this and the validators who are ultimately tasked with some of our most challenging decisions.

As always, please let us know your thoughts.

20 Likes
A Framework for the Future
Balancer hack: claim process
Balancer hack: claim process
2

Hi Gnosis community,

I’m Luca, the founder of Serenita. We manage a relatively high number of Gnosis Chain validators, where most of the stake has been delegated to us. Because of the fact most of our stake is delegated, we’re interested in gauging community sentiment around this hard fork. We are currently not aware of broad discontent about this rescue operation, and therefore plan to support it with the intent to return hacked funds to their owners.

As you may be aware of, a soft fork is currently in place on Gnosis Chain that prevents the attacker from moving funds through what is basically a very targeted form of censorship (with a single address being censored). Shortly after the hack, we have made the decision to support the soft fork, mainly to buy the Gnosis community some time to make an informed decision on what to do next.

Ideally, we wouldn’t have had to take that decision in the first place. In response to the current situation we would like to (help) set some ground rules based on which we can more easily, and quickly, decide what to do if a similar situation should reoccur. The sooner we can define these rules, the better. If Gnosis Chain is to be a place for low-risk DeFi where we want to do everything possible to rescue hacked funds, we should define exactly what that term means, and the exact conditions under which we want to take severe measures like censoring an address. We strongly want to avoid any unnecessary censorship and irregular state changes.

Onto the hard fork. The hard fork spec defines a BALANCER_RESCUE_ADDRESS set to 0x7Be579238a6a621601Eae2c346cDA54d68F7dfee, and also says that this address “should be a robust multisig or Safe configuration”. However,0x7Be579238a6a621601Eae2c346cDA54d68F7dfee is currently a Safe with a single (seemingly EOA) signer - 0xd722EC6853e6EbAaf8664602A37855FAe872E482. I expect more signers will be added to this Safe before the hard fork date, bringing the Safe under control of the DAO? More clarity on who exactly will be in control of the post-hard fork rescue operation is needed here in my opinion.

Lastly, a huge amount of work went into this behind the scenes, and I want to applaud all involved teams for their efforts!

6 Likes
3

Hello, thank you for this post. I have concerns that don’t seem to be addressed, and I’m not the only one; we already discussed this topic here: Balancer hack: Update - #7 by PeterTheOne

The main concern is the unequal treatment in past cases where such a process could have been applied compared to future cases.

Before we can move forward with the hard fork, it’s vital to define the process surrounding it so that all similar cases can be handled, and not just those that benefit one party or another.

The problem is, are we going to perform a hard fork for every wallet hack or anything else? Won’t this lower the quality of applications, which will assume that a hard fork is possible in case of a hack? Could a malicious state exert pressure to cancel transactions or seize funds?

This hard fork procedure represents a major shift in the philosophy of immutability, and I’m concerned about the potential pitfalls.

Validators are key players whose role is to enforce a set of rules and preserve the chain’s integrity. Accepting the hard fork could set a dangerous precedent, opening a Pandora’s box and bringing the Gnosis Chain closer to traditional finance.

I’m not saying we should reject it, but as described in previous discussions, we need clear and fair rules. Anyone or any protocol that needs it and meets the rules must be able to trigger the process, not just a select few. Therefore, in my opinion, we must first validate this process before approving a hard fork.

5 Likes
4

I can see where unequal treatment is an an emotional argument.

However I would say it’s a smaller one simply amplified by emotions. Ofc there will be situations where the focus is naturally right there vs. any peripheral situation where first some effort is required to figure out what is the situation and if it is real. Also the volume of funds are a factor why some things will naturally get more attention than others. So I would not judge that too much.

The greatest issue is the precedence - if immutability is not a thing, then what prevents the DAO to overwrite Blochchain state more frequently in the future?

So this is the most important point:

Non-immutability means it is merely a more complex and technically more powerful version of SWIFT…
The immutability argument is actually so strong - that it actually should kill any idea about overriding the blockchain state from the beginning.

However here we are - and we don’t want to to reward sneaky hackers as well.

Also we are humans and make mistakes and accept that other people make mistakes an - in both suboptimal governing decisions, as well as in smart contract implementations^^.
This fatherly attitude to tolerate mistakes and fix things to not let the kids feel the hardship of reality is basicly the main argument for the hardfork - and I am really too soft to completely reject it as well. :stuck_out_tongue:

So out of my own faulty judgement I would position myself like this:
A hardfork can be done, IF we substitute the otherwise unproductive painful backwards consequences with productive forward-looking consequences. These Consequences need to be agreed on before the singning-off to the hard-fork.

Here an example of what that could be:
- Hightened awareness about the risks and thus greater vigilance of developers to not make these mistakes - also more rigorous battletesting before deployment of a system.
- Some more whitehat-hacker-guilds need to be all around and theorizing about possible hacks of the deployed systems and taking action before a blackhat group do. How we are going to achieve this? idk :confused: Blackhat groups are often state-funded and have massive idealistic, monetary and status-related incentives. Their motivation to get the exploits is insane. Compare this to an idealist without funding - ( the funding makes little difference).
- Active work needs to be done to massively decrease the technical possibility of future hardforks. E.g. rewards for the good validators favours network rules vs DAO compliance.

As such this case could be seen as a forgivable baby-mistake as the technology is appearently not mature yet similar to the ETH hardfork. And the “Baby” is luckily still small enought so that we can just overrule it’s misbehaviour by over own power.

Maybe these examples are a bit lame but the point is clear. If we donot have a consensus about the seriousness of these implications of potentially violating immutability without positive consequences I would strongly oppose any hardfork.

Rephrased to the opposite:
If the only learning is: The blockchain is actually not immutable as there only needs to be a consensus of less then 60 people to overwrite the blochchain state and it would be done fast as long as these people see it fit. Then this would be the worst possible learning from this incident.

Cheers!

1 Like
5

I’m a bit surprised by this post, to be honest. According to the second update on the hack:

Now that we have some breathing space, we will share our learnings and put the next decision — whether or not to execute a hard fork — to the DAO for a vote.

We will also propose a set of principles to guide us in any future challenges. This way, we can align on a framework for how various parts of the chain infrastructure (bridges, validators, stablecoin issuers, etc.) can and should react under specific scenarios. These will also be shared for open debate.

I was expecting at least a forum post presenting different proposals for the hard fork (and I would also like to know whether any alternatives were considered). I do not feel confident about executing a hard fork while notifying all validators only 10 days in advance. It feels rushed, even though I know you have been working extremely hard since the hack to find the best possible solution.

Although I was not affected by the hack, I want to thank everyone involved for the hard work. That said, in my view, this is not a decision that the community should be expected to make within such a short time window. Hard forks are usually announced months in advance in order to properly coordinate the community and gather feedback.

Additionally, more clarity about the Safe that is intended to receive the funds would improve transparency regarding what validators are actually being asked to vote on. Given the current situation (as mentioned by serenita_luca), this does not look good.

I’m not against the hard work, but a little bit of more transparency and communication will be very much appreciated, on top of a temperature check of the DAO to understand if the DAO want’s to do this step in the history of the chain. Again, thanks for the hard work you are doing

5 Likes
6

@serenita_luca Thank you for your message and support! I want to reassure you that we are well aware of the current configuration of the Safe. The signers will indeed get swapped out soon, well before the fork. We used a single signer to have the exact same setup for our shadow forks as for mainnet for safety reasons. In hindsight, we should have swapped out the signers before going public, and this will most likely happen on Monday. I will keep everyone here posted as soon as this happens.

To be honest, I cannot really comment on this. It sounds to me like the issue linked to GIP-127 had major differences:

  • It resulted in way less lost funds
  • It was not hardforkable, as the funds moved and the state of the chain changed a lot
  • To my understanding, the issue didn’t come from a nefarious hack but from a misconfiguration by a service provider (kpk?)

That being said, I’m not super familiar with the other issue, so don’t quote me on that.

I doubt that this will happen. To contextualize this a bit: Gnosis core developers and 4 client teams just spent 1.5 months exclusively on reacting to this hack, fixing it, testing it and deploying it. This delayed Fusaka and came at a cost to Gnosis and everyone involved. I definitely agree that we need stronger guidelines that define when this is an option we are willing to consider. I have a few reasons for why I think this is the right decision in this particular case:

  • This was a very big hack, with $150m hacked in total, but also a significant percentage of Gnosis Chain’s TVL (3%+ after EURe and osGNO froze the funds on their own).
  • Other chains reacted by freezing and recovering funds immediately, without validators even being able to oppose this choice, and the community seemed to be mostly in favor of it.
  • I don’t see any ethical issues about this. It was clearly a well coordinated hack for nefarious reasons, and there’s no good reason to leave that money in the attacker’s hands.
  • It is technically fairly straightforward and safe for the chain.

Nothing is immutable. Both Bitcoin and Ethereum, arguably the most cypherpunk chains, also went through hardforks that reverted hacks or issues. To me there’s a very important distinction between us spending 1.5 months of time and money into patching a hack in a very public and open source manner that is validated by a majority of the Gnosis Chain validators, and banks being able to arbitrarily close or debit accounts. But I agree on most of the rest.

We have rarely given much more time than that to update nodes on Gnosis Chain. The latest example was for Pectra: releases were cut on April 23rd and the hardfork happened on April 30th, which is only 7 days to update. I can assure you that on the technical side, nothing was rushed. We’ve done 3 hardforks, tested all 4 clients extensively, and only decided to go through with this once everything was figured out and tested to the best of our ability.

I’m curious about what types of proposals we could have come with. Balancer tried to recover the funds and negotiate with the hacker, but that didn’t work out. So I only see 2 options left: hardfork or let the hacker keep their funds. In terms of conversations, we have always talked about a potential hardfork, and I believe that if there had been any significant push back we would have heard about it already. It’s been more than a month after all.

I’m all for having this conversation publicly of course, but ultimately the validators are entitled to run any software they see fit, and they are not forced to follow the DAO’s decision on any of this. So in my opinion (and I’m not talking for Gnosis Ltd here), there is no strong argument to do a DAO vote first, even if conversation around this would be appreciated.

5 Likes
7

To back up this claim, can you provide us with the details of the affected individual wallets and the amounts? I imagine you must have calculated this internally, as I can’t believe Gnosis LTD would have been so eager if the majority of the affected funds belonged to the DAO and a few millionaire co-founders of the same entity. If, after removing these two amounts from the total loss, the sum becomes small enough, why can’t we use the DAO to cover the losses instead of setting a precedent? Otherwise, what is the loss threshold for our lords at Gnosis LTD to deign to save us?

Can you explain what you mean by ‘community’? Over on the forum most responses favored freezing the funds while we talked about what comes next. I don’t see any public conversation — is ‘community’ being used to mean only a private discussion room among a select few?

There was a difference in treatment between the two situations. In one case, you immediately blocked the funds; in the other, you hid the truth from your users, causing them to lose hundreds of thousands of euros. Once again, I will ask my question, what is the loss threshold for our lords at Gnosis LTD to deign to save us?

I believe you are liars; I am no longer a GNO holder because I have finally understood that you have misled us.

image
image730×283 31 KB

I invested in GnosisChain because I thought I was joining one of the most decentralized networks after Ethereum; that has been your primary marketing argument for months.
The reality?

image
image1545×569 121 KB

Probably between 100~300 individual validators, and it doesn’t take into account your wallet splits made by the DAO or even your founders to inflate this number.

image
image1545×569 91.8 KB

You have actively paid people to run validator at a loss. Everyone is outraged when it comes to Solana, but for some reason that escapes me, you get a round of applause while people sip their smoothies.

Go ahead with your hard fork, give the money back to your co-founders and by proxy to the DAO, but stop pretending to be something you are not. You are not and never will be a decentralized blockchain; you are the German love child of Solana’s malice and BSC’s centralization.

8

Yes I think this would be a point the community needs some consens.
What are the exact conditions that could allow for a consensual hardfork?
Maybe I should make a GIP to define a “hardfork immutability exception clause” who wants to help drafting?

Well strictly speaking you are right. In a way things are not perfect yhea, but why are we here?
And how close to this is BTC and ETH right now?
Did the ETH community make an hardfork to contain this hack?

The immutability of Blockchain state is definitely an super important part of it.

I strongly believe there needs to be hard pressure on it to become more and more safeguarded. Otherwise it will turn into an “easy” choice would cancel out the reason why I am here.
The evolution has to go into the direction to make it a harder choice and at best render it completely impractical.

Fully agree, we need a table of Options and what they mean exactly and then cast a vote.
IMO a version where the funds are not recovered/compensated in a different way - could be an alternative - is it so?

Huh? The hacker should be prosecuted anyways in the real world - independent of any “blockchain fixes”.

OT:
Just generally this is one of the great problems all along we corrupt law and technology out of laziness and incompetence to actually get the real criminals in the real world. And yhea also because of our failure to evolve the world and our society (culture) into something where the risk-reward ratio of behaving well vs nasty is clearly favoring the good behaviour.

Yes exactly this is the point we should discuss for a decision on it.

Hahaha this one is a bit too off xD However I undestand your feelings. If you already have decided that the reason why you are here has been canceled out. Then so be it.
However I think there are obvious issues of which you have pointed out some and we really have to address them well. I think the xDai and Gnosischain as this ecosystem is at very interesting spot and I feel the desire of many people is to do it right.
E.g. The KPK termination was one of the very positive evolutions in this sense.
However we have to do so much more - especially on the reflection.
Why are we having these problems?
E.g. Why did the xDai community leave?
Etc. …

Well one thing is definitely community synchronization and and consens about fundamentals.
Discrepancy of the powerful core that can just do things. Vs many people with higher ideals and significantly less stake.
And the issue of a hardfork response for “fixing” a hack is exactly at the heart it it :slight_smile:

So I hope we can get some good progress in the sense of some unity around important ideals.
This opposing to the greater feeling that an “elite” just decides they know better anyways - overshadowing their great work they did in the background.

Cheers!

9

@ NolanV
aving lost everything on the Arbitrum channel, if I could recover some funds on EURe/sDAI, it would already be a relief, and for that, I say Gnosis is the best channel to do it! Arbitrum abandoned us! So, stop criticizing it; I myself was affected by this hack, so you can’t understand the victims!

10

Mister Dubi :arrow_heading_down:

Thanks for your contribution to this discussion.

11

It was in response to your review of the fork, that Gnosis was the best channel for taking swift action to prevent hackers from having free rein. This is something other decentralized channels haven’t done. And so, I commend this effort. Especially as a victim myself, I can understand all the victims who might have hope for future recovery opportunities.

12

Will funds deposited in Balancer and staked in Aura be recoverable?

13

You’ve been able to destack for a while now. You’ll just have default tokens named ERC-20, derived from xDAI for EURe/sDAI from the pool.

No longer under the name BPT because of the fork; the LP token name has become neutral.

The LP token contract:

https://gnosisscan.io/token/0xdd439304a77f54b1f7854751ac1169b279591ef7

So, everyone who bought before November 3rd is eligible for a refund. Anyone who bought after that date is ineligible; they’re trying to cheat by believing their purchase of 0.00001 xDAI for this LP token will be worth $1 on the redemption date.

14

This isn’t reassuring. If I understand you correctly, it has to be a major hack before anything is done, but there’s no clear definition of that. That’s the problem. You’re talking about 150 million, and if I’m not mistaken, that’s the impact on all the channels, not just Gnosis (forgive me if I don’t have a clear grasp of the figures; I haven’t followed the hack closely, but the issue of the fork is important).

Okay, so what constitutes a major hack? 150 million? 149 million and we do nothing? Or is 100 million the limit?

But if it’s 100 million, then why not 99 or 95 million?

We could go on like this for a long time. At what point are we going to say the hack, and therefore the people affected, become insignificant to the DAO (or the group of people who control the DAO)?

Are there other criteria, such as the people impacted, or the protocol itself being hacked?

If it’s a protocol that interests you, will you perform a hard fork, but not for another protocol?

For example, if RealT gets hacked, will you also seek to recover the funds since it represents nearly half of the TVL and the protocol that pays the most fees to the channel, or will you refuse the hard fork because RealT isn’t supported by the channel? To avoid focusing solely on RealT, for whom I work, we can ask the same question for others like Sushi, and other smaller protocols.

The problem is that to accept a hard fork, as I said in the other thread, you need a strict framework that no one can deviate from, a framework applied by the DAO for the DAO itself, not for the needs of a group of people. This is the risk with uncontrolled processes that aren’t discussed at a broad scale.

Out of all the users on the chain, how many are even aware of what you’re going to do?
How many people participated?
Is this truly representative?
Are the people who delegated their GNO for staking actually aligned with this?

2 Likes
15

I don’t quite understand the argument about the time spent developing the solution. From the very beginning, questions that have been raised here were brought up, but there was no debate. The people who worked on the Forck solution did so without even discussing it before starting the work.

I’m not saying it’s bad work or that it shouldn’t have been done, but simply that the process could have been different if other decisions had been made, and therefore this argument doesn’t make sense to me.

2 Likes
16

With RealT, we’ve been here for many years now, and we knew xDai channel and their amazing team who supported us.

When the channel was bought by Gnosis, we initially saw it as a positive development, but we were forced to acknowledge that it severely damaged our relationship with the channel’s maintainers, reaching a breaking point.

It wasn’t a sudden, violent decline, but a gradual one, like the degradation of the agreements and services provided by the chain, such as RPC. Exchanges became increasingly lengthy and less productive. Little by little, we felt we no longer mattered, even that we were bothering the Gnosis team (I know this from a source who told me; it’s just hearsay, but I believe it).

Internally, we also considered and studied the possibility of leaving GnosisChain, or at least proposing an alternative. We had to acknowledge that it’s not easy to meet our requirements; GnosisChain, in its current technical state, is unique in the ecosystem.

All this to say that when there are choices, attitudes, and processes implemented and accumulated that don’t please the community, and when the same offering exists elsewhere, then they leave. That would probably have been the case for us if we had found a credible alternative.

There are genuinely good things about the GnosisChain, but I think there’s a real lack of consideration for the protocols on it, a problem with decision-making that doesn’t facilitate access to the chain for users, and absurd decisions (such as USDC.e, on which I’ve already commented and criticized the fact that the .e extension is everywhere on all chains, representing USDC bridged from ETH, whereas here it’s the one that will hypothetically be adopted by Cicle).

Therefore, there isn’t one answer to this question, but rather a series of decisions that lead to a result.

In itself, this isn’t a problem when it’s acknowledged, but here in the discussions we see misleading responses, or at least responses that lack definition.

When we read that the community is in favor, which part of the community is being referred to, and where were the debates or votes held?

I face the same kind of problem in our ecosystem. It’s clearly stated and acknowledged that RealT has significant control over the ecosystem’s service providers for various reasons, and we’ve also announced our intention to progressively decentralize. We take community feedback into account through the DAO. For example, RMMv2 (a fork of aave v2) has been inactive for a while, but funds remain invested. We hold the contracts and full rights to the protocol. RealT wants to shut it down permanently, although we could have done so unilaterally. We submitted the proposed closure and the process to the DAO, and before making any changes, we waited for the vote results.

That’s what’s missing here, it potentially stems from good intentions (or not, because there are various opinions, I’m not able to judge the reality as it stands) but I denounce the process, which inevitably leads to ever greater excesses.

Unfortunately, I’m limited to 3 replies in this thread, so I can’t continue responding. It’s a real shame, I had other things to say.

3 Likes
17

Thank you to the Gnosis team and the client teams for the speed and professionalism of the response so far. The coordinated soft fork and the clear plan toward a hard fork show that Gnosis Chain takes security, users, and ecosystem responsibility seriously.

This has nothing to do with abstract debates on decentralization. When a clearly identified exploit of this scale on an established protocol occurs, and when the chain has the technical and social ability to remediate it, choosing not to act is not neutrality. It’s rather a decision to let the consequences stand. At this point, with the soft fork already preventing further movement of illicit funds, the remaining choice is whether those funds are ultimately returned to the affected users or allowed to remain under the control of the attacker. Either make people who lost a lof of money whole, or fund an illegal enterprise.

A chain that cannot correct an exploit when it has the tools to do so is not “more decentralized”, it is simply less accountable. Taking responsibility here strengthens, rather than weakens, Gnosis Chain’s credibility as a serious environment for DeFi and for users who rely on it to safeguard real economic value.

12 Likes
18

100%

Thank you for the insight, to what happened with xDai :slight_smile:

So my short summary of what I fully support and suggest:

  1. Discuss and destill a framework which contains a condition under which a hard-fork is possible.
  2. With the framework for possible immutability violations and other temporarily accepted “sins” there might be the need of synchronizing on a forward-looking vision with the Cypherpunk ideals being like a north-star.
  3. Acknowledgement of reality. (If the Gnosis “Elite” is activly pushing forward with higher efficiency without great in-depth discussions - that might be temporarily fine and it should be clear in the community.) Of course it makes most sense with the North-Star and the observable progress in this forward-looking vision.
1 Like
19

Why didn’t you defend my intervention with as much fervor when Gnosis hid a vulnerability from its users for months, causing the loss of hundreds of thousands of euros? At what monetary amount do you consider it a company’s duty to intervene?

You are one of the people in this ecosystem whom I hold in the highest esteem; please answer my question.

20

Not nothing - but you are right that for the effective decision it is not directly relevant.
More important is though the “code is law” situation + immutability.

Honestly I think the decision is kind of clear and set in stone already.
Exactly your reasoning is the exact line of argument why that would be done.
So I suppose we will decide to not let the negative consequences stand.

However why is there any debate at all?
It is about the other consequences of the specific actions being taken.
The violation of “Code is law” and “Immutability” is not without consequences.
And those consequences need to be contained/managed.