Ethereum consensus core developer at Lighthouse, Sigma Prime. Bridging the gap between research and implementation he is contributing on multiple exciting fronts of the Ethereum roadmap.

Dapplion

After the merge, Ethereum has a finality gadget to provide an economic assurance that transactions will never be reverted. When 2/3 of the validator set are online and agree, we finalize. Otherwise, we enter a period of non-finality which can be very long, up to a few weeks. Long non-finality has never happened in Ethereum's history and could trigger a cascade of failures that will kill liveness. How can we harden the network against this? How high are the stakes?

Stage 1

How long non-finality could kill Ethereum

 Hello, hello, thank you. Let me drop the water. So hello everyone, I'm Dap Lyon, Ethereum Cordep at Consensus at Sigma Prime. And today I'm going to talk about how long a fanatic could kill Ethereum. Or some healthy dose of fear mongering to get all the core devs here to care a little bit more about this issue. So just to start, let's get everyone on the same page. This talk is not labeled as an expert, so we'll start with what the hell is finality. So Ethereum has a strong crypto-economical guarantee where if a block is finalized, it will not get reverted unless two-thirds of the stake, so an insane amount of money gets destroyed. For the purpose of today, what you have to know is when we finalize something, everything that descends from it is a possible block or state. Everything else we don't care about. So when the network fails to reach finality, in other words, two-thirds of the chain agree on something, then this section of blocks and states just keeps growing. Because the finalised checkpoint doesn't get updated and it just keeps going in the past. And in the past. So today we're going to run through an experiment. I'm going to show a hypothetical failure mode of the Bitcoin chain and we'll talk on all these possible concepts and how they tie together into something that could actually kill Ethereum. So let's start with some finalized checkpoint and here are some descendant block A. But this is a longer chain. I'm just representing here shortly for succinctness. Next, oops, some client has a bug. With more than one-third of the state, say like a consensus issue where they reject the block that they shouldn't. And this causes a chain split. So the faulty client cannot produce on top of A, so it just goes on its own fork and mines B prime, X prime, so on and so on. Everyone else stays on the majority fork, so B and X, and with these three dots, I'm representing a long chain. If we're talking about one day of non-finality, this could be 7,200 blocks, so quite a lot. The first thing that happens is most clients spend not significantly, but a decent amount more of this space during non-finality as the amount of blocks and states that they have to keep for fast access is higher. Then, now it's when things get interesting. Let's assume that one of the faulty clients, for some reason, be another bug or some network problem, they lose out of sync or they crash. And they think that B prime is the head. They will act on it and produce this block, X prime prime, which forks from B prime. This block is very expensive to process. And we'll show now why. And everyone else must process. Everyone in the fork X prime and everyone in the fork X. So these blocks are expensive because in Ethereum, states are not only dependent on the parent but also on time. When you want to process block X prime, you have to load the state at B prime and perform something that's called a state advance. You have to run this code, which is relatively expensive and increases linearly with state size, a lot of times. So in this case, if it's one day of non-finality, you have to do this 225 times plus hashing. So I don't know the numbers, but we are talking a few minutes, potentially. And we can have many of these blocks. And these blocks are very expensive, and what they can cause is that now some nodes get overwhelmed. They trigger resource exhaustion. So let's consider now one of the non-faulty nodes at x. They have to ingest these expensive blocks, and now they also fall out of sync. So they start to produce these annoying blocks that, again, are expensive to process. And here you can start to see how things spiral out of control. Now that we have a decent amount of forking, disk space really shoots up. Because, again, we are extending the space of possible blocks and states not only in the dimension of time, but also in the dimension of forking. And now clients can start, again, depending on the client and how they run, to run out of space. If a client has no space, it stalls. It cannot progress. It needs manual intervention. So because of that, let's consider one of these forks. Now it doesn't have enough participation, because clients are losing disk and going out of sync. Then you have reorgs, which are expensive. And another interesting failure mode is that some ELs cannot handle reorgs past a certain death. If that happens, they stall because they don't have enough state to process the chain. So more clients going offline. Yada, yada, yada, yada. Also, other things to consider, clients could crash with OEM depending on how their state cache on memory works. And again, triggering sync issues. So in these accidental, again, accidental failure modes, any sort of bug can spiral out of control as it causes resource obstruction, which causes faulty blocks, which causes resource obstruction, and so on. And this behavior is quadratic with time to finality. Like the longer finality goes on, the more expensive these operations can become. But that can be triggered by an attacker. All of this so far has been accidental. And accidental failure mode is problematic because it has a lot of stake. But if an attacker wants to exploit this and has significantly less stake, potentially like some hundred keys, it can truly wreak havoc. So the problem, as I was saying, is these very expensive blocks that do a lot of skips. And you can create a lot of objects that exploit this. So for blocks, you need stake, you need to have a valid proposal signature, but you can brute force valid proposal slots. If you have maybe like 100, so like the crusader, the low-carb crusader could easily attack the chain in the finality of like one day and create about some hundreds of blocks, which again, consider each one takes one or two minutes to process, you can do the network like that. Aggregated stations also require stake. You have to brute force the aggreter duty. And unaggregated stations, you don't require stake as you can just produce an invalid at the station and it will still force nodes to compute the shuffling. The good news is that we are fixing this last attack vector on Electra and you will still force nodes to compute the shuffling. The good news is that we are fixing this last attack vector on Electra, and it will also require stake. So look at some history. We have had some issues with finality in the past. The most relevant one happened in 2023, where we lost finality for about four epochs. What happened is something very close to this failure case where some nodes considered to be synced, and they propose a very expensive, not as expensive as the ones we're talking about, but some attestations that take some time to process. Unfortunately, Prism had some caching issue where it did this work over and over. And it didn't have a well-structured queueing system to protect themselves against clogging. So any relevant work to progress the chain, like blocks and attestations, just didn't happen and that resulted in the loss of finality. Also Medalla is the quintessential event of nonfinality. I'm not going to talk in detail here, but we can talk about it later. Also GoEarly suffered a very brutal death with a bunch of cascading effects. And funny enough, the peer-dashed devnet that we ran recently had a massive reorg of 130,000 blocks that triggered this edge case where none of the ELs could continue because they stalled without state. So yeah. What's worrying about failure mode regarding non-finalities is that it's very spirally. Things cause things and it gets pretty out of hand. So far we have talked about accidental issues that relate to client bugs, network partitions, these sort of things. But the worst case that we should consider is what's the possible longest non-finality that we could see on the beacon chain? And that's related to this failure case where say that we have supermajority geth, they have a consensus issue where they mint infinite eth. So it's not a fork that we can canalize. In that case, if they finalize the wrong fork, they will be stuck there. So on the correct fork, in this case chain B, we will have to wait for them to leak out. That's going to take a long time. in this case, chain B, we will have to wait for them to leak out. That's going to take a long time. So if Geth is like 70%, we are looking at 32 days. So that's in my opinion, that's the worst case that we should aim for. And this is realistically something that could happen if for some reason Geth gets a significant amount of market share. So what can we do? What can we do? Easy, just don't do bugs. Don't release clients with bugs, then we don't have non-fidelity and easy, not a problem. Now for real, there are no strong mitigations, it's just we need to harden the client. So don't run out of this space, don't run out of memory, don't get exhausted. And third, we're going to look and explore if we can actually reject some of these useless network objects to protect against the most blatant cases of those. All of these solutions, they work on this. It's not actually a trilemma, but it's a trade-off space. In Ethereum, we work really, really hard to preserve liveness. That's why we have this hybrid consensus mechanism where the chain goes into this mode justfidelity mode just because we want to preserve liveness. We don't want to be in a situation where no operator intervention is mandatory to recover the chain. At the same time, we don't want to die and we want to process everything in a timely fashion. So the first point and and the most dangerous, is running out of disk space. Here, there are a bunch of easy rules that I think most clients follow. I think Lighthouse is one of the ones that don't follow all of them, but we are working on that. And in Lighthouse, we are now introducing this very interesting optimization, which is storing everything as divs. That's commonly known in consensus folklore producing this very interesting optimisation, which is storing everything as divs. That's commonly known in consensus folklore as three states, and that's going to be rolled out in the next version, but it only affects the freezer database. What we want to do next is apply the same principles, but for unfanelised states. And that's going to be massive, because now every time we want to store a state, instead of spending 200 mechs, we can just spend half. And we have really good compute and apply times for a bunch of different divs. How this is going to look like, we'll have this hierarchical div structure where if you want to recuperate, say, the state in red, you load snapshot and then iteratively load the div, apply, load, apply. This is very, very efficient in terms of disk space, and as you saw the numbers here, it's actually pretty fast and even faster than replay. The complication with the unfinalized section of the chain is that we have moving finality, which complicates a bit the design. But if we just extend the diffs into some finalized range, keeping at least one per layer, the design works. And pruning is not that complicated. For memory, there is not much you can do. You just have to make sure that your state cache doesn't blow up in these circumstances. And again, three states, memory helps a lot. And this is already in production in Lighthouse. And the last one, and I think this is the most important, you need to have some strong queuing system where you don't let yourself get exhausted from this garbage that can come from the network. Looking at the main net incident that we talked about, if instead of having this queue, you split it into sections, one that has higher priority for things that are useful to you, in this case, let's say, the sending of heads, and then everything else gets processed with lower priority, we should be safe in this case. Lighthouse at the moment is working on revamping all our queuing system, and we're going to do something with this, also with fairness, so P0 doesn't exhaust P1. P0 doesn't exhaust P1. And the last point is, okay, we know that these blocks are really bad. Can we just ignore it? That would be really nice. And unfortunately, no. If we want to preserve liveness, there are a lot of edge cases where if we start to ignore blocks, then the chain could stall. So I think this is the only one I've seen that I like, and it's on Prism in production. This is by POTUS. If you get things that extend something previous to justified, it's more complex than this, but just to simplify, you can ignore them. And this would have protected Prism from the issue that we saw before. Unfortunately, this is just one of the many failure cases that the chain can have. So it's not like a bulletproof. So what are the next steps? Well, testing, testing, testing, and testing. The issue with non-finality is that we have to do Pektra, we have to do Pirdas, we have to do so many things that unfortunately dealing with non-finality is that we have to do Pektra, we have to do so many things that unfortunately dealing with non-finality just goes like priority number three or four. I talk with the Ethereum DevOps team, the PandaOps, and what we should do is have some cyclic tests, either continuously or maybe like quarterly, where we test non-finality. We want to uncover these bugs ahead of time and not when finality dies either in an important DevNet or main net. Also in Sigma Prime, we have a tool that we have been using with great success in this source of attack nets, and it's insanely good at killing networks. So we are with Michael in the process of revamping this tool, and we'll definitely use it in these tests. So yeah. But, yeah, I mean, just to not be too fear-mongering here, the beacon chain has been exceptionally stable through its lifetime. And the only time it lost finality was for epochs, which is nothing. We have a robust set of operators who are very diligent, hands on, the same for core devs. We have triaged and fixed issues really quickly. So again, we have to work on this. It's my mission as a core dev to make sure that Ethereum never loses liveness. But yeah, we've been doing great so far and we should be proud of it.

https://streameth-develop.ams3.digitaloceanspaces.com/transcriptions/6736c6b89dbb7a90e1cd35b3.vtt