PeerDAS is expected to be one of the most interesting improvements of the Pectra hard fork, enabling long-awaited sharding on Ethereum, unleashing L2 scaling.

PeerDAS is however just the start with up to 1-2 MB of blob space per slot. We look into the techniques jointly developed by our Codex Research Team and EF researchers to improve this by orders of magnitude, targeting 32 MB (and beyond) of data availability space.

Stage 1

Csaba is senior researcher in the Codex.storage Research group, working on compact proofs of storage, zero knowledge proofs, data availability, and p2p technologies. He is also part of a team working with the EF on the design of p2p networking for Data Availability Sampling. In the past, Csaba was working and publishing on various aspects of p2p networks, anonymity overlays, AI for microcontrollers (TinyML), IoT and wireless mesh technologies.

Csaba Kiraly

From PeerDAS to FullDAS: towards massive scalability with 32MB blocks and beyond

 Hi, hello. So, yes, I have a long title. So I will be speaking about data sampling. And it's a long title and I didn't want to make it longer, but actually I have to make a small remark, it's not even blocks. It's data space that we are speaking of. So let's see who we are first. I'm Chaba, I'm from the research team from Codex. We are working on decentralized storage network, providing high durability guarantees. And our research team is also working with the EF on data sampling and also the client teams. OK, so I think you are all familiar with this chart. You have seen it like 10 times during this DEF CON, I think you are familiar with this chart. You have seen it like ten times during this DevCon, I suppose. We had the IP 4844 and the transaction fees on L2 went down, right? And the right side we see those little, little ones. That's very good. That's very nice of 4844. But I hear that we have a problem. And yeah, to see that problem, I have to zoom in. Sorry. So if we zoom in, we see this problem. So there is a slight increase in L2 transaction fees. And actually, there's a problem. L2 transaction fees are driven by blob space, by data space, and that's running out. problem, and if that happens, prices can rise fast. data availability solutions, in codex we are also building one. speak of the Ethereum data availability solution, so we should scale that Ethereum data availability solution. a little bit behind these numbers. What you see here, maybe you have already seen this one as well, this is the daily average size of blocks on the network. And you see that it was rising and then after the merge it started to rise and then with 4844 it reduced. So this looks fine, right? Now you see this less often. This is the problem. So this is what our node operators are seeing. This is the blobs, so this is the blocks and the blobs cumulative plot together. So this is the data amount that they are seeing. And now we can start worrying, right? There is more data to handle. But we are scientists. We can look at this differently. So let's see it on a log scale. And then it's not that bad, actually. Just like Moore's law. It's actually a different exponent, but we are better than those CPU guys, so why would we? Okay, we just have to do that little technology evolution which keeps this running. And then we have a solution, right? We can draw that line. We just need more Bob space. And that's a plan. And here's a plan. We need to build this and then forward this. And we just have to be a bit lousy about the scale. So it's an unknown scale here and an unknown scale there. And then we solve the problem. What we want is more Bob space and what we want is less node traffic for our operators. We just have to work out these little networking details. So let's see those little networking details that we have to work out. But first, let me have a quick detour, networking. But this is, I was speaking of average box size till now. But average was a daily average. And in the day many things happen, right? Every 12 seconds something is happening. So what you see here is the tail distribution of block size during a long period. You don't even see the small blocks, you just see the large ones. And this was before 4844. And what you see that there were blocks beyond two megabytes. And our nodes have to handle those. What if two blocks of such come one after another? And you have to handle that. So our technology has to handle these things as well. And we were measuring whether you can handle that. And the good news is that you can handle that. So the network was running and live. And what you see here is for different block sizes. These are the bins. Small up to 32 kilobyte blocks. And then between one megabyte and two megabytes. How much time nodes need to get the blocks. And the bigger the block, the more time is needed. This is a distribution. So the nodes which are receiving it only here. and there are many nodes which are receiving it before. Almost everyone is receiving it before four seconds. So things are fine. But clearly it doesn't scale very well, as you see. If we want to increase the block size, then it would go to the right. So we arrive to our point how to scale layer one from the networking side. things. change the slot time, the slot timers, we can change resource requirements. It's a good discussion. It is a bit controversial. We have small gains. We have a lot to lose. So let me not go into that one. Then we can change the gossip sub, the protocol, the networking protocol. We can do improvements on gossip sub for large messages. And this is in the making. It's important. But it's not my talk today. And it's relatively small gains compared to what we need. Still important. Then we can do sharding and data with sampling. And this is my talk today. This is large gains, I hope. And then there's the fourth thing. We can do distributed block building. This is kind of intertwined with many things, but this is large gains. But basically, that would mean re-syncing the whole data flow of the blob transaction from submission of the transaction until the end. Okay, so let's focus on point three, the sharding and data sampling over the network. So data sampling, let's just key concepts. So you need to prepare the data. We were speaking of blobs. There are blobs. There We were speaking of blobs. There are blobs. There's a package of blobs. And then we need an encoding. We need an encoding because we want to sample from it and for the sampling to be effective, we need a laser coding. For the laser coding, we need segmentation. So we need to segment it, chop it to pieces. We need to extend it with a laser coding, then we have to commit to it so that it cannot be, nodes cannot send false pieces basically. And once we've done this, we are ready with the data preparation. And here we have different structures, we have what we call the one-dimensional PIRDA structure where you have blobs which are extended and committed to, or we have the two-dimensional structure where you have blobs which are extended in two-dimension. If you look carefully, you see that actually this is also two-dimensional. The difference is that it's not extended by code in this dimension, and that has consequences. Okay. And then you just have to sample. It sounds easy, right? The question is where you are sampling from and how you are is. And then you just have to sample. It sounds easy, right? The question is where you are sampling from and how you do this sampling. So I wasn't saying anything about that. So let's just see that. So we were typically speaking of DAS, but itA-S, but it's actually a D-A plus an S. So there is the D-A part, which is a property of the network. It's a global property. The data is available. The data is not available. And to have the data available in the system, I have to disperse it in the system. So I have to send it out, different pieces to different nodes, which are custodying it, and only then I can start sampling. So this is the DA part. It's ensuring data ability, it is doing the sharding, nodes are only receiving pieces. And then there is the sampling part. Once the data is in custody at different nodes, you can start test sampling, asking for pieces and getting pieces. Little graphical representation. So you have a builder. It does, in the example I was doing it with the 2D encoding. So it does the 2D encoding. And then you have those beacon nodes in the system. And what you are doing, you are sending out these things, pieces of this, actually rows and columns, to different nodes. Someone is getting only two rows and two columns. Someone is getting a bit more. Someone is getting the whole thing because he's learning hundreds of validators and he's securing so much money, but it's better for him to get the whole thing. Plus, doing that, he can support the system. That's one side. That's just the data availability. Now you have to sample, and that's an individual node, and actually every individual node has to sample. That's his own view. So he's picking a few pieces, and then asking for those from custody. So you're getting this piece from here, that from there, that from there, and if it manages to sample, then things are good. Okay. So we have this kind of two sides of things, and you can see that on the network these are a little bit different. And we need them to behave differently. So we need them to be secure, robust, fast, and cheap. And we need sampling to be secure, robust, fast, and cheap. So we need the same properties for the two things. It's just different protocols. Okay. So I was saying that there is PIRDAS and there is FURDAS. And there is a difference in the data structure. Actually there are a number of differences. If you scan the QR code, we have a light up on FURDAS which is starting from PIRDAS. And some of these techniques apply to PIRDAS, some of them to FURDAS. So what you see here is PIRDAS. Some of these techniques apply to PIRDAS, some of them to FURDAS. What you see here is PIRDAS. In PIRDAS, you have one-dimensional extension, and that also means that when you are sampling, you cannot just get a small piece because it would not tell you probabilistically enough. You have to get a full column. So your sample is a column. Which is more data than a cell that you can sample if you have this two-dimensional extension. But to make this work, you need changes in the networking, changes in the other coding, changes in many things. So, let me just give you a comparison of PIRDAS and FURDAS. Actually let me go back and just go through this list. So we have different techniques here. The first category is about sampling, it's about how I'm selecting what to sample, how I'm selecting those small pieces that I will be looking for. Then the second category is about erasure coding and the using of the erasure coding. Because what I can do is if the data is erasure coded, I can use that as part of the transmission. So when I'm sending the data, I can already use the erasure coding to recover data and then send it on to others. And that's a very important property. We are always using erasure coding when we are doing transmission, actually. Take your mobile phone, take whatever. Here, we can use erasure coding very efficiently to send data. And when we are sending a row, we can use erasure coding very efficiently to send data. And when we are sending a row, we can take a piece and we can send it on a column. And when we have received half of a row, we can extend it to the whole row, for example. And then there are techniques which are protocol changes. And I'm not going through all of them. Just showing you the comparison. So peer-to-peer and foo-to-foos. One is cheap, robust, fast and secure. And the other is cheaper, more robust, faster and more secure. And I was inclined to stop here. But let's see a little bit more in detail. Okay. So FooDAS is cheaper. It's cheaper because it's sampling at the cell level. And the cell is 512 bytes, actually it's 590 bytes because it's the data plus the proof, which is still much smaller than a whole column. So whenever we are so the sampling becomes cheaper. It's more robust. Because it can do a little coding which is allowing what I was calling local repair. But what local means is important to understand. Local means in the sense of the code, in the sense of, let me go back here, in the sense of this square. So I only have a few data, I only have half of a column, and I can repair something. I can generate these. I have these and I can generate these. I have half of the pieces here and I can generate this one. I cannot do that up there. And that's a huge difference. That's a huge difference because that means that every single node can do repair. Not just big nodes who are having old data, but every single node. Actually in that construct we don't need big nodes, which by the way we have in Ethereum, so it's not like we don't have them. But we are not relying on them doing the repair in the data structure. And it's faster. Why is it faster? Because we are working on new protocols, both for the PubSub part and both for the sampling part. So and yes, is it more secure? So the light size is still in research. Should it be more secure actually when this is secure enough that we are changing to this? Actually it doesn't. It shouldn't be less secure. That's still in research. This is still in specification and changing. So let me not do more comparisons, because this will change. And actually, this is still in changing. OK. Sorry. Last slide. As I said, it is not just PIRDAS and FURDAS, but we can go beyond. So, I will show you one slide. slide. As I said, it is not just but we can go beyond. we have to rethink the data flow. So how is this data flow? What is happening? about is what is happening in the consensus layer. So, when a block is generated, how that is distributed. But actually, these blobs are coming as transactions from the world. So let's say one blob is coming in. And that goes to the execution layer. And the execution layer has the mempool, and the mempool is redistributing this, sharing this blob between nodes. Actually when it's blobs, it's not shared like when it's normal small transactions. When it's normal small transactions, these transactions are pushed to others. When it blobs, it's just information that I have this blob and then the others are pulling it. Overall effect, these blobs are spreading in the network. Let's see there's another blob that's spreading also in the network. block proposal. block proposal. block proposal. network. network. block proposal. the data is being distributed. the data is being distributed. the data is being distributed. the encoded version is getting spread in the consensus layer. I think you start to see the problem. It's not the problem, it's an inefficiency. We have an inefficiency in the system in which we are distributing the data here, and then we are actually redistributing the data there. And it's the same data. So we can optimize it a lot, which is good. We need these optimizations for Moore's law to happen. So we found another place where we can save a lot and have that curve go up. So what you can do in first instance is that we do the distribution up there, but whenever the node is realizing that he has a piece, he would need this and this, but he has the blobs, he has the data. He can just pop it up from the execution client. Because the data is also there. Now, of course, this plot is optimistic. Because I was putting this wide blob almost everywhere. And I was putting this red blob almost everywhere in the network. In reality, you don't have every blob everywhere in the execution layer. Because if we could do that, like now you actually have, but when we are scaling, we will not be able to do that. Because if we can do that, then we don't have a problem. Every node can handle all the data which is there. We want to scale beyond that. So in reality, you would have some blobs here, some blobs there, some blobs there. But then when you do that distribution, data can pop up. And that's one of the optimizations. Of course, that's a simple one. We can do much more complex ones where we have interaction between the two, and we are avoiding some of these duplications. But for that, we have to rethink the networking stack here, which is that P2P, here, which is using the P2P. We are doing different kinds of gossiping distribution. This is using GossipSub. This is using other protocols. So this is all. This can all be the same. But that's just that, I think. Okay. Thank you. . Okay. We are now here to take some Q&A. So starting from the top, there are blob fees that play blah, blah, blah, blah, blah, blah, Thank you. Thank you. We are now here to take some Q&A. Starting from the top. There are blob fees that play blah, blah, blah. There are blob fees that pay for blob storage. Is this verified at all? What happens if I get the fee but don't actually store the blob data? Okay. BROP fees that pay for BROP storage. I don't know. Yeah. I'm not sure what the question is about. No worries. Question answer, question asker if you have the question still. Yeah. Fair to clarify. Clarify. But I'll mark this as answered. What are standing problems that hinder current adoption? Okay. So it's just in implementation. So the peer-to-peer is in implementation. It's still compatibility problems. Peer-to-peer will be here very soon. It will not be in Paxra, but it will be in the next fork. For full-to-peer, we still have to figure out many things. So there was a question mark on the security, and we have to work on that to make sure that there are no issues with that. Well, why do we need 2D erasure coding? Can't we put all of the points on one polynomial and extend it? So putting them all on one point and extending it would be one huge code, one dimensional, and that would be very bad for repair in the sense that then you can only repair if you have all the data. And that just means that you have difficulties repairing. The 2D has this advantage of having this code local repair property so that the code distance is small, and you can repair pieces. So you can repair a single column, you can repair a single row, and that gives nodes the possibility to start repairing and contributing even before getting the full data. If full DAWs is way better than pure DA DOS, why not choose to implement it in the first place? As I said, peer DOS itself is changing. We were thinking of techniques for full DOS, then we realized they are actually applying to peer DOS, and then they are part of peer DOS. We're still changing that. Full DOS is still much more research. changing that. It's a simple fact of life that there's too much PC still moving in full DOS to implement that. It seems full DOS approach are much, much better than pure DOS. Should we skip pure DOS and implement the full DOS? Is there any other trade-off? off? So, is using many things that PIRDAS does. So skipping is not a direct skip. It's not fully different. It's giving the 2D Azure core. It has advantages, but it has also complexities. Whether we skip or not might save something on the implementation. When we are introducing it, it might also skip something, but we are losing also the graduality. So overall, I'm not sure it's good to skip, but we can think of it once FUDAS is fully evaluated. What are open research questions? So the open research questions is, so the last slide was fully open research question, or everything around that. On full DAS, there are several questions around the sampling, how we can do the sampling safely in the sense that Sibyl nodes cannot attack the system, for example. So there are still things to clarify there, especially attacks, network partitioning, and other things, how they affect security. There's some fighting and upvoting going on. This is fun. If 2D is better, would there be any benefits of 3D? Absolutely. Multidimensional encoding is fun, but no. Actually not. So we just need the local repairability property, and we have enough of that in 2D. 3D would not contribute to that too much. Can full DOS be implemented progressively? I think so. But we didn't define the plug list. So it's something to think of. But we didn't define it yet. Cool. And then I think this question was already answered. I think we already had that. Right. Yeah. Unless someone's aggressively or I just forgot to mark as answered. Cool. So we have a minute and 34 seconds left. Oh, I was actually spending time. Yeah, no, but so I guess there's one piece to this. You can give him feedback on the QR code, so we'll give you some time to tell him how great he did. Collect a card. But other than that, I think our next session, well, I guess...

https://streameth-develop.ams3.digitaloceanspaces.com/transcriptions/6736cc799dbb7a90e18c1d03.vtt