Researcher in Computer Science

Since my first Ethereum transaction in 2017, my work has shifted from non-blockchain topics in parallel programming and high-performance computing to a full focus on blockchain security testing, performance, and interoperability. 

With my team at the Singapore Blockchain Innovation Programme, NUS, we are building the first open-source, GPU-accelerated EVM execution engine, speeding up transaction execution for various purposes.

Minh Ho

In this talk, we present CuEVM, an EVM executor implemented in CUDA for running a massive number of transactions in parallel. Its primary application is to accelerate fuzzing by testing transactions in multiple sandbox EVMs on GPUs. Additionally, we have integrated it into Goevmlab to support a broader range of use cases. We will discuss the design choices, challenges, results, and future plans to leverage CuEVM beyond fuzzing.

Stage 3

CuEVM: GPU-Accelerated EVM for Security and Beyond

 Morning everyone. Okay, it's my pleasure to start the session today. And yeah, my name is Minh and I'm from the parallelizing on the stuff and today I will introduce about our work on parallelizing EVM execution. So yeah, let's go into the details. So this is a very high level overview of what this talk is about. Basically, we have GPUs, very powerful accelerators, invented decades ago, mainly for gaming, but then it found its use cases in other domains as well. People have used it for general purpose computing, scientific computing, and recently AI picked up the trend. That's what made NVIDIA become the largest company in the world by market cap. So yeah, on our blockchain domain, we found a use case in mining. And you have seen people use it for mining Ethereum for a long time before the merge, but then after the merge, a lot of compute power on all of those GPUs. Some of them come back to become AI training inference, but some of them find that new use case in GK. So you have seen quite a few talks in this DEF CON about accelerating prover, generating proof faster using GPU. But this talk is about something different. So we want to find, okay, those GPUs, accelerator power, what can we use? What else can we use it for to help the Ethereum ecosystem. So we think about one more use case beyond GK to run the transactions in parallel. So nowadays, most of the transactions you can see is not just simple transfer from one account to another. So the transactions now are more complex smart contract executions. And those are quite compute intensive sometimes. That is also one of the reasons for people to set up like block limit and block scale limit so that the computation is bounded. So, you can verify, run all the transactions in block time. But we don't intend to use this work to run your transaction in mainnet. We found another use case in fuzzing to secure smart contract. The use case we found from our work when we built our in-house fuzzing tool to find bugs in smart contract we got the chance to run and test and also compare with state-of-the-art work in academia and also in industry and we found that there's a very high demand to run transaction and test and you get some feedback you get some result so you can finally the main goal is to find the bugs in smart contract to secure your application and GPU is just the main like just the the the best like accelerator for you to run this kind of thing in this use case because it has the massive parallel programming model you can run like thousands of threads and how about we map those thousands of threads into thousands of EVM instances running parallel so running transactions in parallel first we focus on the niche fuzzing use case, but then along the way when we implement our project, we were approached by different teams, startups, they all building cool stuff, and some of them use some of our source code to fork out and have their own private repo with their own amazing use case in the ecosystem. But this talk will be mainly about fuzzing. I will have one slide about the other use case in the end. So what is fuzzing? Just a very simple high-level understanding. So fuzzing, the main purpose is to find bugs in software. It's one of the popular techniques in software engineering. And recently, there's a trend for people to apply it to find bugs in your smart contracts as well. So this is a very high-level depiction of gray box fuzzing. Basically, you have a tool that generates a lot of inputs. Those inputs have more coverage and more complex than your unit test usually you use. So it's more complex and eventually it's an iterative process. You generate input, you execute it, and then you get feedback. So those feedback can help the further to like generate better input Eventually you can find a input that can trigger the bugs in your smart contract or in other software so the input at so if you map it into Ethereum transaction to test my contracts the inputs are the state and transaction to interact with your DApp, your smart contract. And then the feedback can be traces, branching data, or the updated state after you run the transaction. And those are up to the first tool design and implementation. And yeah. So one example on the right is how they instrument the EVM to catch bugs. I just show very simple example. Some of them are not like the add overflow. It's not very, like it's not exist anymore after Solidity 8 because they inject by code to check it before. They try to revert it before the overflow happened. So for example, if you want to catch overflow, you just instrument the EVM every time you see up-code add, you check the operand, if it's like over the 2 to the power of 2, 5, 6, then it's overflow. Or, another interesting up-code is jump-I, all of your if-else condition in your smart contracts or required statements, it will be encoded into this XAMPP I and those will be the helpful hints for the fuzzing tool to further generate better input. I show one example of the run after this. So this is fuz thing on conventional iterative process on your CPU. Then we try to map it into CPU execution. So you see, our fuzzer can generate thousands of input. Instead of iteratively one by one, now you can generate a bulk of them, like a batch of thousands of them. And then you can offload the execution of thousands of them like a batch of thousands of them and then you can offload the execution of thousands of them into GPU. So this is the main idea, the main proposal that our project is trying to achieve. So after we offload the execution to GPU, we can map one or few threads in GPU to run this transaction, simulate the EVM, and then you get the feedback back to the version on the CPU and try to update it in the node state strategy, generate better input. And yeah, that's about it. You have to work, you have to have some communication between the accelerator and the CPU side, and they all work together to find bugs in your smart contract. Okay, so we offload the EVM execution to GPU, and the EVMM execution basically you execute one transaction and then you're given the world state so this is a very high level of what is running inside. So you see nowadays like transaction interacting with smart contracts are quite complex. They have dynamic behavior. It's not just one single AVM call you They have dynamic behavior. It's not just one single EVM call, you run and you exit. It's actually every time you have a call context, and then each of them have the volatile machine states like stack memory, and they have their own size. And it's evolved over the running at the runtime. We don't know beforehand. So this is also one of the challenge when we try to implement on GPU we cannot just pre-allocate does for some of you know that stack size maximum stack size and maximum column depth if you pre-allocate everything is this gigabyte so and we don't need to do that at the beginning so that is one of the challenges. And when you have call context, and then you can create a new call context, or you can revert back or exit and update the parent state, depending on the result of the child call. So these are the high level. And when we map to the GPU threads, actually, we only consider each of the current context. So, when you enter a context, you have your own stack memory, volatile machine states, and then it will enter the execution loop. Basically, you just keep fetching new opcode and increase program counter, executing the logic. And then eventually, this context either return back, reverse exception, or create a new context. And they keep looping. And we have the decision of how many GPU threads that we can map to execute one EVM instance. And imagine we have to run thousands of them. So the decision is actually dependent on one of the libraries we use. We use a CGPN library from NVIDIA Lab to reduce the complexity of development at the moment. So they have the limitation like how many threads we can configure to compute 1U into 5.6 arithmetic operation. So imagine it's like some big library that you can use out there. And also because they have some limitations, so we currently have a plan to develop our own library for this. They are developed for very general kind of use cases. You can configure the bit width, but then for our use case, you only need to use 256 Uint. So yeah, we can optimize a bit more on this. So this is how you map the EVM to CPU thread. And after we implement mapping using those libraries, and you need to ensure it's correct. And for this one, when we developed this one, we didn't know much about the tools in the ecosystem. So thanks for the advice from Ethereum Foundation, we actually need to implement and compare with other EVMs as well. So we integrate with GoEVM Lab, which is the tool developed by core developers of Ethereum, and that is available there. And we make sure our trades are compatible with all the EVMs as well, then we compare line by line. So we use EIB 3155, make sure it's output correct. And then we compare, and then we use ATX tests to compare. And then currently we pass on the tests in functional folders, the important one. And the one I highlighted here are the ones that are most time-consuming to us, recombined contracts and zero-knowledge. They are just a few recombined contracts, but the logic and the time we spend on them is quite a lot. So, yeah. EVM, of course, is simpler than this. And some of the things we can use open source, but some of them we have to develop on our own, especially easy pairing. Actually, we copied from the Python library from Ethereum, so we write it in CUDA based on the logic there. So after all the tests, the rare corner cases remain, and mainly gas difference. So if your phasing use case is not reliant, the logic is not reliant on gas termination or some of the logic, then you can actually use it correctly. So we are still trying to fix all of the remaining gas difference. So after we are satisfied with the correctness, then we think about optimization. So because time is limited, our developing time, also the time of this talk, so I just pick one of the optim try to optimize the normal transaction that you usually see out there in the real block, in the real transaction. What are the popular, of course, what are the things that are critical to performance, what are the things that you have to execute all the time, so we optimize them first. Basically, we check the statistics. And then we found, OK, it's quite intuitive. It's quite obvious that the stack operations is almost everywhere. You have to optimize the stack first. And then after that, we also check the stats of the stack size. What is the normal stack size that usually transactions use? And what's the stack size, what is the normal stack size that usually transactions use, and what's the memory size, and the program counter, it goes up to 8,000. So we found that, okay, we cannot cache the program counter, it's too much. Sorry, cache the bytecode, 8,000 bytes is too much. But then stack size 20-something, we can allocate the fixed size stack on fast memory. That's what we did. So we should pre-allocate that one first. Very fast stack on share mem. And then after your stack is bigger than that, you can use a slower memory as well. So most of the transactions you use this, you don't use a lot, you don't use a lot you don't use last stack that if we run very fast that's one of the main optimization that we're trying to do we try to optimize for conventional common transaction you can find out there there. Okay, so current implementation, we are happy with one beta release currently. So basically, we support Shanghai. We started with Shanghai, but we couldn't catch up with new EVM hard fork. It's quite a lot of EIVM implementations. So we stopped at Shanghai first, and then executable, we output the JSON trace, and then we have two versions, GPU, GPU, and we also have the share library, so you can use it in two modes. One is you use this share library, it's open source now, and then you can use it to try on Google collaboration. The Google collab, you can use GPU there. So yeah, you can use it in two modes. You can write your Python code to interact with it. And also you can use executable. Yeah. So performance is still depending on the workload. So it requires more comprehensive benchmarking. But yeah, we compare with our own CPU version. It's faster. But one thing is that our CPU version is actually slower than the current EVM implementation out there. So it's not like directly Apple to Apple comparison. And yeah, we are still improving the performance. So this is one of the sample run in fuzzing in real, in action. So you can try Google Colab, the link in our GitHub label. And then one example is a toy example in smart contracts with some of the very obvious bug there and some of the input guarding if conditions, so to make sure that you do some real fuzzing, you make sure you get the correct input. You can run it and eventually it will output the line, the source code line, and the input that trigger the bug, like you see, input equal 400,000 or something, yeah, 40,000, so input four, five, six, seven and trigger the bug. It can show you the source code, the line of bug, this one running, the execution running in GPU. So besides detecting bugs, we also have feedback. So these are feedback feedback branching feedback. One example is this function test branch. It's like there's a condition for input to be 1 million, and you see the current input and the distance before you can break that branch. Those are the gray box fuzzer technique that we implement in here. Those are quite common in gray box fuzzing tools out there. And our current release, the bottleneck, still remains in the way we prepare transactions and we get back the serialized data to update the state of our fuzzing tool. So it's still slow. But we have experimental branch where we remove all of the complex logic. And of course this branch does not conform with the yellow paper anymore. But then it will reach very high throughput. It can reach 60K transactions per second and improving. And we saw some teams have private repo, and some of them clone or fork from our repo with private implementation and optimization. They can reach 100,000 transactions per second as well. But we are open source, and we're trying to make sure it's still confirmed with the Ethereum standard. And beyond fuzzing, this is some of the interesting use case where we talk with other teams, other teams doing cool stuff out there, but they are doing the layer 2 and also fuzzing. But the main thing when we were thinking about parallelized transactions for Ethereum execution client, is it possible? It sounds cool, but then it's not very practical at the moment because you need more transactions. Currently, how many? Hundreds? You need thousands to have, to exploit that parallelism in GPU and also the transaction they are all different they are not very similar so it's not easy to actually speed up on GPU and also to achieve this kind of thing you need more client support like the GPU, you cannot just get that terabyte of workstations. It's not possible in the minute. It's too big to run. And also because of memory content, and also the client needs to ensure it's safe and correct to run those transaction parallel, which it's not. Because transaction, you need to run sequentially one by one after the other, and eventually to get the final state. This one runs in parallel. You have to make sure that all the nodes running it reach the same deterministic output so they can reach consensus. Otherwise, it's not possible at all. So that's what we think at the moment. So we still can find other use cases besides trying to speed up Ethereum execution client. We see other layer 2 teams also can try to have their own execution engine. Also, transaction simulation platform. I saw people use it to simulate swap, for example. And imagine you have a lot of similar transaction testing simulations. And this one is a use case for it. Yeah. You can simulate millions of swaps in a short time, serving a lot of users. Okay, so the last two slides, I want to talk about our team and collaborators. So yeah, working with me, also we have a small team of three researchers and engineers. Working with me is Chung Lee and we are from the Singapore Blockchain Innovation Program, NUS. Also working with me was Dan. And we worked in Singapore before, and he went back to Romania to become professor. And, yeah, we want to thank Frederick and Ethereum Foundation for the advices and the grant support, especially the advices to make sure it's correct and how to use all of the new tools in the ecosystem that we were not really familiar with. Thank you. Yeah, so that's all about this one. And you can find out on the GitHub. And yeah, just feel free to create PR open issues, these questions, and reach out to me. I will try to help. Thank you, Minh. And thank you for this amazing presentation. issues, these questions, and reach out to me. I will try to help. Thank you, Minh. And thank you for this amazing presentation. So I have a few questions. Let's go through them. Let's take the first one. How much work would it take to integrate this in an open source fuzzer like Echidna? You can use it. Currently, we released the shared library. You can use it right away. But the thing is you need to modify your fuzz. Our example toy fuzz, we have that bottleneck to repair thousands of transactions before you send to GPU. That one is quite a big bottleneck. You run it on GPU, it's fast, but then you prepare for it, and you run, you collect the result. It's still quite bad. So currently, we need time to improve the interaction between the library and your fuzzing tool. So it works, but you need to work on piping. Yeah, the interaction, yeah. Okay, on plumbing. All right. Can QVM plug into a simulated node tool like Anvil or EVE Tester? Yeah, so in the end, we need to make sure it's possible, but we need to make sure it's compatible for API calls. So basically, we provide that library. If you have your own adapter, your requirements, so you just send us the state and the transaction you want to run, but make sure the state is not so big. We can manageable, like transfer it to CPU and keep it there. Sorry, GPU memory and keep it there. And it's possible to transfer the state transaction, run and return it back the result, whatever result format that you want. Fantastic. Thank you want. Fantastic. Thank you. All right. Third question. Would fuzzers have to change to adapt to your GPU acceleration? It seems like this is a different version of the same question. Let's mark it as answered. All right. Fourth question. How does the GPU thread get the state from one executing S load? Well, that's specific. Okay. Do you want to take the next one? We have the data structure that resides on global memory. For the state and memory, we have to keep it in the slow memory. We don't do any caching. That one, we keep the structure pointing to the state and we search through that. Basically, if the state is big, so this is one implementation detail. We didn't implement very fast searching in the states. Basically, we have to go through on the array of accounts. So if the state is so big, currently it's quite slow. But yeah, that's how it works. So everything on global memory, we have to search through them. Fantastic. Thank you. All right. We still have 10 seconds, so maybe a more playful question. Have you considered calling it CuteVM other than 3VM? I don't know. But, you know, suggestions, you know? Yeah, yeah. We may change it to this theme. Wonderful. I mean, thank you for your time and all the work you obviously put in your slide. Looking forward to your work. Yeah, thank you.

https://streameth-develop.ams3.digitaloceanspaces.com/transcriptions/6736d06874749a4b891b95de.vtt