February 25, 2024

ChatGPT is phenomenal at discovering bugs with out requiring any significant further context past the supply code. This makes it a wonderful first place to start out your testing journey.

One of many first belongings you be taught as a sensible contract coder is that contracts are extraordinarily unforgiving with regards to vulnerabilities. Contracts are immutable by default. On the similar time, they’re able to dealing with extraordinary quantities of cash. This makes safety (and sensible contract testing) arguably the largest concern for any workforce of web3 builders. 

However auditing contracts—the everyday final of line protection towards bugs—has traditionally been time-consuming and costly. Happily, there was an explosion in highly effective instruments that make safety and sensible contract testing cheaper, easier, and sooner.

On this article, we are going to discover testing sensible contracts utilizing two of those instruments: ChatGPT by OpenAI and Diligence Fuzzing by ConsenSys.

Testing an Ethereum Good Contract with ChatGPT and Diligence Fuzzing

Let’s stroll by way of a sensible contract for an Ethereum undertaking that options susceptible ERC-20 code and see what the 2 instruments can do for us. 

Diligence Fuzzing is a instrument from ConsenSys that implements fuzzing for web3. Fuzzing is a dynamic testing method the place random (or semi-random) inputs known as “fuzz” are generated and injected into code. Fuzzing might help reveal bugs and vulnerabilities that weren’t caught by conventional testing strategies.

And naturally, ChatGPT is a big language mannequin (LLM) AI. Except you’ve been dwelling beneath a rock, you’re in all probability already conscious of ChatGPT and a few of its capabilities. It’s one of the crucial thrilling applied sciences to have come out within the final decade. Not solely can it show you how to with testing (as we’ll see on this article), but additionally with sensible contract improvement, writing assessments, and extra.

Let’s get began!

Step 1: Set up Python, pip, npm, and Node.js

With the intention to set up the instruments required to run Diligence Fuzzing, you first want the next in your native machine: the most recent variations of Python and pip and the most recent variations of Node.js and npm

Examine that you’ve got all 4 working through the use of these instructions:

$ node -v
$ npm -v
$ python –-version
$ pip --version

Step 2: Set up the Susceptible Contract Repository

Along with creating Diligence Fuzzing and the Scribble annotation language, ConsenSys has made public a set of repositories containing susceptible contracts. To maintain it easy, you’ll be utilizing considered one of these.

You’ll use a repo that comprises a susceptible implementation of the ERC-20 normal (the usual for cryptocurrencies and fungible tokens).

Clone this repository utilizing the next command:

$ git clone https://github.com/ConsenSys/scribble-exercise-1.git contracts-test

This repository has a folder known as contracts. Inside this folder, one can find a file known as vulnerableERC20.sol. As talked about earlier, this file comprises a defective implementation of the ERC-20 normal.

That is the contract you’re going to check utilizing ChatGPT and later, Diligence Fuzzing.

Step 3: Create an OpenAI Account

ChatGPT occurs to be extraordinarily proficient at all kinds of duties—together with web3 improvement. (See my earlier article on creating a sensible contract with ChatGPT.) 

You possibly can ask ChatGPT to unpack a posh blockchain idea, write sensible contracts that implement one thing you had in thoughts (as an example, a decentralized lottery), and even clarify a chunk of code to you and discover potential vulnerabilities.

As you could have already guessed, it’s that final functionality we’re excited about for this text.

With the intention to entry ChatGPT, it’s worthwhile to create an OpenAI account. When you’ve created an account, access ChatGPT’s UI.

Speaking to ChatGPT is so simple as speaking to an individual on a messaging app. If that is your first time utilizing the instrument, I strongly advocate you check out some instance prompts (for instance, “Clarify quantum computing in easy phrases”) to get comfy.

Step 4: Discover Vulnerabilities within the Contract 

The important thing to getting proper solutions from ChatGPT is being extraordinarily clear and detailed about your request. Paste the next immediate into the UI:

Following is a brilliant contract I've written in Solidity. As a primary step in testing its security, I need you to look at the code, and let me know of all the safety vulnerabilities that you simply discover.

```
pragma solidity ^0.6.0;

contract VulnerableToken 
  uint256 non-public _totalSupply;
  mapping (deal with => uint256) non-public _balances;
  mapping (deal with => mapping (deal with => uint256)) non-public _allowances;

  constructor() public 
    _totalSupply = 1000000;
    _balances[msg.sender] = 1000000;
  

  perform totalSupply() exterior view returns (uint256) 
    return _totalSupply;
  

  perform balanceOf(deal with _owner) exterior view returns (uint256) 
    return _balances[_owner];
  

  perform allowance(deal with _owner, deal with _spender) exterior view returns (uint256) 
    return _allowances[_owner][_spender];
  

  perform switch(deal with _to, uint256 _value) exterior returns (bool) 
    deal with from = msg.sender;
    require(_value <= _balances[from]);


    uint256 newBalanceFrom = _balances[from] - _value;
    uint256 newBalanceTo = _balances[_to] + _value;
    _balances[from] = newBalanceFrom;
    _balances[_to] = newBalanceTo;

    emit Switch(msg.sender, _to, _value);
    return true;
  

  perform approve(deal with _spender, uint256 _value) exterior returns (bool) 
    deal with proprietor = msg.sender;
    _allowances[owner][_spender] = _value;
    emit Approval(proprietor, _spender, _value);
    return true;
  

  perform transferFrom(deal with _from, deal with _to, uint256 _value) exterior returns (bool) 
    uint256 allowed = _allowances[_from][msg.sender];
    require(_value <= allowed);
    require(_value <= _balances[_from]);
    _balances[_from] -= _value;
    _balances[_to] += _value;
    _allowances[_from][msg.sender] -= _value;
    emit Switch(_from, _to, _value);
    return true;
  

  occasion Switch(deal with listed _from, deal with listed _to, uint256 _value);
  occasion Approval(deal with listed _owner, deal with listed _spender, uint256 _value);

```

ChatGPT ought to provide you with a response that appears one thing like this:

Upon reviewing the code, I've recognized a number of safety vulnerabilities and 
considerations within the offered sensible contract:

1. Lack of Entry Management: The contract doesn't implement any entry management 
mechanism, permitting anybody to name the `switch`, `approve`, and `transferFrom` 
features. Which means anybody can switch tokens on behalf of one other 
deal with and modify allowances.

2. Integer Underflow and Overflow: The code makes use of primary arithmetic operations 
with none checks for potential underflow or overflow. For instance, within the 
`switch` and `transferFrom` features, the subtraction and addition of token 
balances can result in unintended conduct if the ensuing stability exceeds the 
most worth that may be saved in a `uint256`. It's best to embody checks to 
forestall these points.

3. Lack of Require Statements for Enter Validation: The contract lacks require 
statements to validate inputs in numerous features. As an example, within the 
`switch` and `transferFrom` features, there isn't a validation to make sure 
that the recipient deal with (`_to`) isn't the zero deal with. Moreover, 
the `approve` perform doesn't validate that the spender deal with (`_spender`) 
isn't the zero deal with.

4. Potential Reentrancy Assaults: The contract is inclined to reentrancy 
assaults, the place a malicious contract may repeatedly name the `transferFrom` 
perform to empty the token stability of one other deal with. It is suggested to 
use the "Checks-Results-Interactions" sample or implement a reentrancy guard 
to mitigate this vulnerability.

5. Lack of Occasions for Vital State Adjustments: Whereas the contract emits 
`Switch` and `Approval` occasions, it's lacking occasions for important state 
modifications, equivalent to when the whole provide is modified or token balances are 
up to date. Emitting occasions for these occasions helps present transparency and 
allows simpler monitoring of contract actions.

6. Mounted Token Provide: The contract initializes with a hard and fast token provide 
of 1,000,000 tokens, and there aren't any provisions for minting or burning 
tokens. For those who intend to have a hard and fast provide, this will not be a vulnerability. 
Nonetheless, in the event you plan to have a dynamic token provide, you may have to 
implement minting and burning features whereas contemplating correct entry 
management.

To reinforce the safety of your sensible contract, take into account addressing these 
vulnerabilities and implementing further safety measures primarily based in your 
particular necessities.

ChatGPT provides you fairly a couple of suggestions! Whereas they’re largely generic, they’ll nonetheless show you how to uncover some key points. 

Beware, although—in a couple of instances, such because the tip on Entry Management, it’s plain mistaken. And imagine it or not, ChatGPT really finally ends up lacking probably the most evident error in our contract— one thing that has to do with transferring tokens to ourselves. That is a particularly nuanced error that ChatGPT would solely be capable of choose up given further prompting. (Be aware: with sufficient prompting, I used to be in a position to get ChatGPT to search out the error.)

The important thing lesson right here is that ChatGPT can help in debugging, however solely as a place to begin. No less than in the meanwhile, it’s greatest at figuring out errors which are syntactical, apparent, or frequent. This could be most useful for starting or intermediate builders—or possibly for drained and overworked senior ones! For extra nuanced instances, you’ll want to show to extra superior instruments.

Step 5: Signal Up for Diligence Fuzzing

The train above exhibits that, at the very least on the time of writing (June 2023), ChatGPT can do lots—however isn’t a whole substitute for a well-versed web3 developer, particularly one who can leverage state-of-the-art testing instruments.

On this step, you’re going to be taught to make use of one such instrument: Diligence Fuzzing.

As talked about above, fuzzing is a well-established testing method that entails sending random, usually malformed information to a specific program in an try to set off sudden behaviors or crashes. This technique has proved exceptionally good at detecting safety points that conventional testers usually miss.

Diligence Fuzzing implements conventional fuzzing in a web3 surroundings. The instrument has been created to detect important vulnerabilities and unthought-of points utilizing easy annotations in Scribble.

With the intention to use this instrument, it’s worthwhile to first sign up for a free account with Diligence Fuzzing. The Free Tier subscription is greater than sufficient for this tutorial, but when you end up doing numerous important testing, take into account transferring to a paid tier.

When you’ve subscribed, you need to see a dashboard that appears one thing like this.

Step 6: Create a Diligence Fuzzing API Key

With the intention to entry Diligence Fuzzing’s capabilities, it’s worthwhile to create an API key here.

You possibly can identify your key something you need. Maintain this key secret and secure. You’ll want it later.

Step 7: Set up Required Packages and Libraries

First, you’re going to put in Truffle, Scribble, and the Fuzzing CLI in your native machine. Run the next instructions:

$ cd contracts-test
$ npm set up
$ npm set up -g truffle eth-scribble ganache
$ pip3 set up diligence-fuzzing

Step 8: Create a Fuzzing Marketing campaign

Navigate to the contracts-test folder you cloned earlier. Open this repository in your favourite code editor (for instance, VS Code) and open the file known as .fuzz.yml. It’s best to see a line that claims key which has an empty string related to it. Exchange the latter along with your API key.

For Diligence Fuzzing to work appropriately, it’s worthwhile to annotate our contract with guidelines you suppose ought to be adopted. As talked about earlier, the annotation language right here is Scribble.

For this tutorial, you don’t have to know how one can write Scribble your self (although you need to take a couple of moments to get aware of it for future initiatives). The repository you downloaded comes (by default) with an annotated contract within the contracts/resolution folder. You’ll be utilizing this as a base to annotate the susceptible contract.

Exchange the contents of vulnerableERC20.sol with the next:

pragma solidity ^0.6.0;

contract VulnerableToken 
  uint256 non-public _totalSupply;
  mapping (deal with => uint256) non-public _balances;
  mapping (deal with => mapping (deal with => uint256)) non-public _allowances;

  constructor() public 
    _totalSupply = 1000000;
    _balances[msg.sender] = 1000000;
  

  perform totalSupply() exterior view returns (uint256) 
    return _totalSupply;
  

  perform balanceOf(deal with _owner) exterior view returns (uint256) 
    return _balances[_owner];
  

  perform allowance(deal with _owner, deal with _spender) exterior view returns (uint256) 
    return _allowances[_owner][_spender];
  

  /// #if_succeeds msg.sender != _to ==> _balances[_to] == previous(_balances[_to]) + _value;
  /// #if_succeeds msg.sender != _to ==> _balances[msg.sender] == previous(_balances[msg.sender]) - _value;
  /// #if_succeeds msg.sender == _to ==> _balances[msg.sender] == previous(_balances[_to]);
  /// #if_succeeds previous(_balances[msg.sender]) >= _value;
  perform switch(deal with _to, uint256 _value) exterior returns (bool) 
    deal with from = msg.sender;
    require(_value <= _balances[from]);

    // _balances[from] -= _value;
    // _balances[_to] += _value;

    uint256 newBalanceFrom = _balances[from] - _value;
    uint256 newBalanceTo = _balances[_to] + _value;
    _balances[from] = newBalanceFrom;
    _balances[_to] = newBalanceTo;

    emit Switch(msg.sender, _to, _value);
    return true;
  

  perform approve(deal with _spender, uint256 _value) exterior returns (bool) 
    deal with proprietor = msg.sender;
    _allowances[owner][_spender] = _value;
    emit Approval(proprietor, _spender, _value);
    return true;
  

  perform transferFrom(deal with _from, deal with _to, uint256 _value) exterior returns (bool) 
    uint256 allowed = _allowances[_from][msg.sender];
    require(_value <= allowed);
    require(_value <= _balances[_from]);
    _balances[_from] -= _value;
    _balances[_to] += _value;
    _allowances[_from][msg.sender] -= _value;
    emit Switch(_from, _to, _value);
    return true;
  

  occasion Switch(deal with listed _from, deal with listed _to, uint256 _value);
  occasion Approval(deal with listed _owner, deal with listed _spender, uint256 _value);

The feedback in traces 25-28 signify annotations written in Scribble. Even in the event you’ve by no means been uncovered to this language earlier than, the contents ought to be fairly self-explanatory.

You’re all set. Begin fuzzing by working the next command:

make fuzz

This could immediate your terminal to run a collection of instructions.

Discover that the output additionally comprises a URL that gives extra details about your marketing campaign.

This marketing campaign takes roughly 5 minutes to run. However as soon as it’s achieved, you get an in depth report about its findings, together with vulnerabilities.

Discover that beneath the Property part, the report states that one consumer assertion has failed. Upon nearer inspection, you see that it’s this one:

/// #if_succeeds msg.sender == _to ==> _balances[msg.sender] == previous(_balances[_to]);

You possibly can take a look at the transaction particulars to search out out extra.

In essence, it’s telling you there’s a vulnerability when the sender and the receiver are the identical! 

Diligence Fuzzing does an awesome job at a selected activity—it mechanically creates all kinds of inputs to generate unit assessments and system assessments that discover bugs and vulnerabilities in your sensible contract. 

Create a ChatGPT and Diligence Plugins

You possibly can take this even additional by combining the ability of the 2 testing instruments utilizing a ChatGPT plugin. This plugin may:

  • Execute the code to initialize a Diligence Fuzzing marketing campaign towards the contract

  • Parse the outcomes and use them to do one thing attention-grabbing: straight rewrite the code, provide you with strategies for enhancements, mechanically rewrite and retest the code till there aren’t any discovered defects, and extra

You should utilize this starting point to create the plugin—or within the spirit of this text, even ask ChatGPT to write down the plugin for/with you!

Conclusion

On this tutorial, you explored two very highly effective instruments for sensible contract testing and safety audits.

ChatGPT is phenomenal at discovering bugs with out requiring any significant further context past the supply code. This makes it a wonderful first place to start out your testing journey.

Nonetheless, as you’ve seen, ChatGPT tends to go off observe and sometimes produces mistaken solutions. Instead for a whole audit, it has a protracted technique to go. That is the place a instrument like Diligence Fuzzing can shine. It might require just a little extra handbook intervention, nevertheless it’s way more complete when it comes to protection.

Have a very nice day!