> ## Documentation Index
> Fetch the complete documentation index at: https://seilabs-docs-evm-cookbook.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Detailed guide for troubleshooting node problems.

export const RandomPeers = ({format = 'bash', network = 'mainnet', count = 5}) => {
  const PEERS = {
    mainnet: ['d53ab7681ed0df3d3249fc0132df7c3a131c9c1a@45.250.253.40:51656', '4ee8aea5bb58e9038d72e74322e3bba755287398@202.8.8.183:11956', '81409623ae4da3ec7b400cf640dea0b0999a964b@57.128.230.96:51556', '61a5be64b5215786fe5da712584678d0626636b5@87.249.137.71:51556', 'f83c536f43df9a5d900cd3c2f702c04d7dddc7b5@136.243.67.45:11956', 'b8d600a2f568576b5a2df5ee649cf9b809389064@162.19.62.176:26756', '57860b18ed3e1bbe8901ba73f2e63c7e6fe8b3d3@57.129.54.81:26656', '04fc6bba6c5c33034811612dca31c7adda24b299@91.134.60.37:16856', 'de64b779c7f4091e6f1765f5ca4c46f9d3011732@65.108.70.106:46656', '3be6b24cf86a5938cce7d48f44fb6598465a9924@p2p.state-sync.pacific-1.seinetwork.io:26656', '70e0c91b83b5ed1beaca798267f2debdf97dac10@18.156.6.83:26656', 'dd6b1ae002a15c1c8a38e05660f49a93c75d4159@148.251.181.225:26656'],
    testnet: ['71beea83970431f55816eee5f066a611a1dc80f7@p2p.state-sync.atlantic-2.seinetwork.io:26656', '65c257f9275beb1b99ca169ef89743c034b15db0@3.76.192.224:26656', '33588592e477c5238ff2a5d8dc765f85790ef853@23.109.47.225:26656', 'babc3f3f7804933265ec9c40ad94f4da8e9e0017@testnet-seed.rhinostake.com:11956', '8542cd7e6bf9d260fef543bc49e59be5a3fa9074@seed.publicnode.com:56656']
  };
  const pickRandom = (arr, n) => {
    const shuffled = [...arr];
    for (let i = shuffled.length - 1; i > 0; i--) {
      const j = Math.floor(Math.random() * (i + 1));
      [shuffled[i], shuffled[j]] = [shuffled[j], shuffled[i]];
    }
    return shuffled.slice(0, Math.min(n, arr.length));
  };
  const CopyIcon = ({className}) => <svg role="img" aria-label="Copy" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M7 7m0 2.667a2.667 2.667 0 0 1 2.667 -2.667h8.666a2.667 2.667 0 0 1 2.667 2.667v8.666a2.667 2.667 0 0 1 -2.667 2.667h-8.666a2.667 2.667 0 0 1 -2.667 -2.667z" />
      <path d="M4.012 16.737a2.005 2.005 0 0 1 -1.012 -1.737v-10c0 -1.1 .9 -2 2 -2h10c.75 0 1.158 .385 1.5 1" />
    </svg>;
  const CheckIcon = ({className}) => <svg role="img" aria-label="Copied" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M5 12l5 5l10 -10" />
    </svg>;
  const ShuffleIcon = ({className}) => <svg role="img" aria-label="Shuffle" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
      <path d="M16 3h5v5" />
      <path d="M4 20l16.2 -16.2" />
      <path d="M21 16v5h-5" />
      <path d="M15 15l6 6" />
      <path d="M4 4l5 5" />
    </svg>;
  const pool = PEERS[network] ?? [];
  const [peers, setPeers] = useState(() => pool.slice(0, count));
  const [copied, setCopied] = useState(false);
  const shuffle = () => {
    setPeers(pickRandom(pool, count));
  };
  useEffect(() => {
    shuffle();
  }, [network, count]);
  if (pool.length === 0) {
    return <div className="not-prose w-full">
        <pre className="m-0 p-3 rounded-md bg-neutral-100 dark:bg-neutral-800 text-sm opacity-70" style={{
      fontFamily: 'var(--sei-font-mono)'
    }}>
          No peers configured for network "{network}".
        </pre>
      </div>;
  }
  const peerString = peers.join(',');
  const displayText = format === 'toml' ? `persistent_peers = "${peerString}"` : `PEERS="${peerString}"`;
  const handleCopy = async () => {
    try {
      await navigator.clipboard.writeText(displayText);
      setCopied(true);
      setTimeout(() => setCopied(false), 2000);
    } catch {}
  };
  return <div className="not-prose w-full">
      <div className="relative">
        <pre className="m-0 px-4 py-3 pr-28 rounded-lg bg-neutral-100 dark:bg-neutral-900 text-[12.5px] leading-[1.55] whitespace-pre-wrap break-all border border-neutral-200 dark:border-neutral-700 text-neutral-900 dark:text-neutral-100" style={{
    fontFamily: 'var(--sei-font-mono)'
  }}>
          <code style={{
    fontFamily: 'var(--sei-font-mono)'
  }}>{displayText}</code>
        </pre>
        <div className="absolute top-2 right-2 flex items-center gap-1.5">
          <button type="button" onClick={shuffle} aria-label="Shuffle peers" title="Shuffle" className="inline-flex items-center gap-1 px-2 py-1 rounded-md text-[11px] font-semibold border border-neutral-300 dark:border-neutral-600 bg-white/80 dark:bg-neutral-800/80 text-neutral-700 dark:text-neutral-200 hover:bg-white dark:hover:bg-neutral-800 transition-colors">
            <ShuffleIcon />
            Shuffle
          </button>
          <button type="button" onClick={handleCopy} aria-label="Copy peers" title="Copy to clipboard" className="inline-flex items-center gap-1 px-2 py-1 rounded-md text-[11px] font-semibold border border-neutral-300 dark:border-neutral-600 bg-white/80 dark:bg-neutral-800/80 text-neutral-700 dark:text-neutral-200 hover:bg-white dark:hover:bg-neutral-800 transition-colors">
            {copied ? <>
                <CheckIcon className="text-green-600 dark:text-green-400" />
                Copied
              </> : <>
                <CopyIcon />
                Copy
              </>}
          </button>
        </div>
      </div>
    </div>;
};

Knowing common errors and their solutions helps keep your node healthy.

## Common error codes

These are the most frequent errors and their solutions:

### Consensus errors

If you get a consensus error, act quickly and appropriately:

```text theme={null}
Error: "Consensus failure - height halted"
Solution: Check for network upgrades or chain halts
Command: seid status
```

```text theme={null}
Error: "Private validator file not found"
Solution: Restore validator key or check file permissions
Location: $HOME/.sei/config/priv_validator_key.json
```

```text theme={null}
Error: "Duplicate signature"
Solution: IMMEDIATELY STOP NODE - potential double signing risk
Action: Check validator operation on other machines
```

### Network errors

Network errors can prevent your node from participating in consensus:

```text theme={null}
Error: "Dial tcp connection refused"
Solution: Check network connectivity and firewall rules
Commands:
  - netstat -tulpn | grep seid
  - ufw status
```

```text theme={null}
Error: "No peers available"
Solution: Verify peer connections and network config
Commands:

- curl localhost:26657/net_info
```

### Database errors

Database corruption can require immediate attention:

```text theme={null}
Error: "Database is corrupted"
Solution: Reset database or restore from backup
Commands:
  - seid tendermint unsafe-reset-all
  - cp -r backup/data $HOME/.sei/
```

### Diagnostic commands

These commands help you investigate problems and monitor your node:

```bash theme={null}
# Check node synchronization
seid status

# Check validator status
seid query staking validator $(seid tendermint show-validator)

# Monitor real-time logs
journalctl -fu seid -o cat

# View system resource usage
top -p $(pgrep seid)
```

## AppHash mismatch errors

If you get an AppHash mismatch, capture the state to compare it with a known-good version:

```bash theme={null}
# For SeiDB (all supported nodes):
git clone https://github.com/sei-protocol/sei-db.git
cd sei-db/tools
make install
systemctl stop seid
seidb dump-iavl -d $HOME/.sei/data/committer.db -o /home/ubuntu/iavl-dump
systemctl restart seid
```

<Note>As with FlatKV, the memIAVL store has two possible locations. Nodes created before the layout change keep the legacy `$HOME/.sei/data/committer.db` path shown above (it takes precedence when present). New nodes use `$HOME/.sei/data/state_commit/memiavl`. Point `-d` at whichever path exists on your node.</Note>

On Giga Storage nodes, EVM state lives in a FlatKV store instead of the memIAVL trees. An AppHash comparison there also requires a FlatKV state dump. Use the `dump-flatkv` command to iterate and dump the physical `(key, value)` pairs into per-bucket files. The files match the `dump-iavl` format, so the same diff tooling works on both:

```bash theme={null}
# For FlatKV (Giga Storage nodes hold EVM state in FlatKV):
systemctl stop seid
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump
systemctl restart seid
```

<Note>Nodes created before the storage layout change keep the FlatKV store at the legacy path `$HOME/.sei/data/flatkv` instead of `$HOME/.sei/data/state_commit/flatkv`. Pass whichever directory exists on your node to `--db-dir`.</Note>

The `dump-flatkv` command accepts these flags:

* `--db-dir` (`-d`): The FlatKV database directory.
* `--output-dir` (`-o`): The output directory, with one file for each bucket.
* `--height`: The FlatKV target version. The default, `0`, selects the latest available version.
* `--bucket` (`-b`): Restrict the dump to a single bucket (`account`, `code`, `storage`, or `legacy`). The default is all buckets.

For example, to dump only the `storage` bucket at a specific version:

```bash theme={null}
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump --height 12345678 --bucket storage
```

### Comparing EVM state between memIAVL and FlatKV

When you debug an AppHash mismatch that involves EVM state, a byte-for-byte physical dump can diverge between backends, even when the underlying state is identical. This happens because every FlatKV value embeds a per-key block-height stamp (the height at which the key was last written or migrated). On a freshly migrated node, this stamp differs from the memIAVL leaf versions.

The `evm-logical-digest` command works around this with a backend-independent digest of the EVM *logical* state. On both sides, it strips the serialization-version and block-height header. Then it digests only the logical payload: account balance, nonce, and code hash, plus bytecode and storage words. With this digest, you can compare a memIAVL node and a FlatKV node at the same chain height:

```bash theme={null}
# FlatKV digest at a height (WAL-replays to it):
seidb evm-logical-digest --backend flatkv \
    --db-dir $HOME/.sei/data/state_commit/flatkv --height 213200000

# memIAVL digest at the same height (0 = current symlink):
seidb evm-logical-digest --backend memiavl \
    --db-dir $HOME/.sei/data/state_commit/memiavl --height 213200000
```

Each run prints per-bucket `bucket_digest` values and a single `FINAL_DIGEST` line that covers the `account`, `code`, `storage`, and `legacy` buckets. Compare the `FINAL_DIGEST` lines from both backends at the same height. They should match. FlatKV can contain a FlatKV-only migration-version marker that a memIAVL-only node never owns. The command automatically omits that row from the FlatKV final result, so both results cover the same data.

For targeted debugging, the command can also inspect a single normalized bucket instead of printing the global digest. The [seidb tooling section of the technical reference](/node/technical-reference#comparing-evm-state-across-backends) has the full flag reference, including inspect mode, sharding, and `--find-hash`. These examples list the first 50 `account` rows with version metadata, and shard the `storage` bucket under a key prefix by the next 2 bytes:

```bash theme={null}
seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket account --list --list-limit 50 --details

seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket storage --key-prefix 03 --shard-next-bytes 2
```

<Warning>
  **Enable SeiDB State Commit (SC).** Set `sc-enable = true` in the `[state-commit]` section of `app.toml`. The legacy IAVL backend was fully removed, and SC is now mandatory. If SC is not enabled, the node no longer falls back to IAVL. It panics at startup with this error:

  ```text theme={null}
  SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated
  ```

  The `seid debug dump-iavl` command was also removed with the IAVL backend. To inspect state, use the `seidb dump-iavl` tool shown above.
</Warning>

When you report a problem, always include the app hash, commit hash, and block height from your logs.

### Identifying AppHash errors

In logs, AppHash errors usually look like this:

```text theme={null}
ERR wrong Block.Header.AppHash. Expected [EXPECTED_HASH], got [ACTUAL_HASH]
block_id={"hash":"...","parts":{"hash":"...","total":1}} height=[HEIGHT]
```

**Common causes:**

* Using an incorrect node version during sync (make sure that you run the latest version)
* Corrupted or incorrectly applied snapshots
* Database inconsistencies from improper shutdowns
* Syncing with outdated or incompatible peers

**Resolution steps:**

1. **Stop the node immediately.**

2. **Try a node rollback first.** See [Node rollback](/node/troubleshooting#node-rollback).

3. **If the rollback fails, restore from a fresh snapshot:**

   * Download a recent snapshot from trusted providers (Polkachu, PublicNode)
   * Make sure that you use the correct node version
   * Verify that peer configurations are up to date

4. **Restart the node and monitor the logs for continued errors.**

### Peer connection issues as AppHash red herrings

**Important:** Peer connection failures are often symptoms of underlying AppHash errors, not the root cause.

If you see many peer connection errors like these:

```text theme={null}
ERR failed to handshake with peer
ERR failed to send request for peers
ERR peer handshake failed endpoint={} err=EOF
```

**Do not focus only on fixing peer connections first.** Instead:

1. **Scan your logs carefully** for AppHash errors that may appear intermittently
2. **Look for the actual error pattern:**
   ```text theme={null}
   ERR wrong Block.Header.AppHash. Expected [HASH], got [HASH]
   ```
3. **Check whether your node is stuck** at a specific height despite peer connection attempts

**Why this happens:**

* AppHash mismatches prevent proper block validation
* The node cannot advance to new blocks because of a state inconsistency
* Peers may reject connections from nodes with corrupted state
* The network appears to be the problem, but the cause is a local state issue

**Debugging approach:**

1. **First, check for AppHash errors** in your logs (search for "wrong Block.Header.AppHash")
2. **If you find AppHash errors**, treat them as the primary issue
3. **Focus on peer connection fixes** only if no AppHash errors exist

This approach targets the root cause, not the symptoms, and can save hours of debugging time.

### Peer connection and handshake issues

**Identifying peer issues:**

Look for these error patterns in your logs:

```text theme={null}
ERR failed to handshake with peer err="expected to connect with peer \"[EXPECTED_ID]\", got \"[ACTUAL_ID]\""
ERR failed to send request for peers err="no available peers to send a PEX request to (retrying)"
ERR peer handshake failed endpoint={} err=EOF module=p2p
```

**Common causes:**

* Outdated peer configurations with mismatched node IDs
* Network infrastructure changes on the peer side
* A firewall that blocks connections on port 26656
* DNS resolution issues

**Resolution steps:**

1. **Update peer configurations** with current node IDs:

   <RandomPeers network="mainnet" format="toml" />

2. **Verify network connectivity:**

   ```bash theme={null}
   # Test connection to peer endpoints
   nc -zv p2p.state-sync-0.pacific-1.seinetwork.io 26656

   # Check if port 26656 is open for inbound connections
   netstat -tulpn | grep :26656
   ```

3. **Check the current peer status:**
   ```bash theme={null}
   curl http://localhost:26657/net_info | jq '.result.peers | length'
   curl http://localhost:26657/lag_status | jq .
   ```

### Sync performance issues

**Identifying sync problems:**

Monitor these indicators:

```bash theme={null}
# Check sync status and lag
curl http://localhost:26657/lag_status | jq .

# Monitor if height is progressing
curl http://localhost:26657/status | jq '.result.sync_info'
```

**Common solutions:**

1. **Increase the packet payload size** for large block processing:

   ```toml theme={null}
   # In config.toml [p2p] section
   max-packet-msg-payload-size = 1024000  # Increase from default 102400
   ```

2. **Optimize the mempool settings** in `config.toml`:

   ```toml theme={null}
   # In [mempool] section
   keep-invalid-txs-in-cache = true
   ttl-duration = "5s"
   ttl-num-blocks = 5
   ```

3. **If the node gets stuck at a specific height:**
   * Try restarting the node
   * If that does not help, perform a rollback
   * Consider taking a fresh snapshot

**Warning signs to watch for:**

* The current height does not increase over time
* Increasing lag between the current height and the max peer height
* Repeated timeout errors in the logs
* Mempool size that consistently reaches its limits

## Crash and panic debugging

For crashes, panics, or nil pointer exceptions:

* Capture at least 1,000 lines of logs before the crash or 15 minutes of log data, whichever gives more context
* Include the full stack trace, if it is available

### Logging configuration

Proper logging configuration is essential for debugging and monitoring:

```toml theme={null}
# In config.toml
# Set appropriate log level
log_level = "debug"  # Use "trace" for maximum detail

# Choose log format
log_format = "json"  # Use "plain" for human-readable logs
```

Configure log rotation to manage storage:

```bash theme={null}
# Example logrotate configuration
sudo tee /etc/logrotate.d/seid << EOF
/var/log/seid/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 sei sei
    sharedscripts
    postrotate
        systemctl reload seid
    endscript
}
EOF
```

Enable core dumps for crash analysis:

```bash theme={null}
# Set unlimited core dump size
ulimit -c unlimited

# Configure core dump location
echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern
```

## Other common issues and fixes

1. **Sync problems**

   * Check available disk space (`df -h`)
   * Make sure that peer connections work (`curl http://localhost:26657/net_info`)
   * Check that the firewall allows port 26656

2. **Performance issues**

   * Monitor system resources (`htop` or `iotop`)
   * Check disk I/O performance (`iostat`)
   * Analyze network traffic (`iftop`)

3. **Database issues**

   * Run database integrity checks:

     ```bash theme={null}
     seid debug dump-db | grep -i error
     ```

     If you find errors, consider restoring from a recent backup.

   * To keep less historical data, lower `ss-keep-recent` in `app.toml`.

   * To rebuild the node with a smaller database, reset it. Then resync from a
     [snapshot](/node/snapshot) or with [state sync](/node/statesync). The
     reset deletes all chain data, so it is not a pruning tool. Before you
     reset, back up `priv_validator_key.json` and `priv_validator_state.json`,
     as described in [Clean up](/node/statesync#clean-up):

     ```bash theme={null}
     seid tendermint unsafe-reset-all --home $HOME/.sei --keep-addr-book
     ```

     Alternatively, remove old state snapshots manually to free disk space:

     ```bash theme={null}
     rm -rf $HOME/.sei/data/snapshots/*
     ```

## Node rollback

To roll back a node from an AppHash mismatch, first stop the node in your
preferred way.

Next, roll back the node:

```bash theme={null}
seid rollback
```

Then, restart the node.

If you see this error when you try to roll back:

```bash theme={null}
failed to initialize database: resource temporarily unavailable
```

This means that you did not shut down the node properly. In that case, try to shut down or kill the `seid` process directly. If this does not help, restart your machine.

Then try the rollback steps again.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.