Maintenance and Troubleshooting for 24/7 Ethereum Network Infrastructure

You cannot fix what you cannot see. The foundation of 24/7 maintenance is real-time observability. Most professional Ethereum setups utilize the "PGF" stack: Prometheus, Grafana, and an alerting service like Alertmanager.

Key Metrics to Track:

• Peer Count: If your peers drop below 10, your synchronization might stall.
• Disk I/O: High wait times (iowait) indicate your SSD is struggling to keep up with the state growth.
• CPU Load: Spikes in CPU usage can lead to missed attestations or delayed block proposals.
• RAM Usage: Specifically, watch for "Out of Memory" (OOM) kills if your client's cache is set too high.
• Sync Distance: The gap between your node’s head and the network’s latest block.

We recommend setting alerts for when peer_count < 20 or when disk_utilization > 85%. These early warnings give you hours—sometimes days—to perform maintenance before a critical failure occurs.

The Ethereum state grows every second. Without maintenance, your SSD will eventually fill up, causing the node to crash. Pruning is the process of removing old state data that is no longer necessary for the current operation of the node.

For Geth (Execution Client): Geth requires "offline pruning" unless you use specific configurations. This involves stopping the node and running geth snapshot prune-state. Depending on your hardware, this can take 2 to 6 hours.

For Consensus Clients: Most consensus clients (like Lighthouse) handle their database growth much more efficiently, but you should still monitor the /beacon/db folder. Ensure you are utilizing "Check-point Sync" to allow for rapid recovery if you need to delete the database and start fresh.

Pro Tip: Invest in NVMe SSDs. Standard SATA SSDs often lack the IOPS (Input/Output Operations Per Second) required to finish a pruning cycle while the network state continues to move forward.

When looking at logs (using journalctl -fu geth or similar), look for these common red flags:

• "Fatal: Failed to register the Ethereum service: database closed": This usually means a previous instance of Geth didn't shut down properly. Check for ghost processes with ps aux | grep geth.
• "Beacon node is not synced": Check the logs of your Execution client. The Consensus client cannot sync if the Execution client is still catching up.
• "Head slot is too old": This is often a sign of clock drift. Check your system time against a reliable NTP server.
• "JWT authentication failed": Ensure the jwt.hex file is shared correctly between your execution and consensus clients and that the file paths in your startup scripts are accurate.