forked from bitcoinafterlife/bal-server
refactor: remove bal-pusher-enhanced and update docs/env
- Remove src/bin/bal-pusher-enhanced.rs (synchronous pusher variant) - Remove bal-pusher.env and bal-pusher.sh from git tracking (now in .gitignore) - Update all documentation to remove references to bal-pusher-enhanced: * 01_project_overview.md * 02_glossary_and_bitcoin_domain.md * 03_architecture_and_data_flow.md * 04_modules_detail.md * 05_api_reference.md (remove rawblock ZMQ section, update references) * 08_security_audit.md (remove references to bal-pusher-enhanced in DoS and ZMQ sections) * 09_references_and_links.md - Build verified: cargo check passes for bal-pusher and bal-server binaries
This commit is contained in:
@@ -1,16 +0,0 @@
|
||||
RUST_LOG=info
|
||||
|
||||
BAL_PUSHER_DB_FILE=/home/bal/bal.db
|
||||
BAL_PUSHER_BITCOIN_COOKIE_FILE=/home/bitcoin/.bitcoin/.cookie
|
||||
BAL_PUSHER_REGTEST_COOKIE_FILE=/home/bitcoin/.bitcoin/regtest/.cookie
|
||||
BAL_PUSHER_TESTNET_COOKIE_FILE=/home/bitcoin/.bitcoin/testnet3/.cookie
|
||||
BAL_PUSHER_SIGNET_COOKIE_FILE=/home/bitcoin/.bitcoin/signet/.cookie
|
||||
|
||||
BAL_PUSHER_ZMQ_LISTENER=tcp://127.0.0.1:28332
|
||||
|
||||
BAL_PUSHER_SEND_STATS=true
|
||||
WELIST_SERVER_URL=http://welist.bitcoin-after.life
|
||||
SSL_KEY_PATH=/home/bal/privkey.pem
|
||||
|
||||
#your server domain. do not final / only domain.
|
||||
BAL_SERVER_URL="https://we.bitcoin-after.life"
|
||||
@@ -1,14 +0,0 @@
|
||||
RUST_LOG=trace
|
||||
|
||||
export BAL_PUSHER_DB_FILE="$(pwd)/bal.db"
|
||||
#export BAL_PUSHER_BITCOIN_COOKIE_FILE=/~/.bitcoin/.cookie
|
||||
#export BAL_PUSHER_REGTEST_COOKIE_FILE=/~/.bitcoin/regtest/.cookie
|
||||
#export BAL_PUSHER_TESTNET_COOKIE_FILE=/~/.bitcoin/testnet3/.cookie
|
||||
#export BAL_PUSHER_SIGNET_COOKIE_FILE=/~/.bitcoin/signet/.cookie
|
||||
|
||||
export BAL_PUSHER_REGTEST_ZMQ_HASHBLOCK=tcp://127.0.0.1:21332
|
||||
export BAL_PUSHER_SEND_STATS=true
|
||||
export WELIST_SERVER_URL=http://localhost:8085
|
||||
export BAL_SERVER_URL="http://127.0.0.1:9133"
|
||||
export SSL_KEY_PATH="$(pwd)/private_key.pem"
|
||||
cargo run --bin=bal-pusher regtest
|
||||
@@ -21,7 +21,6 @@ The project consists of three primary binaries and two shared libraries:
|
||||
|
||||
1. **`bal-server`**: Async HTTP server (hyper + tokio) that exposes the API for receiving transactions and serving statistics.
|
||||
2. **`bal-pusher`**: Async daemon that listens for `hashblock` ZMQ messages and pushes pending transactions to a Bitcoin node via RPC.
|
||||
3. **`bal-pusher-enhanced`**: Synchronous variant of the pusher that listens for `rawblock` ZMQ messages and computes the block median time from the raw header without RPC calls.
|
||||
4. **`lib.rs`**: Exports the shared modules `db` and `xpub`.
|
||||
5. **`db.rs`**: All database operations and schema creation for SQLite `0.34.0`.
|
||||
6. **`xpub.rs`**: Address derivation from xpub/zpub using BIP-84 and the `bitcoin` crate.
|
||||
|
||||
@@ -14,7 +14,7 @@ An **XPub** (Extended Public Key) is a master key that allows derivation of chil
|
||||
|
||||
## Locktime and nLockTime
|
||||
|
||||
A Bitcoin transaction can include a `nLockTime` field. If it is non-zero and below `500_000_000`, it is interpreted as a **block height** before which the transaction cannot be mined. If above, it is a **Unix timestamp**. The system evaluates whether the locktime has been met by comparing it against the blockchain's median time. See `src/bin/bal-pusher.rs` and `src/bin/bal-pusher-enhanced.rs` for the evaluation logic.
|
||||
A Bitcoin transaction can include a `nLockTime` field. If it is non-zero and below `500_000_000`, it is interpreted as a **block height** before which the transaction cannot be mined. If above, it is a **Unix timestamp**. The system evaluates whether the locktime has been met by comparing it against the blockchain's median time. See `src/bin/bal-pusher.rs` for the evaluation logic.
|
||||
|
||||
## P2WPKH (Pay to Witness Public Key Hash)
|
||||
|
||||
@@ -22,13 +22,13 @@ P2WPKH is a native SegWit output format that reduces transaction size and lowers
|
||||
|
||||
## Bitcoin Block Header (80 bytes)
|
||||
|
||||
A Bitcoin block header is a fixed 80-byte structure containing `version` (4 bytes), `previous block hash` (32 bytes), `merkle root` (32 bytes), `timestamp` (4 bytes), `bits` (4 bytes), and `nonce` (4 bytes). The `bal-pusher-enhanced` binary reads the raw 80-byte block from the `rawblock` ZMQ topic and extracts the timestamp from byte offset 68-72. This avoids the need for an RPC call to `getblockchaininfo`. See `src/bin/bal-pusher-enhanced.rs`.
|
||||
A Bitcoin block header is a fixed 80-byte structure containing `version` (4 bytes), `previous block hash` (32 bytes), `merkle root` (32 bytes), `timestamp` (4 bytes), `bits` (4 bytes), and `nonce` (4 bytes). The `bal-pusher` binary uses the `hashblock` ZMQ topic and calls `getblockchaininfo` to get the `mediantime`.
|
||||
|
||||
## ZMQ Publisher
|
||||
|
||||
Bitcoin Core can publish notifications over ZeroMQ. The project listens to two topics:
|
||||
- **`hashblock`**: Sends the 32-byte block hash when a new block is found. The `bal-pusher` uses this to trigger an update cycle.
|
||||
- **`rawblock`**: Sends the full raw block (including the 80-byte header). The `bal-pusher-enhanced` uses this to derive the timestamp without RPC.
|
||||
- **`rawblock`**: Sends the full raw block (including the 80-byte header). This is not used by the current `bal-pusher` implementation.
|
||||
|
||||
The ZMQ endpoint is per-network:
|
||||
- Bitcoin: `tcp://127.0.0.1:28332`
|
||||
@@ -41,7 +41,7 @@ The ZMQ endpoint is per-network:
|
||||
|
||||
The project communicates with a local Bitcoin Core node via the JSON-RPC interface. Key methods used are:
|
||||
- `sendrawtransaction`: To broadcast a pending transaction.
|
||||
- `getblockchaininfo`: To retrieve the current block height and `mediantime` (used to evaluate locktime). `Note:` `bal-pusher-enhanced` does not use this method to avoid RPC round-trips for median time.
|
||||
- `getblockchaininfo`: To retrieve the current block height and `mediantime` (used to evaluate locktime).
|
||||
- `getblock`: To retrieve block data in `bal-pusher` (for median time calculation).
|
||||
|
||||
Authentication is done via `bitcoincore-rpc` using either `UserPass` or `CookieFile` (the `cookie` file is stored in `~/.bitcoin/.cookie`). See `src/bin/bal-pusher.rs`.
|
||||
|
||||
@@ -27,9 +27,6 @@ User
|
||||
| bal-pusher | (async, ZMQ + RPC + reqwest)
|
||||
| (src/bin/bal-pusher.rs)
|
||||
+-----------------+
|
||||
+-----------------+
|
||||
| bal-pusher-enhanced | (sync, ZMQ + raw header parsing)
|
||||
| (src/bin/bal-pusher-enhanced.rs)
|
||||
+-----------------+
|
||||
| bitcoincore-rpc
|
||||
| sendrawtransaction
|
||||
|
||||
@@ -126,22 +126,4 @@ This is the async daemon that monitors the blockchain and pushes pending transac
|
||||
- `welist_url`: The URL to POST to.
|
||||
- `ssl_key_path`: The path to the Ed25519 private key (`privkey.pem`) for signing stats.
|
||||
|
||||
---
|
||||
|
||||
## `bal-pusher-enhanced.rs` (Synchronous Transaction Pusher)
|
||||
|
||||
**Location:** `src/bin/bal-pusher-enhanced.rs`
|
||||
|
||||
This is a synchronous variant of the pusher that does not rely on the RPC for getting the `mediantime`.
|
||||
|
||||
### Architecture (Synchronous)
|
||||
- **ZMQ:** It uses `zmq::Context` with a `SUB` socket but does not use `zmq` in an async context. It calls `recv_multipart(0)` in a blocking loop (`std::thread::sleep`).
|
||||
- **Topic:** `rawblock` (not `hashblock`).
|
||||
- **Block Header:** It extracts the first 80 bytes (the header) from the raw block. The `timestamp` field is at byte offset 4 + 32 + 32 = 68, and is 4 bytes long (little-endian). It uses `byteorder` to read this. This avoids the `getblockchaininfo` RPC call.
|
||||
- **Block Median Time:** It computes the rolling median time from the timestamps of the last 1000 blocks.
|
||||
- **Preload:** It fetches and sorts the pending transactions from the database at startup, keeping them in memory. This reduces the database round trip.
|
||||
- **RPC:** `sendrawtransaction` is used for the pending transactions, but not for `getblockchaininfo`.
|
||||
|
||||
### Design Notes
|
||||
- The ZMQ socket is blocking and has no timeout. If the Bitcoin node stops sending, the thread will hang indefinitely. The sleep between attempts is `std::thread::sleep(Duration::from_secs(1))`, but this happens *after* a successful `recv_multipart`, not if `recv` blocks. This is a potential DoS vector if the ZMQ connection goes silent.
|
||||
- The `main_result` function is not async and does not use a `tokio` runtime.
|
||||
---
|
||||
@@ -107,7 +107,7 @@
|
||||
|
||||
---
|
||||
|
||||
## ZMQ Messages (consumed by `bal-pusher` and `bal-pusher-enhanced`)
|
||||
## ZMQ Messages (consumed by `bal-pusher`)
|
||||
|
||||
### Topic: `hashblock` (Consumed by `bal-pusher`)
|
||||
- **Format:** A multipart ZMQ message. The first frame is the topic name (`hashblock`), the second frame is the 32-byte block hash.
|
||||
@@ -115,16 +115,9 @@
|
||||
- **Action:** The pusher fetches `getblockchaininfo` from the RPC, gets the updated `mediantime`, then queries and pushes pending transactions.
|
||||
- **Endpoint:** `tcp://127.0.0.1:28332` (or network-specific ports).
|
||||
|
||||
### Topic: `rawblock` (Consumed by `bal-pusher-enhanced`)
|
||||
- **Format:** A multipart ZMQ message. The first frame is the topic name (`rawblock`), the second frame is the raw serialized block data. The first `80` bytes of this second frame are the block header, in which bytes `[68..72]` are the timestamp (little-endian `uint32_t`).
|
||||
- **Trigger:** When a new Bitcoin block is found by the local node.
|
||||
- **Action:** The pusher extracts the block header, reads the timestamp from it, computes the rolling median of the last 11 block timestamps, then evaluates and pushes pending transactions.
|
||||
- **Endpoint:** `tcp://127.0.0.1:28332` (or network-specific ports).
|
||||
- **Note:** This topic is much more bandwidth-intensive than `hashblock` because the entire block is sent over the wire.
|
||||
|
||||
---
|
||||
|
||||
## Bitcoin Core RPC Usage (used by `bal-pusher` and `bal-pusher-enhanced`)
|
||||
## Bitcoin Core RPC Usage (used by `bal-pusher`)
|
||||
|
||||
### `sendrawtransaction` (Both pushers)
|
||||
- **Method:** `sendrawtransaction` (RPC `2`)
|
||||
@@ -136,7 +129,7 @@
|
||||
- **Method:** `getblockchaininfo` (RPC `1`)
|
||||
- **Parameters:** None.
|
||||
- **Description:** Returns the current blockchain state, including the `mediantime` (the median timestamp of the last 11 blocks). This is used to evaluate the `nLockTime` of pending transactions.
|
||||
- **Alternative:** `bal-pusher-enhanced` does not use this method; it derives the block timestamp directly from the `rawblock` ZMQ message to avoid an RPC round-trip and a potential RPC dependency failure.
|
||||
|
||||
|
||||
### `getblock` (Only used by `bal-pusher` for median time)
|
||||
- **Method:** `getblock` (RPC `1`)
|
||||
|
||||
@@ -55,11 +55,11 @@
|
||||
**Status:** Open. **Priority:** High.
|
||||
|
||||
### 4. Denial of Service (DoS) (HIGH)
|
||||
**Location:** `src/bin/bal-server.rs` (HTTP request body), `src/bin/bal-pusher.rs` (ZMQ), `src/bin/bal-pusher-enhanced.rs` (ZMQ).
|
||||
**Location:** `src/bin/bal-server.rs` (HTTP request body), `src/bin/bal-pusher.rs` (ZMQ).
|
||||
**Description:**
|
||||
- The `bal-server` does not limit the size of the HTTP request body. On the `POST /pushtxs` endpoint, it calls `req.collect().await?.to_bytes()` without checking for a maximum body size. A malicious client could send an unbounded or extremely large request (e.g., `100000MB`), which would consume all available memory and crash the server.
|
||||
- The `bal-server` regex for path matching might be expensive if the user provides a malicious path string. For a production system, the regex should be compiled only once at startup and should be very specific.
|
||||
- The `bal-pusher` `recv` call is synchronous and blocking. If the ZMQ connection fails, the thread will hang without any timeout. This is a resource leak if the connection is broken. The `bal-pusher-enhanced` has `recv_multipart(0)` which is also blocking forever. If the Bitcoin Core node is not sending, the pusher will be stuck waiting forever, consuming a thread and not doing other useful work. The ZMQ socket is not reconfigured with `ZMQ_RECONNECT_IVL` or `ZMQ_MAXMSGSIZE`.
|
||||
- The `bal-pusher` `recv` call is synchronous and blocking. If the ZMQ connection fails, the thread will hang without any timeout. This is a resource leak if the connection is broken. The ZMQ socket is not reconfigured with `ZMQ_RECONNECT_IVL` or `ZMQ_MAXMSGSIZE`. If the Bitcoin Core node is not sending, the pusher will be stuck waiting forever, consuming a thread and not doing other useful work.
|
||||
- The `bal-pusher` does not have a rate limiter for the `sendrawtransaction` call. If the database is full or the ZMQ loop is running very fast, it could send thousands of RPC requests to the `bicoind` node, overwhelming it. For example, if the node is slow, the pusher will keep sending requests, potentially blocking the RPC queue or causing a memory leak in `bitcoind`.
|
||||
**Impact:** The server could become unresponsive, crash, or be completely unavailable. The `bitcoind` node could be overwhelmed with `sendrawtransaction` requests, causing a chain failure in the entire Bitcoin infrastructure.
|
||||
**Reproduction:**
|
||||
@@ -98,7 +98,7 @@
|
||||
**Status:** Open. **Priority:** Medium.
|
||||
|
||||
### 7. ZMQ Authentication and Encryption (MEDIUM)
|
||||
**Location:** `src/bin/bal-pusher.rs`, `src/bin/bal-pusher-enhanced.rs`.
|
||||
**Location:** `src/bin/bal-pusher.rs`.
|
||||
**Description:** The ZMQ connection to the `bitcoind` is a plaintext TCP connection (`zmqpubhashblock=tcp://127.0.0.1:28332`). There is no ZMQ authentication (ZAP), no username/password, and no encryption (ZMQ_CURVE or ZMQ_GSSAPI). If the ZMQ port is accessible from the network (not just `127.0.0.1`), any attacker can subscribe to the `hashblock` or `rawblock` topics. The `rawblock` topic is particularly sensitive because it sends full block data, which is large and could be used to fingerprint the `bal` system. More importantly, the pusher does not verify that the `hashblock` is from the intended `bitcoind` node. If an attacker can inject a fake ZMQ message, they could trigger the pusher to evaluate the transactions and potentially broadcast them at an incorrect time, or cause a DoS.
|
||||
**Impact:**
|
||||
- If the ZMQ port is exposed, an attacker can intercept the `rawblock` data to get the full block contents, which could be used to fingerprint the node or the system.
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
| XPub/Address Derivation | `src/xpub.rs` | `parse_xpub`, `derive_address`, `get_descriptor`, BIP-84 |
|
||||
| HTTP Server | `src/bin/bal-server.rs` | Hyper + Tokio, routes, handlers, `pushtxs` logic |
|
||||
| Async Pusher | `src/bin/bal-pusher.rs` | ZMQ `hashblock`, RPC + Reqwest, `send_stats` |
|
||||
| Synchronous Pusher Enhanced | `src/bin/bal-pusher-enhanced.rs` | ZMQ `rawblock`, raw block header parsing, no `getblockchaininfo` |
|
||||
|
||||
| Stats (broken) | `src/bin/bal-stats.rs.dontcompile` | Not compiled, incomplete HTML report generator |
|
||||
| Release script | `make_release.sh` | Hardcoded token, `cargo install` |
|
||||
| DB download script | `download_bal_db.sh` | `scp` from remote |
|
||||
@@ -25,8 +25,7 @@
|
||||
| Utility scripts | `lib.sh` | Colored echo functions |
|
||||
| Contrib (install) | `contrib/download_and_install_bal.sh` | Nginx, Certbot, systemd setup, hardcoded xpub |
|
||||
| Contrib (install bitcoind) | `contrib/download_and_install_bitcoincore.sh` | Bitcoind download, GPG verify, systemd, config |
|
||||
| Contrib (install Tor) | `contrib/install_tor.sh` | Tor repository, `ControlPort 9051` |
|
||||
| Systemd service | `bal-server.service` | Runs as `bal` user, `ProtectSystem`, `MemoryDenyWriteExecute` |
|
||||
| Contrib (install Tor) | `contrib/install_tor.sh` | Tor repository, `ControlPort 9051` || Systemd service | `bal-server.service` | Runs as `bal` user, `ProtectSystem`, `MemoryDenyWriteExecute` |
|
||||
| Systemd service | `bitcoind.service` | `zmqpubhashblock` setup |
|
||||
| Systemd service | `tbitcoind.service` | Testnet `bitcoind` |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user