bitcoinafterlife/bal-server
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

docs: add comprehensive knowledge base and security audit

Browse Source 
- Add docs/INDEX.md with navigable index and quick reference guides
- Add 9 knowledge base files covering project overview, Bitcoin domain,
  architecture, modules, API reference, database schema, deployment/security
- Update AGENTS.md with knowledge base reference and update policy
- Add tests/sql_injection_tests.rs with regression tests for SQL injection
- Fix SQL injection vulnerabilities in bal-pusher.rs:
  * Replace string-formatted UPDATE IN with loop + parameterized queries
  * Replace string-formatted UPDATE push_err with parameterized query
  * Add chain name validation in calculate_stats to prevent env var tampering
- Update .gitignore to exclude bal-pusher.env and bal-pusher.sh
This commit is contained in:
svatantrya
2026-07-16 14:11:18 -04:00
parent 64f402eeae
commit 69d877a360
14 changed files with 1328 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

174
docs/06_database_schema.md Normal file
View File

@@ -0,0 +1,174 @@
# Database Schema
## Quick Reference
- **What this file contains:** the full SQL schema, data types, indexing, queries, and data lifecycle for the SQLite `bal.db` database.
- **See also:** [04_modules_detail.md](04_modules_detail.md), [05_api_reference.md](05_api_reference.md), [03_architecture_and_data_flow.md](03_architecture_and_data_flow.md)
---
## Database Technology
- **Engine:** `sqlite` (Rust `sqlite` crate, version 0.34.0)
- **File:** `bal.db` (default, configured in environment)
- **Connection Pooling:** The Rust `sqlite` crate handles connections but does not use a thread pool.
- **Transactions:** The `execute_insert` function attempts to use atomic transactions for batched inserts, but this is not guaranteed for all operations.
---
## Table Schema
### `tbl_tx` (Transactions)
```sql
CREATE TABLE tbl_tx (
txid PRIMARY KEY, -- TEXT: The unique transaction ID (hex string)
wtxid, -- TEXT: The witness transaction ID
ntxid, -- TEXT: The non-witness transaction ID
tx, -- TEXT: The full raw serialized transaction (hex)
locktime INTEGER, -- INTEGER: The locktime value (block height or timestamp)
network, -- TEXT: The network name (e.g., 'regtest', 'bitcoin')
network_fees, -- TEXT: The total fees paid by the user (satoshi)
reqid, -- TEXT: A request ID or client IP for the submitter
our_fees, -- TEXT: The fees paid to us (the operator) (satoshi)
our_address, -- TEXT: The address the operator fee is paid to
status INTEGER DEFAULT 0, -- INTEGER: 0 = waiting, 1 = sent, 2 = failed
push_err TEXT -- TEXT: The error message if the RPC broadcast failed
);
```
- **Indexes:** The `txid` is the primary key, so it is automatically indexed.
- **Notes:** `locktime` is stored as an integer. It is compared against the `mediantime` or `block_height` from the blockchain to evaluate when a transaction is ready to send. The `status` column is the core of the transaction lifecycle state machine. `our_fees` and `our_address` are used to validate the transaction and ensure the correct fee is included before accepting it.
### `tbl_inp` (Transaction Inputs)
```sql
CREATE TABLE tbl_inp (
id, -- INTEGER: Auto-increment ID
txid, -- TEXT: The transaction ID of the transaction being submitted
in_txid, -- TEXT: The previous transaction ID (output being spent)
in_vout -- INTEGER: The previous output index
);
CREATE UNIQUE INDEX ON tbl_inp(txid, in_txid, in_vout);
```
- **Purpose:** Tracks all inputs of the submitted transactions. This allows the database to identify double-spends and ensure the inputs are valid and available.
- **Constraints:** A unique index prevents duplicate entries for the same input in the same transaction.
- **Relationships:** `in_txid` and `in_vout` refer to outputs from previous transactions in the Bitcoin blockchain. The `txid` column refers to the transaction being submitted (the one in `tbl_tx`).
### `tbl_out` (Transaction Outputs)
```sql
CREATE TABLE tbl_out (
id, -- INTEGER: Auto-increment ID
txid, -- TEXT: The transaction ID of the transaction being submitted
script_pubkey, -- TEXT: The hex scriptPubKey of this output
amount, -- TEXT: The amount in this output (satoshi)
vout -- INTEGER: The output index (0-based) in this transaction
);
CREATE UNIQUE INDEX ON tbl_out(txid, script_pubkey, amount, vout);
```
- **Purpose:** Tracks all outputs of the submitted transactions. The server searches for the `script_pubkey` matching the `our_address` for the network to determine if the correct fee is included.
- **Constraints:** A unique index prevents duplicate entries for the same output in the same transaction.
- **Relationships:** `txid` refers to the transaction being submitted. The `script_pubkey` is matched against the known addresses for each network to verify the fee payment.
### `tbl_xpub` (Extended Public Keys)
```sql
CREATE TABLE tbl_xpub (
id INTEGER PRIMARY KEY, -- INTEGER: Auto-increment ID
network TEXT, -- TEXT: The network name (e.g., 'regtest', 'bitcoin')
xpub TEXT, -- TEXT: The extended public key (xpub or zpub)
date_create TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- TEXT: The date the xpub was added
path_idx INTEGER DEFAULT -1 -- INTEGER: The next address index to derive for this xpub
);
CREATE UNIQUE INDEX idx_xpub ON tbl_xpub (network, xpub);
```
- **Purpose:** Stores the master xpub/zpub keys for each network. When the server receives a transaction, it uses these to derive new receiving addresses (if applicable) or to verify the `our_address`.
- **Relationships:** `tbl_xpub` is linked to `tbl_address` via `xpub` (the ID). The `path_idx` tracks which child index is the next unused one for the wallet's derivation path.
### `tbl_address` (Derived Addresses)
```sql
CREATE TABLE tbl_address (
address TEXT PRIMARY KEY, -- TEXT: The Bech32 P2WPKH address (e.g., 'bcrt1q...')
path TEXT NOT NULL, -- TEXT: The derivation path used to create this address (e.g., 'm/84'/1'/0'/0/0')
date_create TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- TEXT: The date the address was generated
xpub INTEGER, -- INTEGER: The ID of the xpub in `tbl_xpub` that owns this address
remote_address TEXT -- TEXT: IP or client identifier that requested this address (if applicable)
);
```
- **Purpose:** Stores all generated addresses. The `our_address` for each network is derived from a specific xpub path. The server can also generate new addresses on demand for clients.
- **Relationships:** `xpub` (FK) links to `tbl_xpub.id`. The `address` is the primary key because it is unique by design. `remote_address` is used for rate-limiting or identifying address ownership in logs.
### `tbl_stats` (Per-Network Statistics)
```sql
CREATE TABLE tbl_stats (
report_date INTEGER, -- INTEGER: The Unix timestamp of the report
chain TEXT PRIMARY KEY, -- TEXT: The network name (e.g., 'regtest', 'bitcoin')
totals INTEGER, -- INTEGER: Total number of transactions submitted
waiting INTEGER, -- INTEGER: Transactions currently waiting (status=0)
sent INTEGER, -- INTEGER: Transactions successfully sent (status=1)
failed INTEGER, -- INTEGER: Transactions that failed to broadcast (status=2)
waiting_profit INTEGER, -- INTEGER: Total fees for waiting transactions (satoshi)
sent_profit INTEGER, -- INTEGER: Total fees for sent transactions (satoshi)
missed_profit INTEGER, -- INTEGER: Total fees for transactions that expired or failed (satoshi)
unique_inputs INTEGER -- INTEGER: The number of unique inputs (for deduplication analysis)
);
```
- **Purpose:** Stores aggregate statistics for each network. The pusher sends this data to a remote `welist` server. The server also reads from it for the `stats` endpoint if `expose_stats` is enabled.
- **Relationships:** `chain` is the primary key. The data is updated by the `bal-pusher` binary.
---
## Data Query Strategy
### Key Queries (from `db.rs` and `bal-pusher.rs`)
- **Get Pending Transactions (by status and locktime):**
```sql
SELECT
txid, tx, wtxid, ntxid, locktime, status,
our_address, our_fees, network_fees
FROM
tbl_tx
WHERE
network = ?
AND status = 0
AND locktime < ?;
```
Used by the `bal-pusher` daemon to find transactions that are ready to broadcast. The `?` placeholders are bound at runtime. `locktime` is compared with the `mediantime` or block height from the ZMQ `new block` event.
- **Insert Transaction:**
```sql
INSERT INTO tbl_tx (txid, wtxid, ntxid, tx, locktime, network, network_fees, reqid, our_fees, our_address)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
```
Used by the `bal-server` when accepting a new valid transaction.
- **Update Status:**
```sql
UPDATE tbl_tx SET status = ? WHERE txid = ?;
```
Used by the pusher after a successful or failed broadcast attempt. The status is set to `1` (sent) and `2` (failed).
**WARNING:** The `bal-pusher` also uses a raw `WHERE txid IN ('...')` format for batch updates. These string formats have **SQL injection risk** because the `txid` strings are concatenated into the raw SQL string without proper parameterization.
- **Search Transaction:**
```sql
SELECT * FROM tbl_tx WHERE txid = ?;
```
Used by the `searchtx` endpoint.
- **Get Address for Rate Limiting:**
```sql
SELECT a.address, x.xpub
FROM tbl_address a
JOIN tbl_xpub x ON a.xpub = x.id
WHERE a.remote_address = ?;
```
Used to check if an IP or client address already has a generated address. This is part of the address reuse logic to prevent users from requesting too many addresses or using the same IP to bypass fees.
---
## Data Lifecycle
- **Creation:** Transactions are created when a user submits a raw hex tx via the `POST /pushtxs` endpoint. The address is inserted when an xpub is configured.
- **Waiting:** Transactions are in `status=0` and are queried by the pusher every new block.
- **Broadcast:** Transactions are pushed to the network via `sendrawtransaction`. If successful, the status becomes `1`. If the RPC returns an error (e.g., `-25`), the status becomes `2` and the error string is stored in `push_err`.
- **Retention:** There is no explicit cleanup mechanism for old records. The `valid_txs` and `invalid_txs` files contain logs of past transaction pushes, but the database itself may grow indefinitely. For a production system, a periodic vacuum or purge of old `status=1` transactions might be required.
- **Backup:** The database is a single SQLite file (`bal.db`). It can be copied directly using `cp` or `rsync` (see `scripts/download_bal_db.sh`). There is no WAL mode or online backup mechanism implemented.