forked from bitcoinafterlife/bal-server
svatantrya
eacb2e1450
- Fix framework references (actix-web, not hyper) - Update all env var names (BAL_SERVER_*/BAL_PUSHER_* prefix) - Add validation.rs module documentation - Fix function signatures in xpub.rs and db.rs - Update API response formats (InfoResponse, StatsResponse) - Fix database schema (date_creation/date_update, push_err, tbl_stats) - Mark fixed vulnerabilities with current status - Add Docker support and actix tuning documentation - Remove outdated references (confy, bal-stats.rs.dontcompile) - Add regression test summary table
205 lines
10 KiB
Markdown
205 lines
10 KiB
Markdown
# Database Schema
|
|
|
|
## Quick Reference
|
|
- **What this file contains:** the full SQL schema, data types, indexing, queries, and data lifecycle for the SQLite `bal.db` database.
|
|
- **See also:** [04_modules_detail.md](04_modules_detail.md), [05_api_reference.md](05_api_reference.md), [03_architecture_and_data_flow.md](03_architecture_and_data_flow.md)
|
|
|
|
---
|
|
|
|
## Database Technology
|
|
- **Engine:** `sqlite` (Rust `sqlite` crate, version 0.34.0)
|
|
- **File:** `bal.db` (default, configurable via `BAL_SERVER_DB_FILE` / `BAL_PUSHER_DB_FILE`)
|
|
- **Connection Management:** Shared `Arc<Mutex<Connection>>` in `bal-server`, single connection in `bal-pusher`.
|
|
- **WAL Mode:** Enabled via `PRAGMA journal_mode=WAL` with retry logic (up to 5 attempts) for concurrent access safety.
|
|
- **Busy Timeout:** Set to 5000ms via `PRAGMA busy_timeout=5000`.
|
|
- **Synchronous Mode:** Set to `NORMAL` via `PRAGMA synchronous=NORMAL`.
|
|
- **Path Validation:** The `open_db` function validates the database path before opening, rejecting directory traversal (`..`), forbidden system directories (`/etc`, `/proc`, `/sys`, `/dev`, `/usr`, `/bin`, `/sbin`, `/lib`, `/opt`), and symlinks.
|
|
|
|
---
|
|
|
|
## Table Schema
|
|
|
|
### `tbl_tx` (Transactions)
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS tbl_tx (
|
|
txid PRIMARY KEY, -- TEXT: The unique transaction ID (hex string)
|
|
date_creation TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
date_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
wtxid, -- TEXT: The witness transaction ID
|
|
ntxid, -- TEXT: The non-witness transaction ID
|
|
tx, -- TEXT: The full raw serialized transaction (hex)
|
|
locktime INTEGER, -- INTEGER: The locktime value (block height or timestamp)
|
|
network, -- TEXT: The network name (e.g., 'regtest', 'bitcoin')
|
|
network_fees, -- TEXT: The total fees paid by the user (satoshi)
|
|
reqid, -- TEXT: A request ID or client IP for the submitter
|
|
our_fees, -- TEXT: The fees paid to us (the operator) (satoshi)
|
|
our_address, -- TEXT: The address the operator fee is paid to
|
|
status INTEGER DEFAULT 0, -- INTEGER: 0 = waiting, 1 = sent, 2 = failed
|
|
push_err TEXT -- TEXT: The error message if the RPC broadcast failed
|
|
);
|
|
ALTER TABLE tbl_tx ADD COLUMN push_err TEXT;
|
|
```
|
|
- **Indexes:** The `txid` is the primary key, so it is automatically indexed.
|
|
- **Notes:** `date_creation` and `date_update` track when the transaction was inserted and last modified. `locktime` is compared against the blockchain's best block height or `mediantime` (for timestamps above `LOCKTIME_THRESHOLD`). The `status` column is the core of the transaction lifecycle state machine. `push_err` stores the RPC error message when status=2.
|
|
|
|
### `tbl_inp` (Transaction Inputs)
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS tbl_inp (
|
|
id, -- INTEGER: Auto-increment ID
|
|
txid, -- TEXT: The transaction ID of the transaction being submitted
|
|
in_txid, -- TEXT: The previous transaction ID (output being spent)
|
|
in_vout -- INTEGER: The previous output index
|
|
);
|
|
CREATE UNIQUE INDEX ON tbl_inp(txid, in_txid, in_vout);
|
|
```
|
|
- **Purpose:** Tracks all inputs of submitted transactions. Enables identification of double-spends.
|
|
- **Constraints:** A unique index prevents duplicate entries for the same input in the same transaction.
|
|
|
|
### `tbl_out` (Transaction Outputs)
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS tbl_out (
|
|
id, -- INTEGER: Auto-increment ID
|
|
txid, -- TEXT: The transaction ID of the transaction being submitted
|
|
script_pubkey, -- TEXT: The hex scriptPubKey of this output
|
|
amount, -- TEXT: The amount in this output (satoshi)
|
|
vout -- INTEGER: The output index (0-based) in this transaction
|
|
);
|
|
CREATE UNIQUE INDEX ON tbl_out(txid, script_pubkey, amount, vout);
|
|
```
|
|
- **Purpose:** Tracks all outputs of submitted transactions. The server searches for the `script_pubkey` matching the `our_address` for the network to verify fee payment.
|
|
- **Constraints:** A unique index prevents duplicate entries for the same output in the same transaction.
|
|
|
|
### `tbl_xpub` (Extended Public Keys)
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS tbl_xpub (
|
|
id INTEGER PRIMARY KEY, -- INTEGER: Auto-increment ID
|
|
network TEXT, -- TEXT: The network name (e.g., 'regtest', 'bitcoin')
|
|
xpub TEXT, -- TEXT: The extended public key (xpub, zpub, ypub, etc.)
|
|
date_create TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
path_idx INTEGER DEFAULT -1 -- INTEGER: The next address index to derive for this xpub
|
|
);
|
|
CREATE UNIQUE INDEX idx_xpub ON tbl_xpub (network, xpub);
|
|
```
|
|
- **Purpose:** Stores the master xpub/zpub/ypub keys for each network. When the server receives a transaction in xpub mode, it derives new receiving addresses from these keys.
|
|
- **Relationships:** `tbl_xpub` is linked to `tbl_address` via `id`. The `path_idx` tracks which child index is the next unused one for the wallet's derivation path.
|
|
|
|
### `tbl_address` (Derived Addresses)
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS tbl_address (
|
|
address TEXT PRIMARY_KEY, -- TEXT: The Bech32 P2WPKH address (e.g., 'bcrt1q...')
|
|
path TEXT NOT NULL, -- TEXT: The derivation path (e.g., 'm/0/0')
|
|
date_create TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
xpub INTEGER, -- INTEGER: The ID of the xpub in `tbl_xpub`
|
|
remote_address TEXT -- TEXT: IP or client identifier that requested this address
|
|
);
|
|
```
|
|
- **Purpose:** Stores all generated addresses. In xpub mode, addresses are derived on-demand per requesting IP.
|
|
- **Relationships:** `xpub` (FK) links to `tbl_xpub.id`. `remote_address` is used for rate-limiting and preventing address reuse per IP.
|
|
|
|
### `tbl_stats` (Per-Network Statistics)
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS tbl_stats (
|
|
report_date TEXT, -- TEXT: The ISO timestamp of the report
|
|
chain TEXT, -- TEXT: The network name (e.g., 'regtest', 'bitcoin')
|
|
totals INTEGER, -- INTEGER: Total number of transactions submitted
|
|
waiting INTEGER, -- INTEGER: Transactions currently waiting (status=0)
|
|
sent INTEGER, -- INTEGER: Transactions successfully sent (status=1)
|
|
failed INTEGER, -- INTEGER: Transactions that failed to broadcast (status=2)
|
|
waiting_profit INTEGER, -- INTEGER: Total fees for waiting transactions (satoshi)
|
|
sent_profit INTEGER, -- INTEGER: Total fees for sent transactions (satoshi)
|
|
missed_profit INTEGER, -- INTEGER: Total fees for failed/expired transactions (satoshi)
|
|
unique_inputs INTEGER -- INTEGER: The number of unique inputs (for deduplication analysis)
|
|
);
|
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_stats_chain ON tbl_stats(chain);
|
|
```
|
|
- **Purpose:** Stores aggregate statistics for each network. The pusher calculates and upserts stats (`ON CONFLICT(chain) DO UPDATE`). The server reads from it for the `stats` endpoint if `expose_stats` is enabled.
|
|
- **Relationships:** `chain` has a unique index. Data is updated by `bal-pusher` via `calculate_stats`.
|
|
|
|
---
|
|
|
|
## Data Query Strategy
|
|
|
|
### Key Queries
|
|
|
|
- **Get Pending Transactions (by status and locktime):**
|
|
```sql
|
|
SELECT * FROM tbl_tx
|
|
WHERE network = :network
|
|
AND status = :status
|
|
AND (locktime < :bestblock_height
|
|
OR locktime > :locktime_threshold AND locktime < :bestblock_time);
|
|
```
|
|
Used by the `bal-pusher` to find transactions ready to broadcast. The `locktime_threshold` constant distinguishes block heights from timestamps.
|
|
|
|
- **Insert Transaction (batched):**
|
|
```sql
|
|
INSERT INTO tbl_tx (txid, wtxid, ntxid, tx, locktime, network, network_fees, reqid, our_fees, our_address)
|
|
UNION ALL SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
|
|
UNION ALL SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
|
|
-- ... more rows
|
|
```
|
|
Used by `bal-server` for efficient bulk inserts.
|
|
|
|
- **Update Status:**
|
|
```sql
|
|
UPDATE tbl_tx SET status = ? WHERE txid = ?;
|
|
```
|
|
Used by the pusher after broadcast. Status `1` = sent, `2` = failed.
|
|
```sql
|
|
UPDATE tbl_tx SET status = 2, push_err = ? WHERE txid = ?;
|
|
```
|
|
For failed broadcasts, the error message is stored.
|
|
|
|
- **Check Duplicate Txids:**
|
|
```sql
|
|
SELECT txid FROM tbl_tx WHERE txid IN (?, ?, ?, ...);
|
|
```
|
|
Used by `bal-server` to batch-check duplicates before inserting. Chunks in groups of 500 for SQLite parameter limit safety.
|
|
|
|
- **Search Transaction:**
|
|
```sql
|
|
SELECT * FROM tbl_tx WHERE txid = ?;
|
|
```
|
|
Used by the `searchtx` endpoint.
|
|
|
|
- **Get All Addresses by XPub:**
|
|
```sql
|
|
SELECT a.address
|
|
FROM tbl_address a
|
|
JOIN tbl_xpub x ON a.xpub = x.id
|
|
WHERE x.xpub = ?;
|
|
```
|
|
Used to load all known addresses for an xpub in a single query, enabling O(1) fee validation in the push handler.
|
|
|
|
- **Get Last Used Address by IP:**
|
|
```sql
|
|
SELECT address FROM tbl_address
|
|
WHERE remote_address = ? AND xpub = ?
|
|
ORDER BY date_create DESC LIMIT 1;
|
|
```
|
|
Used to check if an IP already has a generated address (address reuse prevention).
|
|
|
|
- **Get Next Address Index:**
|
|
```sql
|
|
UPDATE tbl_xpub SET path_idx = path_idx + 1
|
|
WHERE network = ? AND xpub = ?
|
|
RETURNING id, path_idx;
|
|
```
|
|
Atomically increments the derivation index and returns the new value.
|
|
|
|
---
|
|
|
|
## Data Lifecycle
|
|
- **Creation:** Transactions are created when a user submits raw hex tx via `POST /{network}/pushtxs`. Addresses are derived on-demand in xpub mode.
|
|
- **Waiting:** Transactions are in `status=0` and are queried by the pusher every new block.
|
|
- **Broadcast:** Transactions are pushed via `sendrawtransaction`. If successful, status becomes `1`. If the RPC returns an error, status becomes `2` and the error is stored in `push_err`.
|
|
- **Statistics:** The pusher aggregates stats from the database and upserts into `tbl_stats` with `ON CONFLICT(chain) DO UPDATE`.
|
|
- **Retention:** There is no explicit cleanup mechanism for old records. For production, a periodic vacuum or purge of old `status=1` transactions may be required.
|
|
- **Backup:** The database is a single SQLite file. It can be copied directly using `cp` or `rsync`. WAL mode ensures consistency during copies.
|