From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Wed, 10 Dec 2025 12:52:23 -0800 Received: from mail-oa1-f59.google.com ([209.85.160.59]) by mail.fairlystable.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1vTRAb-00056B-6f for bitcoindev@gnusha.org; Wed, 10 Dec 2025 12:52:23 -0800 Received: by mail-oa1-f59.google.com with SMTP id 586e51a60fabf-3f0ee9140dbsf399824fac.1 for ; Wed, 10 Dec 2025 12:52:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1765399935; x=1766004735; darn=gnusha.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-sender:mime-version :subject:references:in-reply-to:message-id:to:from:date:sender:from :to:cc:subject:date:message-id:reply-to; bh=uBgYUqziXc3UIPORMcZvEBRwFkqLMFE8QH5s8d6thA8=; b=PaK5jEJK5l4rUQi/1sjpAaPdwlg6QlhmlsTAJxwai3+YmVgyCcCRxerO1aBXsVB9YY +6Ya2SclBn79llyHvA34ufDGCUUmD4V46UtHKDvYji/+TyeeUFwLUMi/i+KZJr70bO85 Ih6HIsJ7bIVRap6VdztUebypDQGIduGS50GPrxDJXdcVtctQCwwSGCmH1WQ1fut4OnhC 5ckHG/bVL5MbScpbl3V81Un+NOpogFPnRYJx6bA93RV/VUC4/SAQfYSCPxO8pHsh3hKM 1ZYgilcayBve81VAJiVMbDuZHFxjzD07VQv0Ia9j9rJ8GSj4pnv0FSr31FTS1NpWhRV7 ryuQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=voskuil.org; s=google; t=1765399935; x=1766004735; darn=gnusha.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-sender:mime-version :subject:references:in-reply-to:message-id:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=uBgYUqziXc3UIPORMcZvEBRwFkqLMFE8QH5s8d6thA8=; b=NXi8ympnjEHPZ+/NjDWGqvocz4MnVwODtfseCT6NtHtcXHpQsoJ7lkLrVS3ajqSBAe PqjXzeFq2R/wisauy9PG+m9cnQgQUBBpZvLAxc7KiENx4Irj16JvSBy1+6aw6vFwVYUy AE11JICrwLF9qs4HttAnHs091yv8Pc0BgHDldhrobjK8mxN/sfxutgkIWe9Hpg7RaBNV P7qgDWVmgGxu4H9ZrIv+gHQ12d8fbvKzoePUuVH8xxy7ZHYN9q8/gtTHtVmm8gdVCVB4 VJk8v9ZT4kd7sq4Qb6RIiVE6icbcKb1oNzO14Sp+NfmjbkPdiZ3HgElHs18L5ni3H68C oPnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765399935; x=1766004735; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-sender:mime-version :subject:references:in-reply-to:message-id:to:from:date:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=uBgYUqziXc3UIPORMcZvEBRwFkqLMFE8QH5s8d6thA8=; b=MeRPKKMzQ3Bn9moHxwmWKi48RGMIpa8LnKiSZ+xTAMOD9gkb5CFydJgehea3MrdD/x JesZDpdq4PpSQeN7mK9F7tOwJc22wCBZIXnDHLHXfB2XErEW8aZ51Logk0IM/P65jmOE LCObNRZhALvrxnq3p/UOnGkRv4fg2Tsfi3OeOK/cdZL9xeQsCMBe6kSdZNt/qk6TdVVz QfZzY8W6Iutyk10qQcgkIuqsgz6BtOEdzB9xBatBqVuJH14ggPYhzThwfmodGhbYw1jg m/JUgHPmLrDI10A2qaArIQqj3bSjTtywppXHvcHHx6YXJpDoqS29CmxDJufozcag6XOm 5LLA== Sender: bitcoindev@googlegroups.com X-Forwarded-Encrypted: i=1; AJvYcCVTZl5axTqNgZK4Frk9uy8rr6e/q1RIdliYrmA9u1muK3g8IX1ayldP4iWvs0m2UPAJrzjAYfGVvufc@gnusha.org X-Gm-Message-State: AOJu0YwhNPhsyg9T6HK6wlOOeGp67DNDXXwWfyZSc9Mm/wchWQunCQZ7 rP/A+7bpRBwumsQe7ZxkgggHRwTGa327M3FWsICTKYzFNmTukfD84GKD X-Google-Smtp-Source: AGHT+IEw9Hk1T+FPgd1Bp3g8iIghjN9RdfvV/zDbPHQStuRAFhrhcxg4NuiUPRSSzW+Le3lDjHWZkg== X-Received: by 2002:a05:6870:171c:b0:3e8:9cf8:53a1 with SMTP id 586e51a60fabf-3f5d5d6e6f8mr436788fac.15.1765399934437; Wed, 10 Dec 2025 12:52:14 -0800 (PST) X-BeenThere: bitcoindev@googlegroups.com; h="AWVwgWadTC69dpMnpByaEMwQhPUrN+WfGGZGn1p/lNpAS0C7Cg==" Received: by 2002:a05:6870:970b:b0:3f5:cf8c:7f65 with SMTP id 586e51a60fabf-3f5cf8c8844ls171953fac.2.-pod-prod-00-us-canary; Wed, 10 Dec 2025 12:52:10 -0800 (PST) X-Received: by 2002:a05:6808:19a4:b0:44d:aa31:d929 with SMTP id 5614622812f47-4559680d77dmr384515b6e.20.1765399929971; Wed, 10 Dec 2025 12:52:09 -0800 (PST) Received: by 2002:a05:690c:9314:b0:78a:986f:9439 with SMTP id 00721157ae682-78c5aaeb8d7ms7b3; Wed, 10 Dec 2025 12:44:35 -0800 (PST) X-Received: by 2002:a05:690e:e8c:b0:644:5166:3065 with SMTP id 956f58d0204a3-6447859d7e3mr609253d50.21.1765399474281; Wed, 10 Dec 2025 12:44:34 -0800 (PST) Date: Wed, 10 Dec 2025 12:44:33 -0800 (PST) From: Eric Voskuil To: Bitcoin Development Mailing List Message-Id: <97065ca0-662b-4a5d-90c5-7607db41ba2fn@googlegroups.com> In-Reply-To: References: <049858f4-28ba-4ffb-8966-f8bef09035b7n@googlegroups.com> Subject: Re: [bitcoindev] Re: Reducing RAM requirements with dynamic dust MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_10214_1700397484.1765399473693" X-Original-Sender: eric@voskuil.org Precedence: list Mailing-list: list bitcoindev@googlegroups.com; contact bitcoindev+owners@googlegroups.com List-ID: X-Google-Group-Id: 786775582512 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Score: -0.7 (/) ------=_Part_10214_1700397484.1765399473693 Content-Type: multipart/alternative; boundary="----=_Part_10215_974910444.1765399473693" ------=_Part_10215_974910444.1765399473693 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Erik, It's a good idea, that's why we did it. Libbitcoin is a set of libraries, libbitcoin-database being one of 4 that= =20 make up node (system, network, database, and node). (libbitcoin-server is= =20 an additional library that adds comprehensive set of client-server=20 interfaces, making the node useful.) libbitcoin-database is an=20 implementation of a simple query interface (defined as a c++ class) over a= =20 backing store. The store is a templated collection of tables. The tables=20 are mmap-based head and body files used to construct multimaps, hashmaps,= =20 arraymaps, arrays, and blobs. We mock the tables using=20 std::vector, mock the store using the the mocked tables, and test= =20 most of the query interface using the mock store. The structure is highly= =20 relational, and surrogate keys are exposed to the caller for optimized=20 navigation. This is an isolated and clear storage abstraction layer. All validation is= =20 performed within the chain:: classes (e.g. block, header, tx, input,=20 script, etc.) defined in the base level libbitcoin-system. There is no=20 validator coupling to the store. The store retains chain objects, indexes,= =20 and validation state for headers/blocks and txs as they progress through a= =20 state machine. There is no utxo table, just natural relations between=20 objects, indexed and related. Validation correctness of course requires=20 store fidelity, but is totally decoupled from it. We validate blocks=20 concurrently, queueing up 50,000 blocks at a time by default (e.g. with 50k= =20 threads we would validate all at the same time). The store could be replaced with no impact to the query interface (as we=20 already do in testing). So it's not really accurate to imply that=20 libbitcoin's validation is tied to mmap or even append-only. Pruning could= =20 be implemented in the existing model for example. The existing store could= =20 be replaced with something simple and light like SQLite, or a full RDBMS.= =20 We had some interns working the SQLite approach last summer. That would be= =20 more specialized for low performance scenarios, where the custom database= =20 targets ultra high performance.=20 With sufficient RAM there is never SSD access. The store can sync up and=20 just live in RAM, never touching a disk. Since it is append only, it's very= =20 low impact on SSDs. As the store builds, no table body byte is ever=20 re-written. Table heads are hashmaps buckets, small and dynamic. It=20 performs live automated/manual snapshotting, automatic fault=20 detection/recovery on restart, automated disk-full pause/restart, and=20 supports hot backup. Query performance is phenomenal. A warm node can query= =20 the full 5.2 million output Electrum query (very complex relations) in 15= =20 seconds on my 2.1GHz workstation. But at the low end, an off-the-shelf=20 store is sufficient. A clear interface and swappable store makes a lot of= =20 sense. A large utxo set makes no difference in this design. There are no=20 operational problems associated with it. Given the recent fuss about it, I= =20 figured it was worth putting some detail behind it here. This is not=20 theoretical, at this point we are only working on server (client-server=20 interfaces). The utxo set size is a complete non-issue. e On Wednesday, December 10, 2025 at 2:48:43=E2=80=AFPM UTC-5 Erik Aronesty w= rote: > Core: Keep UTXO lookups in RAM as much as possible. This gives predictabl= e=20 > validation under adversarial load and trades memory for protection from= =20 > pathological disk latency. > Libbitcoin: Represent history in append-only, memory-mapped files. Let th= e=20 > operating system decide what stays hot. Accept more disk I/O in return fo= r=20 > simpler concurrency and no explicit UTXO map. > > To create 200 million outputs as an attack, the cost is roughly 48 BTC=20 > based on eight vb per output at low feerates. > > Given the operational problems that a very large UTXO set creates, it may= =20 > be worthwhile to develop a clear storage abstraction layer.=20 > > The goal would be to ensure that validation correctness does not depend o= n=20 > the transitional behavior of a specific storage engine.=20 > > Once the validator has a well defined storage contract, alternate backend= s=20 > such as a libbitcoin style layout could exist as storage formats while=20 > still benefiting from the reliability of the rest of the Bitcoin=20 > implementation. > (I would be interested in something like this.) > > On Wed, Dec 10, 2025 at 10:34=E2=80=AFAM Eric Voskuil = wrote: > >> > Given the increasing RAM requirements, due to the increasing UTXO set >> >> I have been assured multiple times recently that this problem does not= =20 >> exist in Bitcoin Core. I'm not sure about that, but I can say for sure t= hat=20 >> this is not a problem in Libbitcoin and is therefore not problem with th= e=20 >> Bitcoin protocol. >> >> Best, >> Eric >> >> On Wednesday, December 10, 2025 at 1:12:31=E2=80=AFPM UTC-5 uuowwpevskfc= ordh=20 >> wrote: >> >>> Given the increasing RAM requirements, due to the increasing UTXO set, = I=20 >>> suggest seeing the UTXO set size as a controlled variable. A feedback= =20 >>> mechanism sets a dynamic dust level, below from which UTXOs are=20 >>> removed/discarded and thus freeing RAM. >>> >>> Below is an overview essay better expressed by grok, which can also be= =20 >>> seen in here: >>> https://hackmd.io/P-2lzGb8TiC86IOE3OGiYA?view >>> >>> # Enhancing Bitcoin's Scalability: A PID-Inspired Approach to Managing= =20 >>> UTXO Set Growth >>> >>> ## Abstract >>> >>> Bitcoin=E2=80=99s UTXO set is currently an unbounded accumulator that r= isks=20 >>> long-term centralization as node RAM requirements grow without limit.= =20 >>> Existing fee incentives have proven insufficient against sustained=20 >>> low-value output creation (e.g., inscriptions, tokenized assets, dust-h= eavy=20 >>> protocols). This article proposes a soft-fork mechanism that treats UTX= O=20 >>> set size as a controlled variable: a slowly rising target size is defin= ed,=20 >>> and a PID-style feedback controller, updated every difficulty epoch,=20 >>> dynamically raises a minimum-value floor beneath which old UTXOs become= =20 >>> unspendable. The result is bounded, predictable growth of the UTXO set = with=20 >>> ample warning periods, no hard caps on monetary use, and strong resista= nce=20 >>> to bloat attacks=E2=80=94all while remaining fully compatible with a so= ft-fork=20 >>> deployment. >>> >>> ## Introduction >>> >>> Bitcoin, the pioneering decentralized digital currency, operates as a= =20 >>> complex dynamic system governed by consensus rules that ensure security= ,=20 >>> immutability, and permissionless participation. At its core, Bitcoin=20 >>> maintains a distributed ledger known as the blockchain, which records a= ll=20 >>> transactions in a sequence of blocks. Each transaction involves inputs = and=20 >>> outputs: inputs reference previously unspent outputs from prior=20 >>> transactions, while outputs create new spendable units called Unspent= =20 >>> Transaction Outputs (UTXOs). The UTXO set represents the aggregate stat= e of=20 >>> all currently spendable coins in the network, serving as a critical=20 >>> component for transaction validation and wallet management. >>> >>> As Bitcoin has evolved, the UTXO set has grown significantly, influence= d=20 >>> by increasing adoption and diverse usage patterns. This growth, while= =20 >>> indicative of the network's vitality, poses challenges to its long-term= =20 >>> scalability and decentralization. Nodes=E2=80=94computers that validate= and relay=20 >>> transactions=E2=80=94must store and process the entire UTXO set in memo= ry for=20 >>> efficient operation, which can strain resources such as random access= =20 >>> memory (RAM). Unchecked expansion could lead to higher barriers for run= ning=20 >>> full nodes, potentially centralizing control among fewer, well-resource= d=20 >>> participants. This article explores a proposed mechanism to address the= se=20 >>> concerns: an adaptive control system inspired by=20 >>> Proportional-Integral-Derivative (PID) feedback principles, designed to= =20 >>> impose bounded growth on the UTXO set while preserving Bitcoin's=20 >>> foundational properties. >>> >>> ## Motivations for UTXO Set Management >>> >>> To appreciate the need for enhanced UTXO management, it is essential to= =20 >>> understand the current status quo and its vulnerabilities. Bitcoin's de= sign=20 >>> prioritizes efficiency and security through mechanisms like block size= =20 >>> limits, which cap the amount of data added per block (approximately 1 M= B=20 >>> base size, expandable to about 4 MB with Segregated Witness). These lim= its=20 >>> help control the overall blockchain size, ensuring predictable hardware= =20 >>> requirements for storage on hard disk drives (HDDs). Similarly, the=20 >>> difficulty adjustment algorithm maintains a consistent block production= =20 >>> rate of roughly one every 10 minutes by dynamically scaling the=20 >>> computational challenge for miners based on network hashrate. >>> >>> However, the UTXO set lacks comparable built-in constraints. It=20 >>> accumulates as users create new outputs=E2=80=94often in small=20 >>> denominations=E2=80=94without a mandatory mechanism to consolidate or p= rune them.=20 >>> This can result from various activities, including high-frequency=20 >>> microtransactions, the embedding of non-monetary data (such as through= =20 >>> protocols like Ordinals, which inscribe arbitrary information onto=20 >>> satoshis, Bitcoin's smallest unit), or the anchoring of sidechain or=20 >>> tokenized assets that leverage Bitcoin's security but operate externall= y.=20 >>> While these innovations expand Bitcoin's utility, they can inadvertentl= y=20 >>> increase the UTXO count disproportionately to their economic value,=20 >>> elevating transaction fees during congestion and raising operational co= sts=20 >>> for nodes. >>> >>> Existing incentives, such as transaction fees, partially mitigate this= =20 >>> by encouraging users to consolidate low-value UTXOs to avoid higher cos= ts.=20 >>> Yet, these market-driven forces are insufficient against sustained patt= erns=20 >>> of low-value output creation, particularly when driven by external syst= ems=20 >>> that do not bear the full cost of network maintenance. Over time, this= =20 >>> leads to bloat: as of recent estimates, the UTXO set exceeds several=20 >>> gigabytes when loaded into memory, complicating node synchronization an= d=20 >>> validation. Without intervention, projected growth could undermine=20 >>> decentralization, as fewer individuals or entities might afford to=20 >>> participate fully in the network. >>> >>> The motivation for reform, therefore, stems from a desire to balance=20 >>> innovation with sustainability. An ideal solution would allow gradual U= TXO=20 >>> expansion to accommodate genuine monetary usage while introducing feedb= ack=20 >>> to curb excessive accumulation, all without compromising Bitcoin's=20 >>> permissionless nature or requiring a hard fork that could fragment the= =20 >>> community. >>> >>> ## A PID-Inspired Feedback Controller for UTXO Control >>> >>> Drawing from control theory, which studies how systems maintain desired= =20 >>> behaviors through feedback, this proposal introduces a dynamic mechanis= m to=20 >>> regulate UTXO set size. PID controllers, widely used in engineering for= =20 >>> processes like temperature regulation or autopilot systems, combine thr= ee=20 >>> terms: Proportional \(P\) for immediate response to errors, Integral (I= )=20 >>> for correcting persistent deviations, and Derivative (D) for anticipati= ng=20 >>> changes based on trends. In Bitcoin's context, we adapt this framework = to=20 >>> compute a minimum value threshold=E2=80=94or "floor"=E2=80=94below whic= h UTXOs become=20 >>> ineligible for spending, effectively deprecating them over time. >>> >>> Importantly, this deprecation rule constitutes a soft fork: it restrict= s=20 >>> the set of valid transactions to a subset of those previously allowed,= =20 >>> without introducing new capabilities. Upgraded nodes enforce the rule,= =20 >>> rejecting deprecated UTXOs as invalid inputs, ensuring backward=20 >>> compatibility and minimizing disruption. >>> >>> ### Mechanism Overview >>> >>> The controller activates at each difficulty adjustment epoch,=20 >>> approximately every two weeks (2016 blocks), aligning with Bitcoin's=20 >>> existing periodic recalibrations for predictability. >>> >>> 1. **Define a Target UTXO Size Trajectory**: Establish an increasing=20 >>> maximum target for the UTXO set size, denoted as $T(t)$, where $t$=20 >>> represents the epoch number. This could grow sublinearly with time or= =20 >>> blockchain height to reflect organic adoption=E2=80=94 for instance, $T= (t) =3D U_0=20 >>> \cdot (1 + r)^t$, with $U_0$ as the initial size and $r$ a small annual= =20 >>> growth rate (e.g., less than 1%). This permits expansion while preventi= ng=20 >>> unbounded divergence. >>> >>> 2. **Measure the Error**: At each epoch, compute the error $e(t) =3D S(= t)=20 >>> - T(t)$, where $S(t)$ is the current UTXO set size (measured by count o= r=20 >>> aggregate data footprint). >>> >>> 3. **Compute the Floor Adjustment**: Apply the PID formula to adjust th= e=20 >>> value floor $F(t)$: >>> =20 >>> $$ \Delta F(t) =3D K_p \cdot e(t) + K_i \cdot \sum_{k=3D0}^{t} e(k= ) +=20 >>> K_d \cdot \frac{e(t) - e(t-1)}{\Delta t} $$ >>> =20 >>> Here, $K_p, K_i, K_d$ are tunable gains, selected conservatively=20 >>> through simulations to ensure stability (e.g., using root locus analysi= s to=20 >>> avoid oscillatory or unstable roots in the system's characteristic=20 >>> equation). The floor $F(t)$ increases only if $e(t) > 0$, applying to U= TXOs=20 >>> below this threshold. >>> >>> 4. **Deprecation Process**: UTXOs with values below $F(t)$ are marked a= s=20 >>> unspendable in future transactions. To provide fairness and predictabil= ity,=20 >>> implement a grace period (e.g., 4-12 months) during which owners can=20 >>> consolidate affected UTXOs. >>> >>> 5. **Safeguards and Tuning**: Incorporate clamps on $\Delta F(t)$=20 >>> (similar to Bitcoin's difficulty adjustment limits) to prevent abrupt= =20 >>> changes that might cause network congestion from mass consolidations.= =20 >>> Periodic reviews of gains and targets via community governance would ad= apt=20 >>> the system to evolving conditions. >>> >>> ### Stability and Dynamics Considerations >>> >>> In dynamic systems terms, the UTXO set can be modeled as an integrator= =20 >>> accumulating outputs minus expenditures. The PID controller introduces= =20 >>> negative feedback to dampen this accumulation, driving $S(t)$ toward $T= (t)$=20 >>> with minimal overshoot. Simulations would verify robustness against=20 >>> adversarial scenarios, such as deliberate UTXO spam, ensuring the syste= m's=20 >>> poles remain in the stable region of the complex plane. >>> >>> Potential benefits include reduced node resource demands, lower fees=20 >>> during normal operation, and enhanced decentralization. Risks, such as= =20 >>> over-deprecation affecting small holders, can be mitigated through care= ful=20 >>> gain selection. >>> >>> ## Conclusion >>> >>> This PID-inspired approach offers a principled path to managing=20 >>> Bitcoin's UTXO set, fostering sustainable growth while upholding core= =20 >>> principles. By integrating feedback control, Bitcoin can evolve as a=20 >>> resilient dynamic system, better equipped for widespread adoption. Furt= her=20 >>> research, including detailed modeling and community discourse, is essen= tial=20 >>> to refine and potentially implement such enhancements. >>> >>> --=20 >> You received this message because you are subscribed to the Google Group= s=20 >> "Bitcoin Development Mailing List" group. >> To unsubscribe from this group and stop receiving emails from it, send a= n=20 >> email to bitcoindev+...@googlegroups.com. >> To view this discussion visit=20 >> https://groups.google.com/d/msgid/bitcoindev/049858f4-28ba-4ffb-8966-f8b= ef09035b7n%40googlegroups.com=20 >> >> . >> > --=20 You received this message because you are subscribed to the Google Groups "= Bitcoin Development Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to bitcoindev+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/= 97065ca0-662b-4a5d-90c5-7607db41ba2fn%40googlegroups.com. ------=_Part_10215_974910444.1765399473693 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Erik,

It's a good idea, that's why we did it.

Libb= itcoin is a set of libraries, libbitcoin-database being one of 4 that make = up node (system, network, database, and node).=C2=A0 (libbitcoin-server is = an additional library that adds comprehensive set of client-server interfac= es, making the node useful.) libbitcoin-database is an implementation of a = simple query interface (defined as a c++ class) over a backing store. The s= tore is a templated collection of tables. The tables are mmap-based head an= d body files used to construct multimaps, hashmaps, arraymaps, arrays, and = blobs. We mock the tables using std::vector<uint8_t>, mock the store = using the the mocked tables, and test most of the query interface using the= mock store. The structure is highly relational, and surrogate keys are exp= osed to the caller for optimized navigation.

This is an isolated= and clear storage abstraction layer. All validation is performed within th= e chain:: classes (e.g. block, header, tx, input, script, etc.) defined in = the base level libbitcoin-system. There is no validator coupling to the sto= re. The store retains chain objects, indexes, and validation state for head= ers/blocks and txs as they progress through a state machine. There is no ut= xo table, just natural relations between objects, indexed and related. Vali= dation correctness of course requires store fidelity, but is totally decoup= led from it. We validate blocks concurrently, queueing up 50,000 blocks at = a time by default (e.g. with 50k threads we would validate all at the same = time).

The store could be replaced with no impact to the query i= nterface (as we already do in testing). So it's not really accurate to impl= y that libbitcoin's validation is tied to mmap or even append-only. Pruning= could be implemented in the existing model for example. The existing store= could be replaced with something simple and light like SQLite, or a full R= DBMS. We had some interns working the SQLite approach last summer. That wou= ld be more specialized for low performance scenarios, where the custom data= base targets ultra high performance.

With sufficient RAM there = is never SSD access. The store can sync up and just live in RAM, never touc= hing a disk. Since it is append only, it's very low impact on SSDs. As the = store builds, no table body byte is ever re-written. Table heads are hashma= ps buckets, small and dynamic. It performs live automated/manual snapshotti= ng, automatic fault detection/recovery on restart, automated disk-full paus= e/restart, and supports hot backup. Query performance is phenomenal. A warm= node can query the full 5.2 million output Electrum query (very complex re= lations) in 15 seconds on my 2.1GHz workstation. But at the low end, an off= -the-shelf store is sufficient. A clear interface and swappable store makes= a lot of sense.

A large utxo set makes no difference in this de= sign. There are no operational problems associated with it. Given the recen= t fuss about it, I figured it was worth putting some detail behind it here.= This is not theoretical, at this point we are only working on server (clie= nt-server interfaces). The utxo set size is a complete non-issue.

e

On Wednesday, December 10, 2025 at 2:48:43=E2=80=AFPM UTC-5 Erik Aron= esty wrote:
<= div dir=3D"ltr">

Core: Keep UTXO lookups in RAM as much = as possible. This gives predictable validation under adversarial load and t= rades memory for protection from pathological disk latency.
Libbitcoin: Represent history in append-only, memory-mapped files. Let the = operating system decide what stays hot. Accept more disk I/O in return for = simpler concurrency and no explicit UTXO map.

To create 200 million outputs as an attack, the cost is roughly 48 BTC b= ased on eight vb per output at low feerates.

Given the operational problems that a very large UTXO set creates, it ma= y be worthwhile to develop a clear storage abstraction layer.

The g= oal would be to ensure that validation correctness does not depend on the t= ransitional behavior of a specific storage engine.

Once the validat= or has a well defined storage contract, alternate backends such as a libbit= coin style layout could exist as storage formats while still benefiting fro= m the reliability of the rest of the Bitcoin implementation.

(I wo= uld be interested in something like this.)

On Wed, Dec 10, 2025 at 10:34=E2=80=AFAM Eric Voskuil = <er...@voskuil.org> wr= ote:
> Given the increasing=C2=A0RAM requirements, due to the = increasing=C2=A0UTXO set

I have been assured multiple times r= ecently that this problem does not exist in Bitcoin Core. I'm not sure = about that, but I can say for sure that this is not a problem in Libbitcoin= and is therefore not problem with the Bitcoin protocol.

Best,
Er= ic

On Wednesday, December 10, 2025 at 1:12:31=E2=80=AFPM UTC-5 uuowwpevskfcor= dh wrote:
Given the increasing= =C2=A0RAM requirements, due to the increasing=C2=A0UTXO set, I suggest seei= ng the UTXO set size as a controlled variable. A feedback mechanism sets a = dynamic dust level, below from which UTXOs are removed/discarded and thus f= reeing RAM.

Below= is an overview essay better expressed by grok, which can also be seen in h= ere:
<= div style=3D"font-family:Arial,sans-serif;font-size:14px">
# Enhancing = Bitcoin's Scalability: A PID-Inspired Approach to Managing UTXO Set Gro= wth

## Abstract

=
Bitcoin=E2=80=99s UTXO set is currently an unbounded accum= ulator that risks long-term centralization as node RAM requirements grow wi= thout limit. Existing fee incentives have proven insufficient against susta= ined low-value output creation (e.g., inscriptions, tokenized assets, dust-= heavy protocols). This article proposes a soft-fork mechanism that treats U= TXO set size as a controlled variable: a slowly rising target size is defin= ed, and a PID-style feedback controller, updated every difficulty epoch, dy= namically raises a minimum-value floor beneath which old UTXOs become unspe= ndable. The result is bounded, predictable growth of the UTXO set with ampl= e warning periods, no hard caps on monetary use, and strong resistance to b= loat attacks=E2=80=94all while remaining fully compatible with a soft-fork = deployment.

## Introduction

Bitcoin, the pioneering decentralized digital = currency, operates as a complex dynamic system governed by consensus rules = that ensure security, immutability, and permissionless participation. At it= s core, Bitcoin maintains a distributed ledger known as the blockchain, whi= ch records all transactions in a sequence of blocks. Each transaction invol= ves inputs and outputs: inputs reference previously unspent outputs from pr= ior transactions, while outputs create new spendable units called Unspent T= ransaction Outputs (UTXOs). The UTXO set represents the aggregate state of = all currently spendable coins in the network, serving as a critical compone= nt for transaction validation and wallet management.

<= /div>
As Bitcoin has evolved, the UTXO set has grown significantl= y, influenced by increasing adoption and diverse usage patterns. This growt= h, while indicative of the network's vitality, poses challenges to its = long-term scalability and decentralization. Nodes=E2=80=94computers that va= lidate and relay transactions=E2=80=94must store and process the entire UTX= O set in memory for efficient operation, which can strain resources such as= random access memory (RAM). Unchecked expansion could lead to higher barri= ers for running full nodes, potentially centralizing control among fewer, w= ell-resourced participants. This article explores a proposed mechanism to a= ddress these concerns: an adaptive control system inspired by Proportional-= Integral-Derivative (PID) feedback principles, designed to impose bounded g= rowth on the UTXO set while preserving Bitcoin's foundational propertie= s.

## Motivations for UTXO Set Manage= ment

To appreciate the need for enhan= ced UTXO management, it is essential to understand the current status quo a= nd its vulnerabilities. Bitcoin's design prioritizes efficiency and sec= urity through mechanisms like block size limits, which cap the amount of da= ta added per block (approximately 1 MB base size, expandable to about 4 MB = with Segregated Witness). These limits help control the overall blockchain = size, ensuring predictable hardware requirements for storage on hard disk d= rives (HDDs). Similarly, the difficulty adjustment algorithm maintains a co= nsistent block production rate of roughly one every 10 minutes by dynamical= ly scaling the computational challenge for miners based on network hashrate= .

However, the UTXO set lacks compara= ble built-in constraints. It accumulates as users create new outputs=E2=80= =94often in small denominations=E2=80=94without a mandatory mechanism to co= nsolidate or prune them. This can result from various activities, including= high-frequency microtransactions, the embedding of non-monetary data (such= as through protocols like Ordinals, which inscribe arbitrary information o= nto satoshis, Bitcoin's smallest unit), or the anchoring of sidechain o= r tokenized assets that leverage Bitcoin's security but operate externa= lly. While these innovations expand Bitcoin's utility, they can inadver= tently increase the UTXO count disproportionately to their economic value, = elevating transaction fees during congestion and raising operational costs = for nodes.

Existing incentives, such = as transaction fees, partially mitigate this by encouraging users to consol= idate low-value UTXOs to avoid higher costs. Yet, these market-driven force= s are insufficient against sustained patterns of low-value output creation,= particularly when driven by external systems that do not bear the full cos= t of network maintenance. Over time, this leads to bloat: as of recent esti= mates, the UTXO set exceeds several gigabytes when loaded into memory, comp= licating node synchronization and validation. Without intervention, project= ed growth could undermine decentralization, as fewer individuals or entitie= s might afford to participate fully in the network.

The motivation for reform, therefore, stems from a desire to= balance innovation with sustainability. An ideal solution would allow grad= ual UTXO expansion to accommodate genuine monetary usage while introducing = feedback to curb excessive accumulation, all without compromising Bitcoin&#= 39;s permissionless nature or requiring a hard fork that could fragment the= community.

## A PID-Inspired Feedbac= k Controller for UTXO Control

Drawing= from control theory, which studies how systems maintain desired behaviors = through feedback, this proposal introduces a dynamic mechanism to regulate = UTXO set size. PID controllers, widely used in engineering for processes li= ke temperature regulation or autopilot systems, combine three terms: Propor= tional \(P\) for immediate response to errors, Integral (I) for correcting = persistent deviations, and Derivative (D) for anticipating changes based on= trends. In Bitcoin's context, we adapt this framework to compute a min= imum value threshold=E2=80=94or "floor"=E2=80=94below which UTXOs= become ineligible for spending, effectively deprecating them over time.

Importantly, this deprecation rule cons= titutes a soft fork: it restricts the set of valid transactions to a subset= of those previously allowed, without introducing new capabilities. Upgrade= d nodes enforce the rule, rejecting deprecated UTXOs as invalid inputs, ens= uring backward compatibility and minimizing disruption.
### Mechanism Overview

<= span>The controller activates at each difficulty adjustment epoch, approxim= ately every two weeks (2016 blocks), aligning with Bitcoin's existing p= eriodic recalibrations for predictability.

= 1. **Define a Target UTXO Size Trajectory**: Establish an increasing = maximum target for the UTXO set size, denoted as $T(t)$, where $t$ represen= ts the epoch number. This could grow sublinearly with time or blockchain he= ight to reflect organic adoption=E2=80=94 for instance, $T(t) =3D U_0 \cdot= (1 + r)^t$, with $U_0$ as the initial size and $r$ a small annual growth r= ate (e.g., less than 1%). This permits expansion while preventing unbounded= divergence.

2. **Measure the Error**= : At each epoch, compute the error $e(t) =3D S(t) - T(t)$, where $S(t)$ is = the current UTXO set size (measured by count or aggregate data footprint).<= /span>

3. **Compute the Floor Adjustment**: = Apply the PID formula to adjust the value floor $F(t)$:
=C2=A0 =C2=A0
=C2=A0 =C2=A0 $$ =C2=A0\Delta F(t)= =3D K_p \cdot e(t) + K_i \cdot \sum_{k=3D0}^{t} e(k) + K_d \cdot \frac{e(t= ) - e(t-1)}{\Delta t} $$
=C2=A0 =C2=A0
<= div>=C2=A0 =C2=A0Here, $K_p, K_i, K_d$ are tunable gains, selected co= nservatively through simulations to ensure stability (e.g., using root locu= s analysis to avoid oscillatory or unstable roots in the system's chara= cteristic equation). The floor $F(t)$ increases only if $e(t) > 0$, appl= ying to UTXOs below this threshold.

4= . **Deprecation Process**: UTXOs with values below $F(t)$ are marked as uns= pendable in future transactions. To provide fairness and predictability, im= plement a grace period (e.g., 4-12 months) during which owners can consolid= ate affected UTXOs.

5. **Safeguards a= nd Tuning**: Incorporate clamps on $\Delta F(t)$ (similar to Bitcoin's = difficulty adjustment limits) to prevent abrupt changes that might cause ne= twork congestion from mass consolidations. Periodic reviews of gains and ta= rgets via community governance would adapt the system to evolving condition= s.

### Stability and Dynamics Conside= rations

In dynamic systems terms, the= UTXO set can be modeled as an integrator accumulating outputs minus expend= itures. The PID controller introduces negative feedback to dampen this accu= mulation, driving $S(t)$ toward $T(t)$ with minimal overshoot. Simulations = would verify robustness against adversarial scenarios, such as deliberate U= TXO spam, ensuring the system's poles remain in the stable region of th= e complex plane.

Potential benefits i= nclude reduced node resource demands, lower fees during normal operation, a= nd enhanced decentralization. Risks, such as over-deprecation affecting sma= ll holders, can be mitigated through careful gain selection.

## Conclusion

This= PID-inspired approach offers a principled path to managing Bitcoin's U= TXO set, fostering sustainable growth while upholding core principles. By i= ntegrating feedback control, Bitcoin can evolve as a resilient dynamic syst= em, better equipped for widespread adoption. Further research, including de= tailed modeling and community discourse, is essential to refine and potenti= ally implement such enhancements.

--
You received this message because you are subscribed to the Google Groups &= quot;Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to bitcoindev+...@googlegro= ups.com.
To view this discussion visit https= ://groups.google.com/d/msgid/bitcoindev/049858f4-28ba-4ffb-8966-f8bef09035b= 7n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups &= quot;Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to bitcoind= ev+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoind= ev/97065ca0-662b-4a5d-90c5-7607db41ba2fn%40googlegroups.com.
------=_Part_10215_974910444.1765399473693-- ------=_Part_10214_1700397484.1765399473693--