Support optimize other file types

.github/workflows/run.yml

@@ -30,7 +30,7 @@ jobs:
       - name: Setup node
         uses: actions/setup-node@v4
         with:
-          node-version: '20.11.0'
+          node-version: '23.7.0'
           cache: 'npm'
 
       - name: Build rust
@@ -41,7 +41,7 @@ jobs:
           mkdir -p ./testdir
           export psql_url=postgresql://psql:psql@localhost:5432/psql
           export encryption_pass=supersuper
-          RUST_LOG=info cargo run -- -o 'allow_other' --ttl 0 test1 $(pwd)/testdir &
+          cargo run -- -o 'allow_other' --ttl 0 test1 $(pwd)/testdir &
           sleep 5
           npm install
           npm run test
@@ -51,7 +51,7 @@ jobs:
         run: |
           export psql_url=postgresql://psql:psql@localhost:5432/psql
           export encryption_pass=supersuper
-          RUST_LOG=info cargo run -- -o 'allow_other' --ttl 180 test1 $(pwd)/testdir &
+          cargo run -- -o 'allow_other' --ttl 180 test1 $(pwd)/testdir &
           sleep 5
           npm run test
           umount $(pwd)/testdir

PRAGMA.md

@@ -7,7 +7,7 @@ All journal_modes are supported by SQLitesuperfs but only 2 are recommended. SQL
 DELETE is the SQLite default and it works fine.
 
 ## Journal_mode = TRUNCATE
-TRUNCATE is at this time the SQLitesuperfs recommendation. It only runs maybe 5% faster than DELETE but it results in less dead tuples for the PSQL server. So it is better for the health of the server.
+TRUNCATE is at this time the SQLitesuperfs recommendation. It runs faster than DELETE and it results in less dead tuples.
 
 ## Journal_mode = WAL
 Everyone wants to know about WAL. WAL is the crowd favorite and on a normal FS it is the best journal_mode. Many people think only WAL allows multi-reader but DELETE and TRUNCATE allow multi-reader also. The main advantage of WAL is that readers never block writes. With DELETE and TRUNCATE an INSERT *may* block waiting for a SELECT to complete. The SQLite locking protocol is actually very good and so there is only a small window in which for this to happen but it can happen.

README.md

@@ -1,5 +1,5 @@
 # SQLitesuperfs
-FUSE fs with PostgreSQL backend, block-level encryption, and optimizations for SQLite multi-tenancy.
+SQLite FUSE fs with PostgreSQL backend, block-level encryption, and optimizations for multi-tenancy.
 
 ## Why
 [Lock.host](https://github.com/rhodey/lock.host) allows apps to attest to their code and to encrypt comms with clients. Lock.host also allows apps to create persistent keys. SQLitesuperfs was created to allow apps to keep persistent state (SQLite) and to keep that state private. All existing open-source encrypted filesystems use AES which fundamentally is vulnerable to timing attacks. Timing attacks mean "do not run this in the cloud". [Libsodium](https://doc.libsodium.org/) is used and is not vulnerable to timing attacks.
@@ -36,8 +36,9 @@ All write operations are atomic (one PSQL txn) so this is good. However the linu
 SQLitesuperfs is at this time single threaded. SQLite is fundamentally 1-writer-multi-reader and so it does make sense for SQLitesuperfs to add a read thread pool. Contributions welcome! Keep in mind multiple apps are multiple namespaces (schemas) and do not have lock contention with eachother though they all share PSQL I/O.
 
 ## Quick Test
-Nodejs is in this repo to do tests
+Nodejs is in this repo to do tests:
 ```
+docker compose up -d psql
 mkdir -p ./testdir
 export psql_url=postgresql://psql:psql@localhost:5432/psql
 export encryption_pass=supersuper
@@ -54,7 +55,7 @@ SQLitesuperfs passes `make test` in the official SQLite source tree. So you can:
 + Run `make test` from the mount
 
 ## Performance
-Early testing is showing SQLite on SQLitesuperfs to be 2x to 3x slower when the PSQL server is localhost. When the PSQL server is eg Amazon Aurora in the same AZ SQLitesuperfs will be slower than 3x. SQLitesuperfs is primarily about *private hosted SQLite impossible => possible*. SQLitesuperfs even at 100x would be an achievement compared to Fully Homomorphic Encryption being 100,000x to 1,000,000x.
+When the PSQL server is localhost SQLite is 15% faster than native FS thanks to SQLitesuperfs buffering. When the PSQL server has 0.2ms simulated RTT latency SQLite is 2x to 3x slower than native FS. SQLitesuperfs is primarily about *private hosted SQLite impossible => possible*. SQLitesuperfs even at 100x would be an achievement compared to Fully Homomorphic Encryption being 100,000x to 1,000,000x.
 
 The read thread pool described earlier will help and additionally a custom [VFS](https://sqlite.org/vfs.html) can be added to improve writes but TBH I am very happy already to build apps on this so I dont say I will be back here to add either *super* soon.
 
@@ -64,6 +65,14 @@ The read thread pool described earlier will help and additionally a custom [VFS]
 + Something with Merkle Trees and [TEEs](https://en.wikipedia.org/wiki/Trusted_execution_environment) can stop rollback (future)
 + (Every encrypted filesystem is vulnerable to rollback)
 
+## Also
+SQLitesuperfs optimizations can be applied to data structures like [tinyraftplus](https://github.com/rhodey/tinyraftplus) if they have a lockfile protocol similar to SQLite. The first and second commands are equivalent. And the third command is what would be used to optimize SQLite and tinyraftplus at the same time:
+```
+sqlitesuperfs super1 /tmp/super1
+sqlitesuperfs super1 /tmp/super1 --pattern db-journal,db
+sqlitesuperfs super1 /tmp/super1 --pattern db-journal,db --pattern lock,off,log
+```
+
 ## Also
 Multi-Host Same-Namespace is not really on the roadmap but it is possible. Most of what you want from Multi-Host is: 1. durability 2. horizontal scaling. 1 is covered by PSQL and 2, well, just use a bigger host (and keep code simple). Multi-Host Same-Namespace would be something like LiteFS with SQLite WAL mode where you have 1 primary and N (asynchronous) replicas. LiteFS is cool but it should be noted that LiteFS durability is weaker, COMMIT always returns before replicas replicate.
 

package.json

@@ -7,12 +7,13 @@
     "mounttest": "export $(cat .env | xargs) && RUST_LOG=none,fuse=debug,sqlitesuperfs=info cargo run -- test1 $(pwd)/testdir",
     "mounttestq": "export $(cat .env | xargs) && RUST_LOG=none,sqlitesuperfs=info cargo run -- test1 $(pwd)/testdir",
     "mounttest0": "export $(cat .env | xargs) && RUST_LOG=none,fuse=debug,sqlitesuperfs=info cargo run -- --ttl 0 test1 $(pwd)/testdir",
-    "test": "tape test/1basic.js && tape test/2dir.js && tape test/3rw.js && tape test/4link.js && tape test/5misc.js && tape test/6sql.js",
-    "test-umount": "tape test/7umount.js"
+    "test": "tape test/1basic.js && tape test/2dir.js && tape test/3rw.js && tape test/4link.js && tape test/5misc.js && tape test/6sql.js && tape test/7raft.js",
+    "test-umount": "tape test/8umount.js"
   },
   "devDependencies": {
     "tape": "^5.9.0",
     "better-sqlite3": "^12.4.6",
+    "tinyraftplus": "^0.3.0",
     "split": "^1.0.1"
   }
 }

schema.sql

@@ -98,7 +98,7 @@ begin
   insert into :ns.blocks (ino, num, buf) values (i, n, b)
     on conflict (ino, num) do update
       set buf = b;
-  update :ns.inodes set size = sz where sz > 0 and id = i;
+  update :ns.inodes set size = sz where sz >= 0 and id = i;
   return 1;
 end;
 $$ language plpgsql;