Full schema insertion + schema testing #848
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5bec84c to
919a01f
Compare
98c2b96 to
9101db8
Compare
Contributor
Author
|
@bgentry Still needs a little more work, but test runs go from ~2m45s to ~1m, which is pretty nice. I'm not even completely sure why the improvement is that good, but I think we were waiting a lot longer than we thought waiting for test databases. Although only N = num CPUs tests were running in parallel at any given time, I think what happens is that Go boots a whole bunch of test cases, but may pause them while it engages others, some of which may have been holding a test database and leaving the available pool smaller for other test cases that might need one. Ungating the number of schemas allowed at any given time I found that the main package will generate up to 125 of them before tests start finishing fully and it starts reusing them. Not 100% sure about this theory, but whatever it is exactly, the speed increase dramatic. I'm still blocked on one major thing which is that sqlc's diff --git a/riverdriver/riverpgxv5/internal/dbsqlc/copyfrom.go b/riverdriver/riverpgxv5/internal/dbsqlc/copyfrom.go
index ee29edf..5d18b02 100644
--- a/riverdriver/riverpgxv5/internal/dbsqlc/copyfrom.go
+++ b/riverdriver/riverpgxv5/internal/dbsqlc/copyfrom.go
@@ -48,6 +48,11 @@ func (r iteratorForJobInsertFastManyCopyFrom) Err() error {
return nil
}
-func (q *Queries) JobInsertFastManyCopyFrom(ctx context.Context, db DBTX, arg []*JobInsertFastManyCopyFromParams) (int64, error) {
- return db.CopyFrom(ctx, []string{"river_job"}, []string{"args", "created_at", "kind", "max_attempts", "metadata", "priority", "queue", "scheduled_at", "state", "tags", "unique_key", "unique_states"}, &iteratorForJobInsertFastManyCopyFrom{rows: arg})
+func (q *Queries) JobInsertFastManyCopyFrom(ctx context.Context, db DBTX, schema string, arg []*JobInsertFastManyCopyFromParams) (int64, error) {
+ tableName := []string{"river_job"}
+ if schema != "" {
+ tableName = append([]string{schema}, tableName...)
+ }
+
+ return db.CopyFrom(ctx, tableName, []string{"args", "created_at", "kind", "max_attempts", "metadata", "priority", "queue", "scheduled_at", "state", "tags", "unique_key", "unique_states"}, &iteratorForJobInsertFastManyCopyFrom{rows: arg})
} |
efcd2cd to
3db7a7e
Compare
Contributor
Author
|
@bgentry Alright, I managed to resolve the copyfrom problem by leveraging pgx's generated interface once again and putting some code in I'm still writing tests for I also rebuilt the |
0b8592e to
842cc2c
Compare
Contributor
Author
|
Alright, new tests in place for |
c9a4459 to
4411a48
Compare
brandur
added a commit
that referenced
this pull request
May 10, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 10, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 10, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 10, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 10, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 10, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 10, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 11, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 14, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 14, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 14, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 14, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 14, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 14, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 14, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
brandur
added a commit
that referenced
this pull request
May 14, 2025
Here, add an implementation for SQLite. The new driver passes the entire driver test suite (including some newly added tests), and checks out against a new end-to-end suite of client tests targeting it. I think at this point we can consider it to be largely functional, although with minimal real-world vetting, but at least alpha quality that we can start putting in front of users. The main reason this turned out to be quite an effort is that SQLite, although nominally supporting a lot of the same syntax Postgres does, just has a lot of unexpected limitations and quirks that all need to be tracked down separately and a workaround added. For example: * SQLite doesn't really have data types. Everything is either an integer, real, text, or blob, and anything more sophisticated just has to be shoehorned into one of these types. For example, there's no boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb gets pushed into a blob. * The most annoying missing data type is a timestamp. Date/times are stored as either Unix integers or strings, and SQLite provides a number of obnoxious, error-prone, built-ins to work with them. * No arrays. * No modification operations in CTEs. (Although counting my lucky stars that CTEs are supported at least.) * No listen/notify. Aside from that though, we had to get some testing infrastructure in place, and our testing has traditionally been quite affixed to Postgres, which would've made a lot of changes necessary. However, the schema additions in #848 do help to make this quite a bit easier with the driver-based `TestSchema` and `TestTx` functions, and because they raise schema automatically, it doesn't complicate test setup instructions in any way by adding extra steps. A new `TestDriverRiverSQLite` test has been added to run the full barrage of driver tests for SQLite. I've also added a new family of `TestClientWithDriver` tests that are similar to `TestDriver*` in that they run for each of our supported drivers, but in this case they do a series of end-to-end tests to more fully exercise the entire River client. I added a respectable set of base client driver tests that walks each driver through all basic functionality (including for features that are higher risk of failing from one database compared to the next like `JobList`), but as with most tests, they could stand to be extended further, which I'm sure we'll get to in the future. An example test for SQLite has been added, demonstrating the use of River with a SQLite driver and in-memory SQLite database. A few caveats: * I've found that the combination of SQLite's reduced capabilities + sqlc's bugginess [1] make bit batch insert/update operations basically impossible (I tried every workaround I could possibly thin of), so these are done in loops of individual operations instead. I *think* this is okay for now though. For one, sqlc will hopefully get these limitations fixed eventually, and for two, SQLite databases will often be running locally, meaning the round trip cost per operation is much lower than what we'd see in a hosted Postgres somewhere. * It's not totally clear that having a SQLite implementation will be "worth it enough" in the long run given that it will add some difficulty to non-trivial future database-related operations. My hope is that as it's in a prerelease state, we can gauge how bad it is to keep up to date. If it feels like way more effort than it's worth, we can still axe it before it ever becomes a functional driver. [1] sqlc-dev/sqlc#3802 (comment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Here, introduce a full version of schema injection as initially started
in #798, and exposing a new
Schemaoption to make it configurable evenoutside the test suite.
This involved fixing a lot of bugs from #798, and the only way to make
it possible to root them all out was to make full use of schemas across
the entire test suite. The original "test DB manager" system has been
replaced with a new
riverschematest.TestSchemahelper that generates aschema for use with a test case or prefers an existing idle one that was
already generated for the same test run.
TestSchemaruns migrations for generated schemas which also means wedon't need to use
testdbmananymore, with tests capable ofbootstrapping themselves at run time. We reuse schemas to avoid this
extra work when possible, but migrating a new schema is also
surprisingly fast, taking up to 50ms, but getting more down to a stable
~10ms once things are warmed up. Here's a series of sample timings:
Removal of the "test DB manager" system also means that we can ungate
test from
-p 1because they're all able to run in parallel now. Thelimiting factor I ran in is that we need to keep max pool connections
within each package's tests to a relatively modest number (I found 15
seemed to maximum success) so parallel packages don't exceed the default
Postgres configuration of 100 connections.
Something that can be kind of annoying is that in case a schema isn't
used properly somewhere in a test case (i.e.
TestSchemais run, butthen not used), inserts/operations will go the default schema, which
will leave debris there, and that will interfere with test cases using
TestTx(with test DB manager, all debris would go to a differentdatabase so you wouldn't notice). To remediate this, I've added a
cleanup hook to
TestSchemathat looks for leftovers that may have beenadded to the default schema. This isn't perfect because those leftovers
may have come from another test case running in parallel or which ran
previously, but it helps to zero in on the original source of the issue.