|
@@ -6,25 +6,29 @@ PR 2 of the review fixes plan (`docs/review-fixes-plan.md`). Part of 0.5.2; the
|
|
|
|
|
|
|
|
## What changed
|
|
## What changed
|
|
|
|
|
|
|
|
-**Projects over 512 MB decompressed are refused (S2).** `xpj.Read` used to read the whole gzip stream into memory, however large it was. In the review, a 1 MB file padded with spaces used 3.1 GB and took 10 seconds. `Read` now reads through `io.LimitReader` and stops one byte past the new `xpj.MaxDecompressed` (512 MB, as decided). Past that, it returns `xpj: project is larger than 512 MB when decompressed`. A comment on the constant gives the reason. Device projects are about 3 MB decompressed, so real projects are far below the limit.
|
|
|
|
|
|
|
+**Projects over 256 MB decompressed are refused (S2).** `xpj.Read` used to read the whole gzip stream into memory, however large it was. In the review, a 1 MB file padded with spaces used 3.1 GB and took 10 seconds. `Read` now reads through `io.LimitReader` and stops one byte past the new `xpj.MaxDecompressed`, 256 MB. Past that, it returns `xpj: project is larger than 256 MB when decompressed`. A comment on the constant gives the reason.
|
|
|
|
|
+
|
|
|
|
|
+**The limit is 256 MB, not the plan's 512 MB** (bajdev, 2026-11-12, during this PR). The limit counts only the project file once it is unpacked: the pads, the pool's list of file names and the sequences with their events. WAV files are separate files in the data folder and are not read through `xpj`.
|
|
|
|
|
+- A blank project is about 3 MB, almost all of it the 128 empty pad definitions.
|
|
|
|
|
+- Each recorded event takes about 480 bytes, and an empty sequence about 12 KB.
|
|
|
|
|
+- So 256 MB holds about half a million events, around 4,000 for each of 128 sequences. DemoFile has 225 events in total.
|
|
|
|
|
+
|
|
|
|
|
+The Decisions table and PR 2's tasks in `docs/review-fixes-plan.md` now say 256 MB.
|
|
|
|
|
|
|
|
**Test.** `TestDecompressionLimit` builds its gzip streams in memory with `gzip.NewWriter`; nothing new is committed:
|
|
**Test.** `TestDecompressionLimit` builds its gzip streams in memory with `gzip.NewWriter`; nothing new is committed:
|
|
|
- A minimal project padded with spaces to one byte over the limit is refused with that exact error.
|
|
- A minimal project padded with spaces to one byte over the limit is refused with that exact error.
|
|
|
- The same project padded to exactly the limit still decodes.
|
|
- The same project padded to exactly the limit still decodes.
|
|
|
- Against the old `Read`, the over-limit case loads without error, so the test fails.
|
|
- Against the old `Read`, the over-limit case loads without error, so the test fails.
|
|
|
-- It takes about 11 seconds, and `go test -short` skips it.
|
|
|
|
|
|
|
+- It takes about 3 seconds, and `go test -short` skips it.
|
|
|
|
|
|
|
|
## Notes
|
|
## Notes
|
|
|
|
|
|
|
|
-**Memory is still high near the limit.** The limit stops the read early, but memory still grows well past 512 MB:
|
|
|
|
|
-- **Over the limit:** a file padded to 3.1 GB (6 MB compressed) is now refused after 1.3 seconds, at a peak of about 1.3 GB. The buffer `io.ReadAll` grows can reach roughly 2.5 times what it holds before the check runs.
|
|
|
|
|
-- **Exactly at the limit:** a file that passes the check still costs about 2.3 GB while it decodes, because the JSON decoder keeps its own copy of the padding.
|
|
|
|
|
-
|
|
|
|
|
-This is what the 512 MB decision implies, and it no longer grows without bound. If you want it lower, there are two options for a later PR:
|
|
|
|
|
-- Lower the limit. Device projects are about 3 MB, so 64 MB would still leave plenty of room.
|
|
|
|
|
-- Read into a buffer sized up front, which saves the growth but not the decoder's copy.
|
|
|
|
|
|
|
+**Memory near the limit, as measured on the server:**
|
|
|
|
|
+- A file padded to 3.1 GB (6 MB compressed) is now refused after 0.3 seconds, at a peak of about 600 MB. The buffer `io.ReadAll` grows can pass the limit before the check runs.
|
|
|
|
|
+- A file exactly at the limit still loads, at a peak of about 925 MB, because the JSON decoder keeps its own copy of the padding.
|
|
|
|
|
+- At 512 MB, these were about 1.3 GB and 2.3 GB.
|
|
|
|
|
|
|
|
-**The test needs memory.** `TestDecompressionLimit` peaks at about 2.3 GB while it decodes the at-the-limit file. The server ran it with no trouble alongside the other tests. Use `go test -short ./...` to skip it.
|
|
|
|
|
|
|
+**The test needs memory:** about 860 MB while it runs. `go test -short ./...` skips it.
|
|
|
|
|
|
|
|
## Testing
|
|
## Testing
|
|
|
|
|
|
|
@@ -37,6 +41,6 @@ This is what the 512 MB decision implies, and it no longer grows without bound.
|
|
|
|
|
|
|
|
**On the laptop (PowerShell, in the repo):**
|
|
**On the laptop (PowerShell, in the repo):**
|
|
|
1. `git fetch; git checkout limit-decompression`
|
|
1. `git fetch; git checkout limit-decompression`
|
|
|
-2. `go test -run TestDecompressionLimit -v ./internal/xpj`: `PASS`, in about 10–20 seconds. It uses about 2.3 GB of memory.
|
|
|
|
|
|
|
+2. `go test -run TestDecompressionLimit -v ./internal/xpj`: `PASS`, in a few seconds. It uses about 900 MB of memory.
|
|
|
3. `go test ./...`: all `ok`.
|
|
3. `go test ./...`: all `ok`.
|
|
|
4. `wails3 dev`: load `testdata\device\DemoFile.xpj` and `testdata\accepted\MpckitT1.xpj`. Both open as before.
|
|
4. `wails3 dev`: load `testdata\device\DemoFile.xpj` and `testdata\accepted\MpckitT1.xpj`. Both open as before.
|