Lightnews — Scholar-powered news

fjallrs.bsky.social @fjallrs.bsky.social · 1d

Without having numbers (right now), LMDB will probably read absurdly fast, while not so much space- or write efficient.
Most, if not all data, fits into RAM here. Would be interesting to benchmark 100s of GBs.

But I found redb to be slower (sometimes much slower) than LMDB, pretty much always.

1 1 2

fjallrs.bsky.social @fjallrs.bsky.social · 1d

And here's another read-heavy bench with 4K values, 5% sync random updates and 95% Zipfian reads.

ReDB uses 4x more disk space, and writes 6x slower. Interestingly, it's also ~5x slower in point reads - I think ReDB is not handling large values well; haven't read too much into its implementation.

1 2

fjallrs.bsky.social @fjallrs.bsky.social · 3d

Synchronous writes (fsync) I might add

1

fjallrs.bsky.social @fjallrs.bsky.social · 3d

95% Zipfian reads, 5% random updates on a Kingston PLP SSD with 16K JSON blobs - sled DNF! (OOM)

fjall uses LZ4 compression

2 3

Reposted

fig (aka: [phil]) @bad-example.com · 4d

constellation runs on a bare metal server with 4 dedicated CPU cores, 16GiB memory, 1TiB fast NVMe SSD attached, and 500mbit unmetered network connection.

Why @why.bsky.team · 4d

turns out raspberry pis are really fast these days

2 3 45

fjallrs.bsky.social @fjallrs.bsky.social · 7d

**slightly slower in reading

2

fjallrs.bsky.social @fjallrs.bsky.social · 7d

*bench tool

1

fjallrs.bsky.social @fjallrs.bsky.social · 7d

sled (bloodstone) is not in the bench too rn, because you can't compile it next to sled w/ compression; plus it's super unstable - in general I found it to be faster in writing, smaller on disk, but similar or slightly slower in writing than sled 0.x...

2 2

fjallrs.bsky.social @fjallrs.bsky.social · 7d

Blob GC working again (now fully automatic) - much less impact on foreground work (old GC was stop-the-world)

1 8

fjallrs.bsky.social @fjallrs.bsky.social · 10d

Hopefully I will have results in 6 hours 👀

1 3

fjallrs.bsky.social @fjallrs.bsky.social · 10d

Noo 🥲

I wanted to expand it to a 1-hour run but it crashed because of a bug in the workload (I think), in the middle of the night.

1 2

fjallrs.bsky.social @fjallrs.bsky.social · 10d

Fun coloured bars and lines

3 4 6

Reposted

fjallrs.bsky.social @fjallrs.bsky.social · 16d

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!!

jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing!

bye

Sponsor @fjall-rs on GitHub Sponsors

Support fjall-rs's open source work

github.com

6 8

fjallrs.bsky.social @fjallrs.bsky.social · 16d

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!!

jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing!

bye

Sponsor @fjall-rs on GitHub Sponsors

Support fjall-rs's open source work

github.com

6 8

fjallrs.bsky.social @fjallrs.bsky.social · 16d

Looks like the default fillfactor is 100%, which should be good for disk space. So it's probably dead row overhead and lack of page compression (?) plus maybe secondary indexes.

1 1

fjallrs.bsky.social @fjallrs.bsky.social · 18d

But yeah, it feels like things are slotting into place.

I have more than 50 git stashes and branches floating around... it's been a bit messy, but getting there.

fjallrs.bsky.social @fjallrs.bsky.social · 18d

I think I was lucky to fall into the pit of success because I think I made more progress the last two days than the last two months combined...

V3 pre-release(s) very soon... 👀

1 2

fjallrs.bsky.social @fjallrs.bsky.social · 20d

Add to that, sweeping API changes for better ergonomics that easily touch over a hundred files - Rust's doctests are a godsend here, I would have gone insane already without them

1 2

fjallrs.bsky.social @fjallrs.bsky.social · 20d

ngl v3 is a bit more involved than I originally planned or anticipated; I don't want breaking changes for a while, so it becomes this chase of making sure everything is really future-proof