Lightnews — Scholar-powered news

fjallrs.bsky.social @fjallrs.bsky.social · 20h

Synchronous writes (fsync) I might add

fjallrs.bsky.social @fjallrs.bsky.social · 20h

95% Zipfian reads, 5% random updates on a Kingston PLP SSD with 16K JSON blobs - sled DNF! (OOM)

fjall uses LZ4 compression

1 3

Reposted

fig (aka: [phil]) @bad-example.com · 2d

constellation runs on a bare metal server with 4 dedicated CPU cores, 16GiB memory, 1TiB fast NVMe SSD attached, and 500mbit unmetered network connection.

Why @why.bsky.team · 2d

turns out raspberry pis are really fast these days

2 3 45

fjallrs.bsky.social @fjallrs.bsky.social · 4d

**slightly slower in reading

2

fjallrs.bsky.social @fjallrs.bsky.social · 4d

*bench tool

1

fjallrs.bsky.social @fjallrs.bsky.social · 4d

sled (bloodstone) is not in the bench too rn, because you can't compile it next to sled w/ compression; plus it's super unstable - in general I found it to be faster in writing, smaller on disk, but similar or slightly slower in writing than sled 0.x...

2 2

fjallrs.bsky.social @fjallrs.bsky.social · 4d

Blob GC working again (now fully automatic) - much less impact on foreground work (old GC was stop-the-world)

1 8

fjallrs.bsky.social @fjallrs.bsky.social · 7d

Hopefully I will have results in 6 hours 👀

1 3

fjallrs.bsky.social @fjallrs.bsky.social · 7d

Noo 🥲

I wanted to expand it to a 1-hour run but it crashed because of a bug in the workload (I think), in the middle of the night.

1 2

fjallrs.bsky.social @fjallrs.bsky.social · 7d

Fun coloured bars and lines

3 4 6

Reposted

fjallrs.bsky.social @fjallrs.bsky.social · 13d

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!!

jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing!

bye

Sponsor @fjall-rs on GitHub Sponsors

Support fjall-rs's open source work

github.com

6 8

fjallrs.bsky.social @fjallrs.bsky.social · 13d

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!!

jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing!

bye

Sponsor @fjall-rs on GitHub Sponsors

Support fjall-rs's open source work

github.com

6 8

fjallrs.bsky.social @fjallrs.bsky.social · 13d

Looks like the default fillfactor is 100%, which should be good for disk space. So it's probably dead row overhead and lack of page compression (?) plus maybe secondary indexes.

1 1

fjallrs.bsky.social @fjallrs.bsky.social · 15d

But yeah, it feels like things are slotting into place.

I have more than 50 git stashes and branches floating around... it's been a bit messy, but getting there.

fjallrs.bsky.social @fjallrs.bsky.social · 15d

I think I was lucky to fall into the pit of success because I think I made more progress the last two days than the last two months combined...

V3 pre-release(s) very soon... 👀

1 2

fjallrs.bsky.social @fjallrs.bsky.social · 17d

Add to that, sweeping API changes for better ergonomics that easily touch over a hundred files - Rust's doctests are a godsend here, I would have gone insane already without them

1 2

fjallrs.bsky.social @fjallrs.bsky.social · 17d

ngl v3 is a bit more involved than I originally planned or anticipated; I don't want breaking changes for a while, so it becomes this chase of making sure everything is really future-proof