The switch away from a bitmap layer helped, but the artifacts seen in https://gitlab.com/bztsrc/scalable-font2/-/issues/19 are still an issue.
Optimizing the vector layers would solve.