Someone over #dendrite:matrix.org also faces the same issue but neither cause nor fix has been suggested.
Joining time has been reasonable (on par with Synapse homeservers at least) after upgrading Dendrite a few months back.