Add cargo_metadata resolver benchmarks

Eh2406: I am working on cargos resolver, one of the things I don't have is realistic benchmarks. I would love to know which crates happen to have taken the longest amount of time to resolve. Or required the most backtracking.

Icefox: What do you mean by backtracing?

Eh2406: When it discovers that it can't use the newest version of a library. There is not way to get metrics on it (yet), but I could add one if you could make it available.

Adding timing info to this at least is not going to be that difficult, I think. The cargo metadata calls are in general relatively slow, so simple timing should be reasonably accurate, though the cargo_metadata crate has to parse the resulting JSON which will add some overhead.

But hey, that's just another number to measure.

Assigned to
1 year, 6 months ago
1 year, 5 months ago
AREA-Analysis TYPE-Feature

~icefox 1 year, 6 months ago

Trivially done, results here: http://alopex.li/temp/cargo_metadata_times_20190318.csv

Imagine every possible disclaimer against crude benchmarks, they all apply. It's literally just the amount of time it took to run the cargo_metadata library call, and so likely to be I/O bound, have overhead from parsing the JSON output, etc, etc. Still, it's a place to start.

~icefox 1 year, 5 months ago

...I want to put this info in the results but I'm not awake enough to rearrange things nicely. Hmmm.

~icefox 1 year, 5 months ago

Eh2406 reported back that the slowest resolver times were not reproducible, I expect that they were outliers spending a lot of time on disk I/O or something. Would still like a note if we discover something consistently taking >10 seconds. The current setup is really optimized for throughput though, not timing analysis.

Register here or Log in to comment, or comment via email.