Generate a cpio with request logs

Instead of creating a file for each request log, create a cpio archive on startup and keep appending to it. Most of the time, the logs aren't used for anything and deleting a single file is faster/simpler than deleting a direction with hundreds of thousands of tiny files.

The archive name could be something like %Y%M%DT%H%M%S.%nZ_${tid}.cpio. Making per-thread logs will avoid unnecessary synchronization.

The names of the files inside the cpio can remain the same.

Assigned to
11 months ago
10 months ago

~jeffpc 11 months ago

Need to make sure that cpio files without the "TRAILER!!!" dummy file at the end can still be extracted by cpio(1).

~jeffpc 11 months ago

FreeBSD pax(1) reacts to the missing trailer record as an end-of-tape and requests that filename with the next archive be provided:

$ pax -r -f .../path/to/archive.cpio
pax: End of archive volume 1 reached

ATTENTION! pax archive volume change required.
Ready for archive volume: 2
Input archive name or "." to quit pax.
Archive name >

The contents appear to extract ok.

A quick prototype of the idea seems to work ok. It is a bit unfortunate that the thread ids (as returned by xthr_self) are pointer-like on FreeBSD and Linux and therefore look "ugly":

-rw-r--r--  1 jeffpc  jeffpc    11K Oct 23 22:55 1666580133.205537487-0000000800a15800.cpio
-rw-r--r--  1 jeffpc  jeffpc    21K Oct 23 22:56 1666580156.857663212-0000000800a14a00.cpio
-rw-r--r--  1 jeffpc  jeffpc    22K Oct 23 23:08 1666580165.063394684-0000000800a14300.cpio
-rw-r--r--  1 jeffpc  jeffpc    20K Oct 23 23:08 1666580898.261793560-0000000800a15100.cpio

However, using the thread id like this allows correlating the archive with any log messages.

Finally, the cpio header is 76 bytes and the filename is 43 bytes (including the trailing nul). This is much less than the typical file system overhead of creating a new file for each request log entry.

~jeffpc 10 months ago

It probably makes sense to make this change depend on having a config knob to enable/disable request logging (~jeffpc/blahgd#19).

Register here or Log in to comment, or comment via email.