Some servers might use non-utf8.
This quick hack helps detect any that aren't. I haven't found any non-utf8 servers so far.
if let Some(content_type) = head.headers.get("content-type") { // TODO: write a proper parser for content types. let content_type = std::str::from_utf8(content_type.as_bytes()).unwrap(); let content_type = content_type.to_lowercase(); let found = content_type.contains("charset=\"utf-8\"") || content_type.contains("charset=utf-8"); if !found { warn!("Response does not seem to be utf-8. Content Type: {content_type}",); } };
I should probably set the
Accept
header too: https://www.rfc-editor.org/rfc/rfc9110#field.accept
Likely want to use: https://docs.rs/encoding_rs/latest/encoding_rs/