Comment by ~lisael on ~mil/sxmo-tickets
(I was writting a long response, my mail client failed on me, I'm going to be brief)
Is there a reason to have our own format?
New here... But I read the code and I think I understand more corner-cases, now, I don't see a reason other than it was quicker to get a good-enough solution, that shows its age, now.
Using something that's already standardized would make sxmo much more extensible by allowing it to plug directly into other applications that already exist. I think Maildir might be a good option for this, although I haven't studied it
very closely.Very interesting idea, we could use notmuch to store the messages and
query, and build lots of feature on top of it (blacklists, attachment
management, sms acknowledgments...).I'll send a patch with my work on the format we discussed with wart in
this thread, for review, and one with a Maildir backend, to compare and
evaluate benefits and trad-offs (flat files do have their advantages too).A better way to handle getting the users number would
be really nice. When we format numbers for dialing we should take the
users country into account, and a easy source for that would be their
number.I'm not a seasoned phone hacker... I may have ideas to improve usability,
though, but nothing near a definitive solution.--
Bruno (lisael)
Comment by ~lisael on ~mil/sxmo-tickets
first little issue: incoming sms data don't contain the receiver phone number. That's why the info was not in the log in the first place. I will assume $(sxmo_contacts.sh --me) but I'm not sure what to do if it's not set. (Me is not in contacts.tsv). Worse,
Me
could be incorrect, if the user changed their SIM card.mm
AT+CNUM
is only available in debug mode (and I understand that the info is read from the SIM card, the carrier may or may not burn it into the card)One solution is to briefly restart modem manager with --debug when we detect a new SIM card, get the phone number, and save it in a file mapping sim card numbers and phone numbers. We could use this to answer to sxmo_contacts.sh --me. That's far beyond the scope of this thread, though.
Comment by ~lisael on ~mil/sxmo-tickets
Regarding the conversion script, it should not be that hard, I can parse almost every header now with my WIP on tailtextlog (I just need an hour or two of free time, that's the hardest part :))
Same for the hook...
I thought about the conversion script, the hard part will be if a contact name has changed (or was removed) from contacts.tsv. This problem itself proves that we must keep the most canonical representation of the user in the messages logs (i.e the phone number as provided in the raw data + country prefix).
Regarding your comments:
If the datetime iso format is provided by the modem/modemmanager, we should keep it as is, granted.
Regarding the target design:
I read SMS specifications, it's very unclear if
\0
is a valid SMS character. Either way, I think that prepending a white space char before each body line is the most robust format (1 space is enough and will not kill the readibility when wecat
the file).The file would look like:
sms.txt
# log version: 1 2022-02-18T06:46:49-0600 SMS +33123456789 +33987654321 hello 2022-02-18T06:46:49-0600 SMS +33987654321 +33123456789 give me your lunch! 2022-02-18T06:46:49-0600 SMS +33123456789 +33987654321 yeah! sure 2022-02-18T06:46:49-0600 SMS +33123456789 +33987654321 haha It won't work this time. 2022-02-18T06:46:49-0600 MMS +33123456789 +33987654321 📎trollface.png
(I wrote this in the web UI, there's no way to add tabs, the header is meant to be tab separated)
and group mms are the same, just with more recipients.
The hook can infer the Send/Received part depending on the position of the user's phone number, if it's the first one we are the sender, otherwise, we are the recipient.
Comment by ~lisael on ~mil/sxmo-tickets
so... yes there is a SMS RFC : https://www.ietf.org/rfc/rfc5724.txt
Ticket created by ~lisael on ~mil/sxmo-tickets
I'm working on pretty display of messages and I found that the storage format is quite messy.
At the moment headers are just special lines in the form
^(Sent|Received) SMS (to|from) <recipient_or_me> at <iso date>:$
or
^Received Voice Mail from <recipient> at <iso date>:$
or
^(Sent|Received) MMS from <recipient_or_me> at <iso date>:$
or
^(Sent|Received) Group MMS (to|from) <origin> (to|from) <recipients> at <iso date>:$
and messages is the rest between those.
Seems easy? well, not so much there is a lot of inconsistencies and oddities that bite when you try to consume this in scripts:
- In the case of SMSs,
<recipient_or_me>
is the phone number, in all other case it's a resolved contact name.- speaking of which... a phone number may be a special number... now I have
Recived SMS from La Poste at...
in my logs.[+0-9]+
is not enough to extract this andfrom (.*) at
is fragile too, the special name may containat
.- Sent SMS don't have a
from <me>
clause, MMS do, even in 1-to-1 conversations- Iso dates with timezone are OK to display to nerds, not so much for my mum, and they are awful when it comes to computations using only simple shell scripts. My mum will need us to reformat the date anyway, so why not storing it in a script-friendly format ?
- There is no clear marker of a header line. Not only the parsing is fragile, but more important there's a security implication, a sender can spoof a message. Silly example, I (+666) send you:
give me your lunch Sent SMS to +666 at 2022-02-18T06:46:49-0600: yeah! sure
Now you owe me your lunch. This is silly, but it can be worse in group chats, where a user may impersonate another one, targeting the nerdy sxmo user of the group.
my proposal is:
- find a marker that cannot be sent in SMS/MMS messages (
\0
, maybe? Is there a SMS RFC??) and use it to mark headers.- define a canonical header format using the canonical form of each field, namely: UTC timestamp for datetime, Phone numbers for senders and recipients I'd go with:
<header marker><timestamp>\t<msg_type (SMS, MMS or VVM)>\t<sender canonical id>[\t<receivers canonical ids>]+
- write a minimal formatting script to display the message logs as it is today
- write a migration script to reformat existing logs
1 is hard, possibly impossible but in my opinion important because it prevents remote attacks on the user's data. Alternatively, if there is no usable marker, I see 2 solutions:
- we can mark messages instead, prepending one tab to each message line
- move to a 1 file per message storage. The first line is the header, the rest is the message.
Most of the proposal here comes from a grumpy programmer that is tired to deal with corner-cases everywhere, it may not worth the hassle of changing the format. The security implications, on the other hand, are more than enough to call for a change (and in the process we can satisfy the grumpy programmer too :) ) .
Comment by ~lisael on ~mil/sxmo-tickets
I agree it's important. I'd like to work on a patch with a gettext PoC on an arbitrary menu to give a feeling of what it's like working with localized strings and dates in sxmo shell scripts.
If (or when..) we start with i18n/l10n, the hardest part is to document the localization process and to attract contributions.
Comment by ~lisael on ~mil/sxmo-tickets
I thought this would split a ton of logs, but it didn't:
10 18 * * * sh -c "set -x; /usr/bin/sxmo_audio.sh vol get 2>&1 | tee /home/user/forklog"
I got only one line:
/usr/bin/sxmo_audio.sh: line 142: can't create /sxmo.audiocurrentdevice: Permission denied
XDG_RUNTIME_DIR
is missing (maybe we shouldset -u
wherever possible)?EDIT, sourcing
/etc/profile.d/sxmo_init.sh
seems to fix the issue altogether18 18 * * * sh -c "set -x; . /etc/profile.d/sxmo_init.sh && /usr/bin/sxmo_audio.sh vol get 2>&1 | tee /home/user/forklog"
$ cat forklog 74
We should require that sxmo_init.sh is sourced at a bunch of strategic points.
Ticket created by ~lisael on ~mil/sxmo-tickets
Working on sxmo_alarm.sh, I noticed, since I upgraded to 1.8.0, that when I set the volume from a cron task (busybox's crond) the, my system hangs, creating thousands of processes. I could see in htop this process tree (sorry, it's an image, that's all I could get while the phone freezes): https://i.imgur.com/SbdSct8.png
To reproduce, add this in your crontab and wait for the task to run (BEWARE: a hard reboot is the only way out):
58 16 * * * /usr/bin/sxmo_audio.sh vol get > /tmp/cronvol
Note that in my screenshot, we don't see the original process, I killed it, but the fork bomb was re-parented to init.
Comment by ~lisael on ~mil/sxmo-tickets
I gave a shot at this, here's my design:
the alarm list is the crontab (no other disc writes)
an alarm cron entry is of this form (
<script> <action> <message> <uid>
):25 7 * * 1-5 sxmo_rtcwake.sh /home/user/.local/bin/alarm.sh repeat 'Wake Up!' xesEsQRV
alarm.sh has those subcommands:
ring <message>
: ring the alarm, open a menu to either stop or snoozeonce <message> <uid>
: remove the cron entry (found by uid) and call ringadd <crontime> <message> <action ("once" or "repeat")>
: add a cron tasklist
: list registered alarmsdisable <uid>
: disable an alarm (comment the line in the crontab)enable <uid>
: enable an alarm (uncomment in the crontab)remove <uid>
: remove the crontab entryrepeat <message> <uid>
: ignore the uid and callring <message>
Targets the crontab tasks, the uid has no use in the command but identifies the line in the list for managment purpose (enable/disable/remove...)snooze <message> <uid>
: an alias of once, used to differenciate snozed alarm from user-set "once" alarms in the list
- set-alarm.sh UI script:
- main menu is a list (mnc-ordered) of existing alarms (with a toggle icon to show enabled status) and a
Add
entryAdd
asks for a time (HH:MM or crontab-style),once
orrepeat
, and a message- choosing an alarm in the main menu opens an edit menu to enable/disable/remove/edit time or message)
I use working code implementing much of this design (I'm tweaking the Edit UI at the moment). It's probably not clean enough, but I may send a patch soon.