~justinludwig

Seattle

https://blog.swwomm.com/


#2 alerts for "xyz and 0 others" when throttling just 1 item 5 months ago

Ticket created by ~justinludwig on ~arx10/furemcape

Description

When throttling just one item, the alert email has a subject line like furem cape alert level 7 and 0 others. The and 0 others part is confusing and pointless -- why not just send an email with a regular (non-throttled) subject and body?

Environment
  • Furem Cape master+g22a7df9
  • Python 3.7.5
  • Ubuntu 19.10: Linux furemcape2 5.3.0-1009-aws #10-Ubuntu SMP Thu Dec 5 04:55:12 UTC 2019 x8664 x8664 x86_64 GNU/Linux
Repro Steps
  1. Set up email responder to throttle after 5 issues in under 1 minute.
  2. Trigger exactly 6 issues (like 6 Resource Not Seen Before issues) to be emailed in under 1 minute.
Expected Result

6 normal alert emails (eg with subjects like furem cape alert level 7: Resource Not Seen Before).

Actual Result

5 normal alert emails (eg with subjects like furem cape alert level 7: Resource Not Seen Before) -- and 1 throttled email with a subject like furem cape alert level 7 and 0 others.

#1 timeout error when stopping feeder 5 months ago

Ticket created by ~justinludwig on ~arx10/furemcape

Description

When run as a systemd service, the feeder component sometimes won't stop with a regular TERM (-15) signal, and instead requires a KILL (-9) signal.

Environment
  • Furem Cape master+g22a7df9
  • Python 3.7.5
  • systemd 242 (242): +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid
  • Ubuntu 19.10: Linux furemcape2 5.3.0-1009-aws #10-Ubuntu SMP Thu Dec 5 04:55:12 UTC 2019 x8664 x8664 x86_64 GNU/Linux
Repro Steps
  1. Start feeder: sudo systemctl start furemcape-feeder.service
  2. Stop feeder: sudo systemctl stop furemcape-feeder.service
  3. Show last 10 log messages for feeder: journalctl -u furemcape-feeder -n 10
Expected Result

Feeder stops right away, with log messages like the following:

Jan 25 22:21:50 furemcape2 systemd[1]: Stopping Furem Cape feeder component...
Jan 25 22:21:50 furemcape2 systemd[1]: furemcape-feeder.service: Main process exited, code=killed, status=15/TERM
Jan 25 22:21:50 furemcape2 systemd[1]: furemcape-feeder.service: Succeeded.
Jan 25 22:21:50 furemcape2 systemd[1]: Stopped Furem Cape feeder component.
Actual Result

Feeder not stopped for another 30 seconds, with log messages like the following:

Jan 25 00:30:26 furemcape2 systemd[1]: Stopping Furem Cape feeder component...
Jan 25 00:30:26 furemcape2 systemd[1]: furemcape-feeder.service: Main process exited, code=killed, status=15/TERM
Jan 25 00:31:56 furemcape2 systemd[1]: furemcape-feeder.service: State 'stop-final-sigterm' timed out. Killing.
Jan 25 00:31:56 furemcape2 systemd[1]: furemcape-feeder.service: Killing process 7263 (twistd) with signal SIGKILL.
Jan 25 00:31:56 furemcape2 systemd[1]: furemcape-feeder.service: Failed with result 'timeout'.
Jan 25 00:31:56 furemcape2 systemd[1]: Stopped Furem Cape feeder component.