~justinludwig

Seattle

https://blog.swwomm.com/


#2 alerts for "xyz and 0 others" when throttling just 1 item 8 months ago

Ticket created by ~justinludwig on ~arx10/furemcape

Description


When throttling just one item, the alert email has a subject line like "furem cape alert level 7 and 0 others". The "and 0 others" part is confusing and pointless -- why not just send an email with a regular (non-throttled) subject and body?

Environment


  • Furem Cape master+g22a7df9
  • Python 3.7.5
  • Ubuntu 19.10: Linux furemcape2 5.3.0-1009-aws #10-Ubuntu SMP Thu Dec 5 04:55:12 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Repro Steps


  1. Set up email responder to throttle after 5 issues in under 1 minute.
  2. Trigger exactly 6 issues (like 6 "Resource Not Seen Before" issues) to be emailed in under 1 minute.

Expected Result


6 normal alert emails (eg with subjects like "furem cape alert level 7: Resource Not Seen Before").

Actual Result


5 normal alert emails (eg with subjects like "furem cape alert level 7: Resource Not Seen Before") -- and 1 throttled email with a subject like "furem cape alert level 7 and 0 others".

#1 timeout error when stopping feeder 8 months ago

Ticket created by ~justinludwig on ~arx10/furemcape

#Description

When run as a systemd service, the feeder component sometimes won't stop with a regular TERM (-15) signal, and instead requires a KILL (-9) signal.

#Environment

  • Furem Cape master+g22a7df9
  • Python 3.7.5
  • systemd 242 (242): +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid
  • Ubuntu 19.10: Linux furemcape2 5.3.0-1009-aws #10-Ubuntu SMP Thu Dec 5 04:55:12 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

#Repro Steps

  1. Start feeder: sudo systemctl start furemcape-feeder.service
  2. Stop feeder: sudo systemctl stop furemcape-feeder.service
  3. Show last 10 log messages for feeder: journalctl -u furemcape-feeder -n 10

#Expected Result

Feeder stops right away, with log messages like the following:

Jan 25 22:21:50 furemcape2 systemd[1]: Stopping Furem Cape feeder component...
Jan 25 22:21:50 furemcape2 systemd[1]: furemcape-feeder.service: Main process exited, code=killed, status=15/TERM
Jan 25 22:21:50 furemcape2 systemd[1]: furemcape-feeder.service: Succeeded.
Jan 25 22:21:50 furemcape2 systemd[1]: Stopped Furem Cape feeder component.

#Actual Result

Feeder not stopped for another 30 seconds, with log messages like the following:

Jan 25 00:30:26 furemcape2 systemd[1]: Stopping Furem Cape feeder component...
Jan 25 00:30:26 furemcape2 systemd[1]: furemcape-feeder.service: Main process exited, code=killed, status=15/TERM
Jan 25 00:31:56 furemcape2 systemd[1]: furemcape-feeder.service: State 'stop-final-sigterm' timed out. Killing.
Jan 25 00:31:56 furemcape2 systemd[1]: furemcape-feeder.service: Killing process 7263 (twistd) with signal SIGKILL.
Jan 25 00:31:56 furemcape2 systemd[1]: furemcape-feeder.service: Failed with result 'timeout'.
Jan 25 00:31:56 furemcape2 systemd[1]: Stopped Furem Cape feeder component.