~l3kn/org-fc#43: 
org-fc/awk/utils.awk not working on windows

I type org-fc-review and select all. Instead of cards following error appears:

org-fc/awk/utils.awk:18: fatal: 3 is invalid as number of arguments for strftime

gawk -Wversion :

gawk -Wversion
GNU Awk 5.3.0, API 4.0, (GNU MPFR 4.0.2, GNU MP 6.1.2)
Copyright (C) 1989, 1991-2023 Free Software Foundation.

GNU AWK and other dependencies (find, xargs) were installed in the following way

scoop install unxutils
scoop install gawk
scoop reset gawk

scoop is modern package manager for windows — https://scoop.sh

NB: Scoop would also recommend to install uutils-coreutils instead of unmaintained unxutils, but uutils-coreutils package doesn't include findutils like find or grep yet. Maybe I will report this as issue in the scoop repo soon.

Status
REPORTED
Submitter
~artsi0m
Assigned to
No-one
Submitted
1 year, 1 month ago
Updated
6 months ago
Labels
No labels applied.

~artsi0m 1 year, 1 month ago

Might be permissions issue, here is errors from `Messages buffer

Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2023/03
Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2023/06
Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2023/07
Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2023/08
Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2023/09
Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2023/10
Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2023/11
Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2023/12
Access denied - C:/USERS/ARTSI0M/APPDATA/ROAMING/HOWM/2024/01
File not found - -TYPE
File not found - F
File not found - -NAME
File not found - -NOT
File not found - -NAME
Access denied - .GIT
File not found - -PRINT0
gawk: c:/Users/artsi0m/AppData/Roaming/.emacs.d/src/org-fc/awk/utils.awk:18: fatal: 3 is invalid as number of arguments for strftime

~l3kn 1 year, 1 month ago

Regarding the AWK issue, the documentations lists the function as having 1 to 3 arguments: https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html#index-strftime_0028_0029-function-_0028gawk_0029

I'd like to remove the date calculations in AWK at some point and pass timestamps from Emacs so the end of a day and the different time-spans can be configured but that will take some time.

The messages look very odd, like one of the commands is interpreting the flags it's getting as files. I suspect it's the find command.

You can try running (shell-command-to-string "find --version") to check if the correct find is being used.

~artsi0m 1 year, 1 month ago

(shell-command-to-string "find --version") will return: : FIND: Parameter format not correct

And I guess that it is windows internal find, because: (shell-command-to-string "where find") will return:

: C:\Windows\System32\find.exe
: C:\Users\artsi0m\scoop\shims\find.exe

~l3kn 1 year, 1 month ago

As a hotfix, you can try overwriting the current code with something like this:

(defun org-fc-awk--find (paths)
  (format
   "C:\Users\artsi0m\scoop\shims\find.exe -L %s -type f -name \"*.org\" -not -name \".*\" -print0"
   (mapconcat
    (lambda (path) (shell-quote-argument (expand-file-name path)))
    paths " ")))

I'm not sure how a more general solution could look like.

~artsi0m 1 year, 1 month ago

I would try it soon, but just rebooted into OpenBSD for friday night.

Sorry, that I didn't figured this out myself and for the late response.

~artsi0m 1 year, 1 month ago

I tried and I've got a cryptic error:

*** Read error ***  Non-hex character used for Unicode escape: s (115)

~l3kn 1 year, 1 month ago

Ah, I just copied your path into the command but we'll need to use "\" instead of "" in the path. In the current form, it would expect the first "\Us" to be some code for unicode character.

Probably something like this:

(defun org-fc-awk--find (paths)
  (format
   "C:\\Users\\artsi0m\\scoop\\shims\\find.exe -L %s -type f -name \"*.org\" -not -name \".*\" -print0"
   (mapconcat
    (lambda (path) (shell-quote-argument (expand-file-name path)))
    paths " ")))

~artsi0m 1 year, 1 month ago

Yeah, thanks, that changed error messages. I guess using unxutils wasn't really a good idea after all.

if: Org-fc shell error: C:\Users\artsi0m\scoop\apps\unxutils\current\usr\local\wbin\find.exe: invalid predicate `-L'
gawk: c:/Users/artsi0m/AppData/Roaming/.emacs.d/src/org-fc/awk/utils.awk:18: fatal: 3 is invalid as number of arguments for strftime

~l3kn 1 year, 1 month ago

If I remember correctly, there are people using org-fc on Windows with MYSYS.

Alternatively if you're not using symlinks, you can remove the "- L" from the find command. As for gawk and srftime, if you want something that works right now, you can just remove the , 1) in the two places it's used in utils.awk and index.awk. Depending on your timezone, that will mess the review statistics in the dashboard but there should be no other effect on org-fc.

Removing strftime from the awk code would have some other benefits but I'm not sure when I'll find the time to look into that.

~l3kn 1 year, 1 month ago

Where did you get your version of unxutils and when was it last updated? https://en.wikipedia.org/wiki/UnxUtils says it hasn't been updated in 10 years.

~artsi0m 1 year, 1 month ago

Where did you get your version of unxutils and when was it last updated? https://en.wikipedia.org/wiki/UnxUtils says it hasn't been updated in 10 years.

From scoop, https://scoop.sh As I said it before, it showed me notification about the fact, that I use very old package.

~artsi0m 1 year, 1 month ago

If I remember correctly, there are people using org-fc on Windows with MYSYS.

I may try to use emacs from wsl. I remember launching graphical session of emacs from wsl once.

~artsi0m 1 year, 1 month ago*

I tried to use emacs with org-fc from wsl and it was successful. I was using OpenSUSE-Tumbleweed and following guide this two1 guides2.

~artsi0m 1 year, 17 days ago*

I tried to use wsl, but failed. I am thinking about executing find and xargs as powershell commands.

What I get after applying my changed and executing org-fc-review:

if: Org-fc shell error: xargs: invalid option -- 'g'
Try 'xargs --help' for more information.
find: ‘c:/Users/artsi0m/howm/2023/02’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2023/03’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2023/06’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2023/07’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2023/08’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2023/09’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2023/10’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2023/11’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2023/12’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2024/01’: No such file or directory
find: ‘c:/Users/artsi0m/howm/2024/02’: No such file or directory

~artsi0m 1 year, 10 days ago

I added workarounds for org-fc, some functions that redefine org-fc functions invoking find and xargs with powershell 7 commandlets.

Now I can run org-fc-dashboard on windows, but it seems to me that org-fc can't find my cards, because org-fc-dashboard displays that I have 0 cards and I get no cards due to review when I ran org-fc-review-all.

Please, can you give me some hints or some other form of help, so I can run org-fc on windows ?

~l3kn 1 year, 9 days ago

If the review history shows up in the dashboard, we at least know that awk is working, maybe there's some issue with find?

The way you are rewriting org-fc-awk--find the paths argument is never used but if the path you set manually is correct, running (org-fc-awk--find '()) by itself should return some list of files.

Will org-fc-awk--pipe work with your redefined commands? It just inserts | characters between the individual arguments so there's a chance that while your overwrites work individually, they won't work when combined by these |.

~artsi0m 1 year, 8 days ago*

Running (org-fc-awk--find '()) just returns command by itself. If I execute this command in terminal it would return list of all org files in howm. Pipes are present in pwsh, but they may not work exactly like in unix shell.

Sorry for the late response.

~artsi0m 11 months ago

I think I made it work. Now I can run org-fc-dashboard and it seems to me that org-fc finally can find my cards. I also reviewed all my cards two times on windows.

Here is parts my config where I changed org-fc functions: https://github.com/artsi0m/emacs-organizer/tree/da2ce20a146738aec75592e17148f695e017b7bb?tab=readme-ov-file#try-to-use-busybox-from-scoop

Sometimes error with find and short sexpr would appear in Messages, but I cannot reproduce it now and everything seems to be working.

~artsi0m 11 months ago

The error that I get in Messages sometimes:

(:path "-" :cards (
  )  :title nil)

~l3kn 11 months ago

Nice that you got it to work!

I now have access to a windows computer and once I have set up Emacs, I can try to reproduce these bugs.

Regarding the error message you're seeing, something similar seems to happen in the org-fc-cache-hashes function when it's used with directories that contain no flashcards. In your case, there are probably some of these in your org-fc-directories list.

~artsi0m 11 months ago

I now have access to a windows computer and once I have set up Emacs, I can try to reproduce these bugs.

The most easier way to setup emacs on windows is typing winget install GNU.EMacs

But there is also emacs in scoop (scoop.sh), which i used for installing busybox and other commandline apps, that can be called from terminal. It solves wingets problem with finding program in PATH.

Regarding the error message you're seeing, something similar seems to happen in the org-fc-cache-hashes function when it's used with directories that contain no flashcards. In your case, there are probably some of these in your org-fc-directories list.

Probably, there are some. But I can review cards without this error, when executing org-fc-review in *Messages* buffer instead of opened org buffer with flashcards.

~artsi0m 10 months ago

I am sorry. I subscribed to org mode mailing list and have been told there that by removing escaping I can get vulnerability. Also problems with shell-quote-argument highly likely have been caused by myself setting shell-file-name to pwsh.exe I will remove this setting from my config and go back to having shell-quote-argument in org-fc functions.

~artsi0m 8 months ago

Suddenly it's all stopped working again. Either No cards due to review or *.tsv file not found

~artsi0m 6 months ago

It is not working even on GNU/Linux. Should I submit new issue about interconnecting howm and org-fc ?

~l3kn 6 months ago

Are you seeing any error messages or did it just stop working?

~artsi0m 6 months ago

"~l3kn" outgoing@sr.ht writes:

Are you seeing any error messages or did it just stop working?

Sorry, for the late response and incorrect report in the first place.

Yes, I saw error messages. org-fc-dashboards is working although it shows zero cards and when I tried to execute org-fc-review-all it returns "No cards due to riview".

~l3kn 6 months ago

If the dashboard works and shows some review statistics, we at least know that awk is working. Finding and processing files with flashcards requires find and grep, so there might be issues with those.

A simple test is to check if the following gives you a list of files (or some error message that could help us debug this)

(shell-command-to-string (org-fc-awk--find org-fc-directories))

~artsi0m 6 months ago

Yes it gives an error message:

*** Eval error *** Symbol’s function definition is void: org-fc-awk--find

Should I create a new issue because it is being tested on GNU/Linux now, but I use howm emacs package to store files with flashcards ?

~l3kn 6 months ago

The function should be defined here https://git.sr.ht/~l3kn/org-fc/tree/main/item/org-fc-awk.el#L41 and the org-fc-awk file should be loaded when loading org-fc (https://git.sr.ht/~l3kn/org-fc/tree/main/item/org-fc.el#L32)

I'm not sure if howm is the issue then. C-h f should show some functions starting with org-fc-awk-..., otherwise (require 'org-fc-awk) might fail for some reason.

~artsi0m 6 months ago

I'm not sure if howm is the issue then. C-h f should show some functions starting with org-fc-awk-...,

Yes, it is showing up.

~l3kn 6 months ago

Is org-fc-awk--find among them? And if so, where is it defined? I checked the git history and this function has been in org-fc for some years now.

~artsi0m 6 months ago

Is org-fc-awk--find among them?

Yes. It is. It's defined in org-fc-awk.el

Register here or Log in to comment, or comment via email.