[PATCH] Re: out of memory error after "edit-command-line"

[Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>]

On Fri, Aug 12, 2022 at 1:18 PM Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx> wrote:
>
> However, I can't come up with a minimal test case to invoke the
> initial condition.  It's not enough just to do e.g.
> ${(0A)$(</proc/$$/cmdline)} in isolation.
> [...] it appears that findsep(&s, NULL, 0)
> considers '\203 ' to be a separator (because a null byte is in $IFS ?)
> but skipwsep() does not.

OK, that's a red herring.  The real problem is that itype_end(s, ISEP,
1) is not skipping over the '\203 ' pair in Jim's example, whereas it
does in the simple example above.

Once this becomes broken, it remains broken -- the simple example
starts infinite-looping as well.

It seems to come down to this in itype_end():
4359                case ISEP:
4360                if (!wmemchr(ifs_wide.chars, wc, ifs_wide.len))
4361                    return (char *)ptr;

On entry to that case in the simple example, wmemchr() returns nonzero
for wc == 0.  After edit-command-line, wmemchr() starts returning zero
in that case.  It appears ifs_wide has been erased.

The problem starts here in edit-command-line:

  # Compute the cursor's position in bytes, not characters.
  setopt localoptions nomultibyte noksharrays

When nomultibyte is set, inittyptab() is called and erases ifs_wide.
This is not restored when emulation mode ends and multibyte is
re-asserted.

The following patch fixes this example, but might only be a partial
fix for problems with locally flipping the state of various options
(MONITOR, BANGHIST, SHINSTDIN come to mind).  I think really we should
be looping over the options and calling dosetopt() for each one
instead of just memcpy'ing the saved set back on top of the original
... or at least we need a mapping of the subset of options that have
extra code associated with a change via setopt.

diff --git a/Src/exec.c b/Src/exec.c
index f2911807c..77763f536 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -5962,10 +5962,17 @@ doshfunc(Shfunc shfunc, LinkList doshargs, int noreturnval)
 	    sticky = funcsave->sticky;
 	} else if (isset(LOCALOPTIONS)) {
 	    /* restore all shell options except PRIVILEGED and RESTRICTED */
+#ifdef MULTIBYTE_SUPPORT
+	    int flip_multibyte = opts[MULTIBYTE];
+#endif
 	    funcsave->opts[PRIVILEGED] = opts[PRIVILEGED];
 	    funcsave->opts[RESTRICTED] = opts[RESTRICTED];
 	    memcpy(opts, funcsave->opts, sizeof(opts));
 	    emulation = funcsave->emulation;
+#ifdef MULTIBYTE_SUPPORT
+	    if (flip_multibyte != opts[MULTIBYTE])
+		inittyptab();
+#endif
 	} else {
 	    /* just restore a couple. */
 	    opts[XTRACE] = funcsave->opts[XTRACE];