How to avoid echo to native Terminal?

[vorburger]

With the following code, using this StreamPumper from here:

String[] cmd = {"/bin/sh", "-l"};
Map<String, String> env = new HashMap<>(System.getenv());
if (!env.containsKey("TERM")) env.put("TERM", "xterm-256color");
PtyProcess process = new PtyProcessBuilder().setCommand(cmd).setEnvironment(env).start();

OutputStream os = process.getOutputStream();
new StreamPumper("In", System.in, os);

InputStream is = process.getInputStream();
new StreamPumper("Out", is, System.out);

return process.waitFor();

I'm getting an "echo" that I (conceptually) don't quite understand how to get rid of when using Pty4j:

[main] INFO com.pty4j.util.ExtractedNative - Extracted pty4j native in 8 ms
...$ ls
ls
...

I could put a fully running standalone reproducer project e.g. on https://github.com/vorburger/LearningPty4J, if there is interest.

[vorburger]

BTW I don't think this has anything to do with the aforementioned StreamPumper per-se, it's reproducible with a simpler MVP one:

class SimpleStreamPumper extends Thread {
        SimpleStreamPumper(InputStream is, OutputStream os) {
            super(
                    () -> {
                        try {
                            int b;
                            while ((b = is.read()) != -1) {
                                os.write(b);
                                os.flush();
                            }
                        } catch (IOException e) {
                            e.printStackTrace();
                        }
                    });
            start();
        }
    }

[segrey]

@vorburger Sorry, not sure I fully understood the issue.

I could put a fully running standalone reproducer project e.g. on https://github.com/vorburger/LearningPty4J, if there is interest.

Yes, it'd be great. Alternatively, you can describe the issue in the form of a failing test. See PtyTest#testExecCat as a test example.

[vorburger]

@segrey I've learnt a bit more about this space over this weekend (see e.g. https://github.com/vorburger/vorburger.ch-Notes/blob/develop/linux/pty-tty.md).

I'm wondering if this might be an issue that's specific to using System.in & System.out - as opposed to e.g. piping Pty4j's IN & OUT into like a Swing (or Web) Terminal - which I'm guessing is how others use this library.

In enola-dev/enola#1413 I've attempted to go through https://jline.org/docs/terminal, based on a hunch that this may make a difference - but unfortunately it apparently does not.

I'll try to "front" it with a SSH server (with https://jline.org/docs/remote-terminals) to see if that's any different.

Further thoughts & inputs very welcome.

[segrey]

Oh, I think I got what you meant. The problem is that when you start your Java application from terminal, its System.in and System.out is connected to PTY in canonical / cooked mode. In this mode, keypresses are processed and buffered by PTY and pressing Enter flushes the buffered line to the app's System.in. Therefore, when you type ls, first it's typed because of enabled echo in your system terminal, then when you press Enter, ls is passed to the spawned process which also echoes it and you see ls back.

You need to enable raw mode to process all keypresses by your Java app. For example, this code snippet works for me and requires dependency on JLine (implementation("org.jline:jline:3.23.0")).

    String[] cmd = {"zsh"};
    Map<String, String> env = new HashMap<>(System.getenv());
    if (!env.containsKey("TERM")) env.put("TERM", "xterm-256color");
    PtyProcess process = new PtyProcessBuilder().setCommand(cmd).setEnvironment(env).start();

    try (Terminal terminal = TerminalBuilder.builder()
      .streams(System.in, System.out)
      .system(true)
      .build()) {

      terminal.enterRawMode();

      OutputStream os = process.getOutputStream();
      new StreamPumper2("In", terminal.reader(), new OutputStreamWriter(os));

      InputStream is = process.getInputStream();
      new StreamPumper2("Out", new InputStreamReader(is), terminal.writer());

      return process.waitFor();
    }

Hope it will help.

[vorburger]

@segrey Duh; I ALMOST had it (in enola-dev/enola#1413); I had actually tried raw mode initially, but then didn't see any output anymore (that's why there was an "it's worse" inline comment). Thinking back why, it must have been because I had originally used System.in & System.out, then later realized that (with JLine) it should of course being using terminal - but then hadn't retried with raw mode earlier today. It now works for me as well (of course); see enola-dev/enola#1416.

BTW perhaps terminal.input() & terminal.output() is "safer" than terminal.reader() & terminal.writer()? It seems to work great as well, in my testing. (I'm thinking of a scenario say if this is used in something more "generic" (than a Terminal Emulator), where you want to be able to launch a shell or editor, but also perhaps some other executable which may return binary data - in such cases you wouldn't want Character-based Reader/Writer to muck with your bytes. The "something generic" is because I might one day "plug" this into https://github.com/vorburger/ch.vorburger.exec/, with some sort of generic API like this.)

#172 raised to document this and make it easier for the next person coming along.

Also jline/jline3#1295 raised for @gnodet.

[segrey]

BTW perhaps terminal.input() & terminal.output() is "safer" than terminal.reader() & terminal.writer()?

Yes, definitely, you're right, thanks.

[segrey]

@vorburger BTW, you might want to use this.is.transferTo(this.os); in the StreamPumper here.