Talk:Ncat-Lua

In this wiki page I'd like to give you guys an update on Ncat's capabilities and propose a couple of new features. Their purpose is to allow better Ncat extensibility - easier adding application-layer protocol support, such as WebSocket and filters, as in my tiny IRC client demo. I also mentioned a few simple solutions to serious Ncat-Lua limitations. Since there's quite a lot of text, here's a table of contents:

Current capabilities of the filters

I believe that we're pretty close to have a complete implementation of interfaces allowing to add filter scripts. At the moment, Lua scripts can write to both the connection and Ncat's standard input and output, allowing the scripts to implement new application-layer protocols in a quite transparent way - after running the, for example, WebSocket script, the user doesn't have to care about the protocol at all, just typing text (or piping it somehow) into the terminal and reading already processed responses, which lets you forget about WebSocket whatsoever (at least until something breaks).

You can also go a step farther and implement a client using the current feature set - as an example, in one of my branches I implemented a tiny IRC client that connects to a hardcoded channel and displays all the messages in a readable form, allowing the user to type in his own input, which will then be sent to the channel. I believe it's especially fun when you take into account how little work it is to implement such a client - the code was 72 lines long, 37 if you removed comments, blanks and debug output. All portable, without platform-specific code!

Limitations of the current filters implementation

In this section I'll list all the limitations of the current implementation of Lua filters interface. Please note that it's unlikely that all the problems will be solved during this Google Summer of Code project - at the moment I'm writing this, I'm in the middle of 10th week of work, out of 16. One of the things I learned is that delivering features takes time - to give you a better picture of how this works, I wrote the most important code for --lua-exec feature in about three hours, but discussing the implementation, polishing it, porting to Windows, testing, writing documentation and demonstration scripts... made it two weeks. And it's a relatively simple feature, compared to for example "no script stacking" problem I listed below. Though, I thought it could prove useful to list all the problems that are yet to be solved - hopefully I'll manage to spark up some discussion on these topics.

Lack of select()

There are a few limitations of what the soon-to-be-merged "ncatins" feature can do. At the moment, although Lua scripts can read from both the connection and Ncat's standard input, they have to do it in a synchronous manner. This means that when we hit the code that says "read a line from Ncat's stdin", the script will block until it actually reads the line. Put this way, it doesn't sound like a problem (isn't it what we wanted to do in the first place?), but consider what happens when we get some other form of input in the meantime.

For example, in an IRC script we could write a loop like "read a line from the socket, then read from Ncat's input, rinse and repeat". We connect to the channel, Dan says something, we get the input, it gets printed out... then there's the user's turn, which has to type in something in reaction to Dan's remark. But in the meantime, the fellow decides to correct himself and writes another message. Dan won't see it until he writes his own part and it's already too late, he pressed enter.

Instead, we could write the loop this way: "wait for either the socket or Ncat's input to have some data, then either read from the socket or ncatin". This way we only read from the inputs when there's something to read and we can avoid the problem I just described.

As for the solution, I already have a patch waiting that implements io.select() on *nix systems that successfully solves the problem. For the feature to complete, though, I'll have to port it to the alternative Microsoft's POSIX-ignorant system we all love. Since Windows obviously doesn't have a fully working select() call, it might prove pretty hacky and complicated. Believe me, I just can't wait.

Reimplementing the loops, no shared state, one-connection limit

These two problems stem from the way --lua-exec works: we spawn a new, independent script that only controls its own connection. In order to parse data in a repetitive manner, it has to create an infinite loop. One could argue that it's a bit ugly, because Ncat already has its own read-write loop and a callback could fit better here.

Personally, I have some doubts about this approach - --lua-exec showed clearly how long it takes for a feature to be brought to a working state and implementing a callback-driven subsystem would require a lot of interface changes, which could introduce a lot of nasty bugs. The concept has its advantages, though, and I'll mention them in next paragraphs/subsections.

As for the "no shared state" problem, it's another consequence of how --lua-exec works. Because the processes have no way of talking to each other and each has its own set of variables, there's no way to implement, for example, a chat server this way without some ugly hacks - we can't keep track other connections in order to broadcast to them. That's also a problem that a properly designed callback-driven subsystem could probably solve.

And the last problem: one-connection limit. At the moment we can only write to our current connection - the incoming one in the listen mode or the outgoing in connect mode. Sometimes, though, it could be useful to make new connections - for example, a DNS server could want to connect to its master server in order to forward requests or a script that works on HTTP (for example, a proxy) could want to establish a new connection to a remote HTTP host (or a couple of ones). I believe that callback-driven system would face pretty much the same problems here as the --lua-exec feature - making Ncat handle more than one connection would require quite a few interfaces to be changed - most notably, I believe that I would have to transform struct options defined ncat_core.h as a global variable into a set of local variables.

One-connection problem - a possible solution?

The one-connection problem could have a simpler solution though - while reading the Ncat's reference manual, I discovered the popen() call. While it's obviously not enough to give a full control over the connection (popen() spawns an unidirectional pipe, making the connection either read-only or write-only), there's an alternative that would involve exposing a way to spawn an Ncat subprocess and return stdin, stdout (and possibly stderr) pipes to the user. That would involve pretty much the same tricks that were used in --*exec Ncat features - fork() and dup() on POSIX, CreateProcess and redirected named pipes on Windows. It feels kludgy though, so I'd welcome some feedback on what do you think about it. I'm pretty sure it would greatly extend Ncat-Lua's use cases.

Ugly command-line syntax

The current way to run a websocket script against echo.websocket.org host looks like this:

ncat --lua-exec /possibly/quite/lengthy/path/to/ws.lua echo.websocket.org 80

The obvious problem with it is that it's lengthy. Why can't we use --lua-exec ws.lua and ask Ncat to look for the script in the predefined directory list? That would be especially useful since the basic scripts that sit in the Ncat's source code package aren't currently copied anywhere along with the make install command and user would have to copy them on his own and then specify the path manually. Addressing these issues would definitely make it way more convenient.

I tried to do something about it in my d33tah/ncat-lua-with branch, introducing --with command-line switch that was basically --lua-exec with ncatins and io.select(), looking for the script in the "./scripts" directory as a proof of concept. Additionally, it supported filter stacking feature for scripts, which I'll discuss in the next subsection.

David quickly pointed me out that in this case, even --lua-exec ws.lua is not good enough - it'd be best not to bother the user about Lua, shortening it to --ws command-line switch. This gives the user experience that makes WebSocket implementation truly invisible to the user, hiding the details of whether it was written in C or in Lua. This raises some interesting problems, though, so I'd like to address this issue specifically in the next big section, "New plugin model". Before I do that, the last problem:

No filter stacking

Suppose we'd for some reason like to encode all our connection in Base-64 or any other encoding we have a script for. Moreover, we'd like our encoded data to be sent over Websocket. Or perhaps we'd like to call --exec or --sh-exec in addition to running the WS filter. Although current "ncatins" feature allows you to control the stdin and stdout of Ncat, it still holds exclusive control over them. In order to do that, Ncat would have to keep track of all the filters used connect them.

Consider the base64-and-websocket example. When we type some data into Ncat, it gets sent over to base64's Ncatin. It then encodes is with base64 and normally, would send the output to the socket. Yet, in order to put it in a Websocket frame, we'll need to receive the base64-encoded data from base64's stdout and put it again in Websocket's ncatin. Then, Websocket prepares the frame and sends it to stdout, finally performing the actual send operation. To make it even more complicated, the decoding has to go backwards - the chain starts with the websocket script, which ncatouts the base64-encoded data, which base64 reads from its stdin and sends over to the terminal. It's a real mind bender, but the resulting feature is also really powerful.

I already implemented this feature for Lua scripts only (and without Windows port) in d33tah/ncat-lua-with branch. It basically involves lots of anonymous pipes connected in a way that a write to one process's ncatout redirects the data to the read end of the other process's stdin. Again, the socket abstraction concept David suggested could probably make this task easier.

New plugin model

In this section, I'd like to propose a new model for the Lua plugins that would basically turn the --lua-exec scripts into something way more convenient. The idea is to change:

ncat --lua-exec /possibly/quite/lengthy/path/to/ws.lua echo.websocket.org 80

to:

ncat --websocket echo.websocket.org

I believe that Websocket is a good example in which we could predict potential problems and discuss solutions.

How to find the script?

So, well, Ncat was run with the --websocket option. How do we locate the script responsible for this command-line switch? I can see two options, one not excluding another:

"How do we locate the script responsible?" Forgive me, but I think this question reveals a flawed basic assumption, that a feature must be implemented by running a Lua script. --websocket is not meant to be a shorthand for --lua-exec websocket.lua. The problem with --lua-exec websocket.lua is not that it makes for long command lines—it is that --lua-exec is the wrong model for application-layer tunnels like WebSocket. That is, any implementation of a WebSocket layer that says, "Dear websocket.lua, please take over execution of the program" is likely to be the wrong solution. The proposed --with option, in my opinion, is not useful because it is merely a shorthand for --lua-exec and doesn't address the real problem.

In my opinion, both of the options below (--option implies a file option.lua, and an index of option→script mappings) are unacceptable.

What is the right thing to do instead? Just handle --websocket like any other option—parse the command line and then call option-specific C code as appropriate. Presumably, the C code for --websocket will open a Lua file somewhere, but the name of this file is not necessarily implied by the option name, nor necessarily known to anything except the C code itself. If we add another option that is backed by Lua code, it does not have to work the same way as --websocket. The code that is run for each option is option-specific. To look at it another way: --lua-exec is meant to be general, running any script you ask it to in the same way. --websocket is not meant to be general, rather it does some very specific internal thing that happens to load a Lua file somewhere. David (talk) 23:25, 10 August 2013 (UTC)

1. Create a one-to-one mapping between the command-line switch and the script filename. For example, --foo would always point to /usr/share/ncat/scripts/foo.lua or C:\Program Files (x86)\Nmap\Ncat\Scripts\foo.lua. This has the advantage of being simple and pretty fast - we only look for one file in at most several directories and if it's not there, we can be sure that the command-line switch is invalid. In addition to that, we could let the user specify some NCAT_SCRIPT_PATH environment variable, in which Ncat would start looking for the scripts (or even override the default path).

The problem, though, is that it's not really flexible. For example, implementing aliases or adding a few scripts that would differ with only some details based on the environment (like listen-mode Websocket and connect-mode Websocket) would require creating new files.

2) Create an index of scripts. This could for example work like this: every --lua-exec script specified in some special directory could contain a few variables set, like this:

help_description = "Switches Ncat to WebSocket mode"
command_line_starters = { "websocket", "ws" }

(there could be more of these, such as "arguments" and some script metadata in case we'd start collecting the scripts database the way it's done with NSE)

Then, once we populated /usr/share/ncat/scripts with such scripts, we'd run ncat --reindex-scripts and create an index file, which would hold a list of the scripts.

Further invocation problems

In his e-mail at the very beginning of my GSoC project ( http://seclists.org/nmap-dev/2013/q2/445 ), David proposed that it would be more convenient to call:

ncat --websocket wss://example.com:8000/ 

instead of:

ncat --websocket --ssl example.com 8000

This leads to problems of allowing the script to change the parsing of whole command line, or - if we assume that the "unknown" part is always the URI, getting the host and port out of the URI (and possibly some script arguments, for example path in the HTTP URL). The way I see it, the --lua-exec scripts could have an optional function that would receive the URI and translate it to an associative array with keys: host, port, protocol (TCP/UDP/SCTP), SSL (binary). If the URI is not valid, the function would return nil. It could even be possible to find scripts based just on the URI - imagine typing in ncat wss://example.com:8000/ and dropping right into the connection. If URI parsers of two scripts returned not-nil, the user would be prompted to specify the correct command-line switch.

I think the discussion above presumes the wrong basic model for an application layer like this. The presumed model is this:

1. Parse the command line.
2. Resolve the target host and port.
3. Exec a Lua script based on the command line, and pass it the given host and port.

As you note, this is problematic because step 2 needs to parse a URL and step 3 is the part that knows about ws URLs. But let go of the exec mindset and you can use a model like this:

1. Parse the command line. If --websocket appears, set websocket_mode = true.

2. if (websocket_mode) {
     parsed_url = parse_target_as_wss_url(target);
     do_websocket_mode(parsed_url);
   } else {
     resolve(target, port);
     business_as_usual(target, port);
   }

In general, don't be afraid of using C code for the things it's good at and already used for, like command line parsing. do_websocket_mode could even do the URL parsing itself—but you see how the difficulties of this section go away when you push a little bit of the logic upward. --websocket is not meant to have a general-purpose implementation. It is meant to have a single-purpose implementation. David (talk) 23:43, 10 August 2013 (UTC)