Ncat/Socket abstractions

NOTE: this article is a feature that is currently under heavy development. Its current purpose is to document the feature for project's internal purposes. For this reason, both the functionality and the interfaces might change and the document might not reflect the latest modifications. The feature is not ready for production use yet. The text marked in red means that this is a temporary behavior that is expected to be changed soon. Should you have any feedback on it though, feel free to edit this document, its talk page or write to dev@nmap.org.

User's guide

Filtering basics

Ncat's „Lua filters” (also known as „socket abstractions”) is a feature that allows the Ncat user to run simple Lua scripts („filters”) that manipulate the incoming and outgoing Ncat data. For example, take the „rot13.lua” script that is bundled with Ncat. When the user runs:

ncat -L rot13.lua example.org

It establishes a connection to example.org, using the default port number and protocol (TCP port number 31337). From now on, whatever Ncat receives either from the user (via the terminal) or from the remote connection will be „filtered” by rot13.lua script, which performs ROT13 encoding (see here: https://en.wikipedia.org/wiki/ROT13).

This means that whenever you type in „HELLO”, „URYYB” will be sent instead which is the same message enciphered using ROT13. Likewise, if Ncat receives „URRYB”, it will print „HELLO”. This of course works for any other messages. As a result, the user will encode all its outgoing messages and decode the incoming ones transparently. Provided that the server also sends ROT13-encoded messages, it will feel like any other Ncat session.

Script stacking

Lua filters can also be stacked. Consider the example invocation which basically means „filter all the traffic via rot13.lua, and then, base64.lua”:

ncat -L rot13.lua -L base64.lua example.org

In this case, whenever user types in „HELLO” into the terminal, it will first be encoded using Base64 (see: https://en.wikipedia.org/wiki/Base64). The Base64 representation of „HELLO” is „SEVMTE8K”. This will then be encoded using ROT13, the result being „FRIZGR8X”. Finally, the message, encoded by both Base64 and ROT13, will be sent to example.org.

There are two things to remember about when using script stacking. First is that the incoming messages will be decoded in the order reverse to the one that was used to encode the data. This means that when Ncat receives „FRIZGR8X” when rot13.lua and base64.lua are enabled, it will first be decoded using ROT13, and then Base64. In other words, most simple filter scripts will give the same output when decoding an encoded message, so that they can be completely invisible to the user.

You should also note that the order in which the scripts are specified in the command line matters. ncat -L rot13.lua -L base64.lua example.org and ncat -L base64.lua -L rot13.lua example.org are two different setups and will work differently. „HELLO” translates to „FRIZGR8X” only if you first Base64-encode your message and then ROT13 it. If you do it in the reverse order, it will be a completely different message, namely „VVJZWUIK”.

--lua-exec vs socket abstractions - the comparison

Moved here.

Filter developer's guide

Filter development basics

A Lua filter is basically a script that returns a table. When an event such as incoming user or network data is detected, Ncat looks for specific keys in the table, such as „recv” or „send”. Here's an example script:

return {
  recv = function (self) 
      local line, err = self.super:recv()
      if line == nil then
        return line, err
      end
      return line:sub(2)
    end,
    
  send = function(self, buf)
      return self.super:send('X'..buf)
    end,
}

This filter adds 'X' letter to every sent line and removes the first character from all incoming data. To execute it, save it in the current working directory and run “ncat -L x.lua”, followed by remaining connect-mode options.

As you can see, „recv” and „send” are special functions that are run by Ncat when specific events occur. The table may contain any number of keys and values of any Lua type, though you should using additional tables inside the filter table, for reasons explained in “initialization” section.

TODO: write about super

Supported methods

Note: in addition to the "Error handling" section, all functions' return values and arguments are being validated if they are used internally. Any erroneous values will result in an I/O error being reported to Ncat, as well as a message in its debug log. Ncat usually reacts to I/O errors by closing the faulty connection.

recv()

Behavior: signalled when there is data to be read, recv() performs a non-blocking read. Returns a string of the maximum buffer size if there's any data waiting to be read. Note that it might not read all the data waiting. If there's no data, the first return value will be nil. Note: this method is currently for event-signalling purposes only. Do NOT call it manually as it may

Arguments: none.

Return value: the string read from the socket.

Error handling: in case of an error, the first argument will be nil, the second is the error message. There is currently no handling in connect-mode.

How Ncat handles the result: whatever the filter at the bottom of chain returns will be returned to Ncat, which in most cases prints the value back to the terminal. Note: it might be used in proxy mode too.

Bugs/TODO:

• the recv() will in the future change to a "simulated blocking read", similar to what NSE offers. This means that the filter execution for this connection will be paused until there is some data read, which is then returned
• there is currently no error handling in connect mode
• in connect mode, this is currently performed AFTER the read happened and recv() only returns the buffer NSE returned. The second recv() call will simply return an empty string.
• perhaps add the numbytes argument that would let you control how much data is read?
• perhaps we should be allowed to call a single read() in connect() or send()?

Example:

return {
  -- Appends "Received: " to every message received
  recv = function (self) 
      line, err = self.super:recv()
      if line == nil then
	return line, err
      end
      return 'Received:' .. line
    end,
}

send()

Behavior: performs a blocking write. Could be called manually from any other event or run as event by Ncat when user entered any text from the Ncat terminal. Note: the connection does not have to be "self" - we could as well call any other connection, as shown in chat.lua script.

* Perhaps these should be two separate methods?

Return values: none

Error handling: none

How Ncat handles the result: the data is sent to the target connection.

Bugs/TODO: add more error reporting?

Example:

return {
  -- Appends "Sent: " to every message being sent
  send = function (self, str) 
      return self.super:send(str)
    end,
}

Implementation details

Initialization

At the startup, global variable called "socket" contains the root "class" that has the lowest-level methods defined. This prototype socket is saved in Lua registry under the key "socket". Then, when loading Lua filters, it is expected that the Lua code will return a table. This table is passed as an argument to make_socket_function that looks like this:

function make_socket(function_self, self, arg2)

    local super = arg2 or self.super
    local ret

    if super then
        ret = {}
        for _, m in pairs({'send','recv'}) do
               --Set the method to one that calls super and passes it
               --all the arguments we got, returning everything we'd get.
            ret[m] = function(self, ...)
                    return table.unpack({self.super[m](self.super, ...)})
                end
        end
    else
        ret = {}
    end

    for k, v in pairs(self) do
        ret[k] = v
    end
    if super then
        ret.super = function_self(function_self, super)
    end
    return ret
end

Note that this function, especially the pairs({'send','recv'}) part might change, see ncat_lua_filters.c, lua_filters_setup() for an inlined Lua version of this function. The function's behavior is a bit tricky, which is why it deserves a detailed comment. It is executed in two cases:

1. To generate the new "current" socket prototype (initialization) to replace the "old" one,
2. To create a new independent instance of the prototype

The function expects three arguments - itself (because it is not stored in the global namespace and hence has no name), the "current" socket and its parent ("old"/"super" socket). In the first case described above, the first argument is the filter returned by the Lua filter code and the second argument is the socket prototype that was previously stored in the registry. If the parent socket is not given, it is read from the socket's "super" field. This is useful in the second case, when we just want to create a new instance of the object.

The function tries to recursively create a new socket instance based on the given prototype. In order to make all sockets have their own identity, a shallow copy of all the filters' keys is performed. If the filter does not define core Ncat methods like recv or send, each of them is pre-set to a function that calls self.super, passing all the arguments and return values. This kind of magic makes "return {}" a valid filter script that simply does not change anything (see scripts/filters/null.lua). Once all the keys and values are copied, socket's table "super" key is set to a copy of prototype's super, created the same way.

(TODO: a diagram showing how copying works for those that might hear about the concept of a queue for the first time)

NOTE: the copy of the filters is a shallow copy. That means that if one of values (or even worse, keys) is a table, won't have all its keys and values copied as well, but shared across all the filter instances of the same filter. It is not advised to use it as a way of creating filter-global variables this way - use global variables instead. (TODO: need some opinions - is it a good advice?)

connect-mode recv() coroutines

This section currently only applies to connect-mode recv() implementation in ncat-sa-take2 branch.

When Ncat wants to schedule a read operation with filters enabled, it does not call nsock_readbytes directly, but uses lua_nsock_read instead. This function calls the recv on the highest-level socket, which leads to the execution of lua_nsock_read_raw, Lua CFunction calling nsock_readbytes. The event handler is changed as well - in order to properly return the read buffer, there is a need for a special function.

The callback-driven mechanism creates a challenge for all functions that use recv(), because in order to return from recv() only after readingTODO: finish this section.

Self-indexing problem

This is a problem I discovered while working on chat.lua script. In my original concept, I wanted the messages that user wrote to be broadcast to everybody but him. Since you can see all the connections in connections[] global variable, all you need to do is iterate over the array and test if it's not "self".

The problem is that above only works if the script is the one mentioned as first in the command-line arguments. The reason is that if we run for example:

ncat -L scripts/filters/null.lua -L scripts/filters/chat.lua -l

We won't find k, where connections[k] == self. The reason is that for each connection, its instance is assigned to filter "null" and we'd actually be looking for k, where connections[k].super == self. As we add more filters, it's getting more complicated - we'd have to walk through every filter for every connection until we find any table that is equal to "self".

One of the solutions to the problem would be to change the connect() interface, adding another parameter - "id", that would be the array index so that connections[id] == self. The value could as well be injected to all filter instances. In chat.lua example, this was "solved" by broadcasting to all the peers, including the one sending the message.

Ideas

According to my meetings with David, one of very important features for the socket abstraction system will be to let the filters make their own connections. So, for example, you have this "HTTP proxy" filter (or something like that). A user connects, the filter takes over and user says "connect to insecure.org, port 80". And now the filter needs to handle two connections - one from the user, the other to insecure.org.

David said he meant a different thing.

This calls for a command-line syntax for passing arguments to the scripts, so you could tell the script which proxy server to use (okay, you could use environment variables for that already, but it's ugly). Personally, I'd love it to be expanded into a whole plugin platform similar to NSE, where users could submit filter scripts for various common protocols and tasks and keep it in a Nmap-project managed repository along with the cool web documentation. I don't think there'll be enough time for this during my GSoC project.