GSoC community ideas

From SecWiki
Jump to: navigation, search

The purpose of this page is to list and present ideas from the community for the Google Summer of Code. See http://nmap.org/soc for more suggestions. Anyone can add your own ideas here. They're currently in no particular order.

Performance Specialist

Speed up Nmap scanning for internet-wide applications

While due to design decisions Nmap has to perform much more tasks than asynchronous port scanners such as masscan or Zmap, we could still improve the scanner's performance to make it work even better while scanning millions of hosts. If I recall correctly, DNS was one of the spots where optimizations could be made. (added by d33tah)

  • This is closely related to the Performance/Optimization Specialist idea further down, so see some of the tasks there.
  • DNS speedups are always good, but measurement would be the first step. How does Nmap's --min-rate compare to these asynchronous scanners? At what point (in packets-per-second) do they diverge and Nmap cannot keep up?

Performance/Optimization specialist

  • Optimize Nmap performance and resource usage
    • Particularly focus on measuring existing --min-rate behavior compared to Zmap and masscan. The common criticism of Nmap being slow usually ignores this scan mode.
    • Consider measuring for improvement of hostgroup sizes. Larger hostgroup sizes could improve parallelization for large scans, but has memory cost.
  • A fully stateless ping scanner, à la zmap|masscan could be added for very large scale scans.
  • Maybe do some of our large-scale scanning research too, like keeping top ports stats up to date.
  • Analyze data dump from Carna botnet for Nmap improvements.

Some items from the todo list:

  • We should figure out why (at least with Nping) raw ethernet frame sends seem to be taking significantly longer than raw socket sends (e.g. using --send-ip or the OS-provided ping utility). This has been reproduced on Linux and Windows. Here's a thread: http://seclists.org/nmap-dev/2012/q4/424
    • Note that David and I tried to reproduce this on his machine and on 'web' and 'research' machines and could not reproduce. Still happens with Fyodor's machine connected with WiFi. Fyodor should test on the same machine using wired and see if that changes anything.
  • Make version detection and NSE timing system more dynamic so that the concurrency can change based on network conditions/ability. After all, beefy systems on fast connections should be able to handle far more parallel connections than slower systems. At a minimum, this at least warrants more benchmark testing.
  • [NSE] Do some benchmarking of our brute.nse. We should check the performance with different levels of thread parallelism. Our initial results show that it isn't helping much for vnc-brute or for drda-brute (which is currently using the multi-thread feature directly rather than through brute.nse library). We should figure out why the threads aren't helping more, and whether there is something we can do to fix it. It would also be interesting to compare speed with Ncrack for services we have in common.
  • Consider implementing both global and per-host congestion control in the IPv6 OS detection engine. Currently it handles congestion globally (one CWND and SSTHRESH shared by all hosts). This works fine but it may not be the most efficient approach: if the congestion is not in our network segment but in a target's and we are os-scanning hosts in different networks, then all hosts get "penalized" because there is congestion in another network, not in theirs.
  • Look into whether we should loosen/change the global congestion control system to address possible cases of one target host with many dropped packets slowing down the whole group. See http://seclists.org/nmap-dev/2008/q1/0096.html .
    • Related possibility: Fix --nogcc to gracefully handle ping scans. Right now it seems to go WAY TOO FAST (e.g. several thousand packets per second on my DSL line).
    • [12/22/09] David says: It still is in one case that I've documented on my wiki. I had an idea to fix it, but on testing it it didn't work. The idea was to treat the global congestion limit differently. Instead of dropping it down to the minimum level on a drop as is done currently, I thought about only dropping it by the amount that the individual host limit drops. For example, if a host had a drop and its limit fell from 25 to 1, then the global limit would change (if it was at 100 to begin with) to 76, not all the way down to 2 or whatever it is. The idea being that the global limit is most important at the beginning of a scan, when there's no information to set host limits, and every host wants to send all its first probes at once. See http://www.bamsoftware.com/wiki/Nmap/PerformanceNotesArchive2#global-cc. I am convinced, though, that some sort of global control is necessary. There's a reason that a web browser limits the number of connections it will make, and doesn't try to download every image file at once and count on the fairness of TCP to sort it out.

Nsock based asynchronous hostname lookup

Hostname lookups are currently performed in a blocking way using getaddrinfo(3). Nsock could provide a dedicated API for this. It would open the door to many further performance improvements. Nsock proxy mode would also benefit such a feature. Note that we want to be able to specify the nameservers to use.

Links:


NSE developer

Nmap Scripting Engine Script Developer

  • Web scanning specialist
  • Discovery scanning specialist
  • Vuln/exploitation specialist

Moving packet.lua from lua to C++

It might make sense for NSE to leverage the existing packet crafting classes, the existing packet.lua isn't as stable and fully-featured as our C++ classes. That would also mean a single packet crafting stack to maintain.

Nsock

Nsock server mode

Nsock should provide a server mode to wait for and accept connections (for ncat and nping). An implementation is proposed on https://svn.nmap.org/nmap-exp/henri/nsock-servermode

Nsock buffered I/O

Profiling nsock and studying the I/O patterns of nmap would help us optimize the way we perform I/O and use memory in nsock. Especially, using I/O vectors (see readv/writev) could improve efficiency and memory footprint (or not) depending on what the profiling reveals.

Other

Test automation specialist

  • Parallelize Ncat tests
    • let Ncat listen on port 0 to choose a random open port (and print it on the debug output). Then tests can start multiple instances at once without port conflicts.
    • http://seclists.org/nmap-dev/2013/q1/234 patch to enable port 0 and comments on the patch.
  • Create Nmap test suite
    • Mock objects such as UltraScanInfo, HostScanStats, OsScanInfo, HostOsScan, etc. will let us write these tests and also target various portions of Nmap for stress testing and security fuzzing.
    • Besides standard regressions, such a test suite would be helpful to work on improving performances. It could also allow us to detect and track down issues that only appear during large scale scans.

Port Nmap to MinGW

Building Nmap on Windows currently requires Microsoft Visual Studio. It would be great to make the whole Nmap project (including Nbase and Nsock libraries and umbrella projects such Nping, Ncat etc.) build under MinGW to get rid of proprietary dependencies, make it easier to cross-compile and simplify the build process. (Added by d33tah)

  • Related: http://seclists.org/nmap-dev/2013/q3/321 - if I recall David's comments correctly, the GSoC project would have to port all the Nmap codebase, not just Ncat and avoid cryptic #ifdefs, making it clear why particular solutions were chosen.

nmaprc.lua

See here: http://seclists.org/nmap-dev/2013/q2/279 (added by d33tah)

Zenmap GUI developer

GSoC community ideas/Zenmap GUI developer

Improved port specification

The --top-ports parameter is incredibly valuable but hardly adaptive. It would be nice to extend the port specification syntax to easily add/remove ports from the top-ports lists.

http://seclists.org/nmap-dev/2012/q3/380 thread about making -p and --top-ports take a union.

  • I rescind my objection to changing the current behavior to preserve a workaround for --exclude-ports, since --exclude-ports was added in GSoC 2014. Bonsaiviking (talk) 20:20, 9 December 2014 (UTC)

WebSocket mode for Ncat

WebSocket is a protocol used by web browsers to provide functionality similar to that of TCP sockets. It consists of an HTTP-based handshake with an Upgrade header field, followed by data using a simple framing protocol. Ncat should be able to use WebSocket in connect mode, emulating a web browser, to interact with WebSocket servers from the command line for example. Ncat should be able to be a WebSocket server, which would provide an easy way to WebSocketize normal TCP programs.

In connect mode, Ncat needs to send only masked frames and in listen mode it must not send any masked frames (RFC 6455 section 5.3). Ncat should know about Ping control frames and reply with a Pong, but it never needs to send Ping.

Potential issues:

  • WebSocket distinguishes between text (opcode 1) and binary (opcode 2) and potentially other types of frames. Text frames are encoded in UTF-8 and can't represent arbitrary binary data driectly. Binary frames would be really nice, but when I tested them in March 2012, Firefox 10 doesn't fire onmessage events when it receives them.
    • comment: The binary frames do work in relatively recent Google Chrome versions. I wrote a WebSocket debugging app called dwst for Chrome. It should be trivial to port it to another WebSocket friendly browser if required, but this won't fix lack of binary frame support in a browser. --Cyberix 14:48, 29 March 2012 (PDT)
    • The websockify proxy works around this with a base64 protocol. David 14:41, 2 April 2012 (PDT)

Links:

There may not be any need to support the earlier "hixie" specifications. However, a full project will include checking how many browsers suport only this specification or do so by default.

David 09:44, 26 March 2012 (PDT)


Ideas with prior effort

Scanning from within Tor

Currently Nmap can't scan Tor hidden services straight out of the box (I heard that there was an experimental patch around that solved this though). Also, scanning the Internet hosts requires quite a lot of dancing. These could be useful features to have. (Added by d33tah)

Equivalent to Scanning through proxies.

Nmap and nping should be able to leverage HTTP and SOCKS proxies for scanning. The current implementation suffers limitations regarding SSL, authentication and proxy types. Someone with a taste for low level C and efficient data structures could work on those limitations. This is a good introduction to the very core of nmap and would be a highly visible enhancement.

Bringing Lua to Ncat, part II

In 2013, a Google Summer of Code project called "Bringing Lua to Ncat" introduced Lua scripting support to Ncat. While some functionality was added, this project was mostly about research and there's still a lot of ideas left to finish and implement. To see the state of the project at the end of the last GSoC, see nmap-exp/d33tah/ncat-sa-take2 (and David's corresponding branch), and mailing list archives. (added by d33tah)


Semi-completed

Optimize zenmap to work with large-scale scans

There's quite a few reports on the mailing list related to Zenmap running out of memory. This might be a sign that Zenmap needs some memory optimizations. We need some profiling to find the underlying cause, followed by fixing the problem. (added by d33tah)

  • Update post-GSoC 2014: Jay fixed a couple issues: first, a string building bug caused loading of large scans to be quadratically slow. Second, the syntax-highlighted Nmap output window consumes a lot of memory. Zenmap will now catch the MemoryError due to insufficient memory and delete the output window to free up memory. This is less-than-ideal because we lose some functionality, and the Python process is still allowed to consume up to the limit of memory available without releasing it to the OS. A more complete solution would be a paging window that can swap out portions of the output that are not being displayed, in order to avoid consuming as much memory in the first place. Bonsaiviking (talk) 20:06, 9 December 2014 (UTC)

nsock based port scanning

Re-implementing connect() scan using nsock would both make nmap able to port-scan through proxies and leverage nsock's efficient I/O mechanisms. It would aditionally be another step toward a pipelined architecture where all scan phases can be mixed. (Added by d33tah, updated by Henri)

Exploring port scanning from within NSE

Being able to write port scanning modules for NSE might open new opportunities. Some experiments have been done, showing that NSE could be well suited to write application-based port scanners (like ftp-bounce). It would enlighten nmap's core and allow implementations to make use of existing libraries. It would also ease prototyping new scan techniques. See http://seclists.org/nmap-dev/2011/q2/307

Fyodor told nmap-dev about a paper that shows two novel and very interesting scan techniques. We might want nmap to be able to leverage them. See: http://seclists.org/nmap-dev/2011/q2/280

That could also become a way to run scans behind proxies, as the NSE stack is already proxy-aware.

New IPv6 OS scan features

“Features” in a machine learning sense. A list of some ideas is in the comments of https://svn.nmap.org/nmap-exp/luis/ipv6tests/vectorize.py. This involves adding them to the Python training programs and the C++ vectorize function in FPEngine.cc.

A related project is to do tests to see which features are the most useful and useless.

WinPcap Specialist

  • WinPcap support for NDIS 6
  • WinPcap privileges
  • No-install DLL support
  • Driver signing
  • Maybe release it with new name and function entry points and maintain it ourselves
    • Possibly this person could do other low-level Windows tasks too

David 11:49, 22 March 2012 (PDT)

Retrieved from "https://secwiki.org/mediawiki/index.php?title=GSoC_community_ideas&oldid=1825"