unbound/doc/plan

Plan for Unbound.

Split into a set of boxes.  Every box will take about 3 weeks to a month
to complete.  The first set of of boxes (approx 5 months) will need coding
by a limited set of people.  But after every box, a 0.x release is done, 
which is then tested and code review is done.  

Every box:
 * implement the features
 * documentation of those features
 * test-framework for the new features
 * tests for the new features
 * speed test of this stage
 * release of 0.x version (0.x for development only)
 * a teleconference(jabber) held to discuss.
 * code review internal couple of days, external a week or so,
   while we continue the next box. 

Roughly the boxes are as follows:
0.0 initial setup - results in network code that forwards queries
    and returns the reply (no cache), but also testbed, svn, maillist.
    One query at a time (nonblocking IO though).
0.1 threads - results in threaded forwarder
0.2 LRU hashtable, results in basic caching forwarder (no DNS parse)
0.3 First functionality - results in caching forwarder (with DNS parse,
    query compare, RR specific updates).
0.4 Basic resolver - module layout, iterator module, scrubber module,
    results in resolver that can service multiple queries per thread.
    This stage takes longer, due to complexity in the iterator module.
    Twice as long; one box for module layout, one box for iterator module.
0.5 Validator - validator module.
0.6 Bigger and better - Operational useful features (config, log, memory)
0.7 Local zones feature - localzones stubzones fwdzones, no leak rfc1918.
    views support; for selective recursive service.
0.8 Library use - resolver validator lib (and test apps)
0.9 Corner cases - be able to resolve in the wild. Run fuzzers.
    Run as many tests as we can think of.
    Go through logs and check for long, unresolved cases
    Use profiler.
0.10 Beta release. Run shadow for a resolver in production for several
    weeks.
0.11 Features features
    aggressive negative caching for NSEC, NSEC3.
    multiple queries per question, server exploration, server selection.
    option to use real entropy for randomness (mix it in once in a while).
    check query, option to enforce qdsection checking (forgery-resilience).
    NSID support.
    Be able to prime roots using several queries (only NS on first).

For boxes 0.5-1.0 the planning is to be revised, at the 0.5 stage external
coders are welcome.  Since the project is bigger, there is room for them.

This is a summary of the items.  Below more detailed work items are spelled
out with a (tentative) directory structure for the project.


Styleguide:
* write working stuff. (it starts to work with no features)
* write tests immediately for every function, every feature.
* document as you go. (doxygen comments, manpages and readme).
* copyright every file BSD. comments every file. clean coding in C.
* every day discuss state of the nation for 10 minutes.

*** Initial setup
* setup svn repo. Makefile with automatic dependencies and configure script.
	* link with ldns.
* listen_dnsport and outside_network services, (unit) tests for them.
	* use libevent to listen on fds.
* setup test infrastructure (tpkg on checkin; testbed on labs test machines).
* daemon version that forwards queries. (listen, send) Tests for it.
	* test by having the outside_net service grab answers from a
	  file instead of network, file of id priority answerpacket.
	  and what query to give this answer to, highprio matches first.

*** Threads
* first simple config file reading/writing and tests on config file.
  (config option is forwarder: yes/no. Cache size. That sort of thing.)
  (very simple format)
* First simple logging (to a file).
* Threads
	* check if pthread lib is the one to use (sys specific is faster?).
	* make config option to have threads.
	* alloc threadable.
	* locks.c
	* Tests with and without threads.
* alloc_service. Tests for alloc service (unit tests in internal structs).
* threading for the network services.
* Make sure threading/libevent starts working on all test machines.
  Use configure to turn off threading/libevent/...
  -- use libevent packaged together if not in system.
  -- maybe also for pthreads/...
* threaded forwarder version.
	* speed test of threaded version.

*** LRU hashtable.
* mini msg/reply structure for LRU hashtable test, simple replay format. 
* hashtable+LRU structure. Tests on structure.
	* tests on enter/remove, finding items.
	* tests on LRU movements.
	* Test on speed of finding items.
* slabbed hashtable+LRU structure. 
	* Test locking; perhaps by having sleeps in some threads to force
	  locks to contend. helgrind.
* daemon upgraded to be a caching forwarder. So it stores all in cache. 
  Replies from cache. Tests on fake-caching forwarder functionality.
	* timeout of data test
	* finding data in cache.
	* finding data not in cache.
	* lru falloff of data.
* Speed test of fake-caching forwarder.

*** First functionality
* implement dname type and unit tests on it. (all corner cases, random cases)
* implement rrset type and tests. (all corner cases, random cases).
* msg-reply structure. unit tests of structure.
	* Test of those rrset pointers
* daemon upgraded to be a caching forwarder. So it stores all in cache. 
  Replies from cache. Tests on caching forwarder functionality.
	* timeout of data test
	* finding data in cache.
	* finding data not in cache.
	* lru falloff of data.
* Test update of one rrset in cached packet.
* Speed test of caching forwarder.

*** Basic Resolver
* Create module interface and module caller algorithm.
* Daemon config to use modules. Test the module caller.
* Create basic iterator and scrubber modules.
	* Test every state of the iterator by passing test data into
	  it.
	* And scrubber.
* Daemon config as cache(iterator).
	* Test daemon
	* Speed test.

*** Validator
* Create validator 
* Test validator on various conditions. By having stored set of
  domains and RRs in those domains to return to validator.
* Validating resolver.
	* Test resolver.
	* Speed test.

*** Bigger and Better
* Config file syntax checker program. Tests on checker.
* Logging first class feature with config options.
* donotqueryaddresses with trie for blocking entire netblocks.
* Memory overhaul, special allocators for hashtable caches, and mesh qstates.
* read root hints from file.

*** Local zones feature.
* Build in local zone features. First the total stop for1912.
* Then 'local content' for minimal serving of localhost.localdomain,
  and so on. 
* Remember jakob's diagram.
	* views support, selective recursive service
* Forward-local-zone to NSD.
	- include in package, autoforkexec on localhost to do so.
* forward local zone to remote server.
* stub zones - send queries for a zone to configged nameserver.
* test local zones
	* for speed
	* for correctness on corner cases
	* for validation
	* in case you get data on localzone in an answer (from rootserver).

*** Library use
* Create library that can do:
	* resolver
	* validator
	* validating resolver.
* Test application that links the library. (Like /usr/bin/host+validating).
	* Test it.

*** Corner cases
* Try to setup corner cases of (mis)configured DNS service/websites.
* Resolve msoft, google, yahoo, etc weird websites.
* Try to resolve many many different queries, perhaps compared with bind.
* create module testers, specific for the modules
	* read a file with cache contents and settings, provide fake
	  environment for module-handle-state-X functions, then check
	  resulting module state structure to correct answer.

*** Beta release. 
* Run shadow for a resolver in production for several weeks.
* Check logs for errors, long queries.
* Run in valgrind, speed profiling (as production shadow).

*** Features features
* aggressive negative caching for NSEC, NSEC3.
* multiple queries per question, server exploration, server selection.
* option to use real entropy for randomness (mix it in once in a while).
* check query, enforce qdsection, checking for forgery-resilience.
* NSID support.
* support TSIG on queries, for validating resolver deployment.
* Be able to prime roots using several queries (like, get only NS first).
* Nicer statistics
* private TTL, dTLS features.


treeshrew/
	validator/ *.c *.h
		module takes qname, qtype, asks next module for answer
		and validates that answer.
	iterator/ *.c *.h
		module takes qname, qtype, iterative DNS queries
		never asks next module.
	services/
		- Routines that provide the callback services for modules.

		alloc_service: L1, L2 alloc service
		outside_network: pending queries helpers.
			pending query structure
		listen_dnsport: listen port53 service.
			request structure
		type_caches/
			rrset_cache
			msg_cache
				rrset and msg cache check local zones.
			infra_cache
			trusted_key_cache
	util/
		- Various components from which to build the rest.

		storage/
		rbtree: redblack tree, for L1 use.
			- copy from NSD.
		hashtable and hashfunc: for L1 use.
		locked_hashtable: for L2 use. -- not needed.
		fragment_hashtable: for L2 use.
		fragment_rbtree: for L2 use.
		slab_allocator: perhaps to support alloc service.

		(in util/ itself)
		locks: selected lock,unlock (spinlock/mutex).
		config: reads, stores config file
		netio: register callbacks to select().
			- use libevent (!)
			- copy from NSD.
		log: error and log handling.
		module.h: module interface
		misc: time() wrapper for speed.

		data/
		msg_reply: qname/qtype/CD/qclass/reply store.
		packed_rrset: main datatype
		dname: compare, printf, parse

	testcode/
		main programs that do unit tests, using testdata
	testdata/
	daemon/
		unbound.c for validating caching recursive dns server.
		scheduler.c for the modules.

	libunbound-all/
		app linkable. Can be configged to do whatever,
		validator, iterator, validating iterator, forwarding stub.
	libunbound-fwd/
		app linkable forwarding stub. Small lib.

	ask_cachor/ *.c *.h
		module takes qname, qtype, returns answer from msgcache.
		could ask cached for answer (and wait for network, 10 ms).
		if not in cache, asks next module.
	cachord/
		main.c, simple udp proto, query or store msg in cache.
		supports option to save cache to disk (absolute time ttls).