- Make it platform agnostic
- Separate FreeBSD related code
- Fix tests
- Make it report all non-loaded modules instead of the first occurrence
only
- Update kyuafile.5 man page
Reviewed by: ngie
MFC after: 2 weeks
Pull Request: https://github.com/freebsd/kyua/pull/270
(cherry picked from commit 939fec44a79323ba06cf0ad60d4b69300a8abbc6)
A new Kyua concept is added -- "execution environment". A test can be
configured to be run within a specific environment. The test case
lifecycle is extended respectively:
- execenv init (creates a jail or does nothing for default
execenv="host")
- test exec
- cleanup exec (optional)
- execenv cleanup (removes a jail or does nothing for default
execenv="host")
The following new functionality is provided, from bottom to top:
1 ATF based tests
- The new "execenv" metadata property can be set to explicitly ask for
an execution environment: "host" or "jail". If it's not defined, as
all existing tests do, then it implicitly means "host".
- The new "execenv.jail.params" metadata property can be optionally
defined to ask Kyua to use specific jail(8) parameters during creation
of a temporary jail. An example is "vnet allow.raw_sockets".
Kyua implicitly adds "children.max" to "execenv_jail_params"
parameters with the maximum possible value. A test case can override
it.
2 Kyuafile
- The same new metadata properties can be defined on Kyuafile level:
"execenv" and "execenv_jail_params".
- Note that historically ATF uses dotted style of metadata naming, while
Kyua uses underscore style. Hence "execenv.jail.params" vs.
"execenv_jail_params".
3 kyua.conf, kyua CLI
- The new "execenvs" engine configuration variable can be set to a list
of execution environments to run only tests designed for. Tests of not
listed environments are skipped.
- By default, this variable lists all execution environments supported
by a Kyua binary, e.g. execenvs="host jail".
- This variable can be changed via "kyua.conf" or via kyua CLI's "-v"
parameter. For example, "kyua -v execenvs=host test" will run only
host-based tests and skip jail-based ones.
- Current value of this variable can be examined with "kyua config".
[markj] This feature has not landed upstream yet.
See the discussion in https://github.com/freebsd/kyua/pull/224 .
Having the ability to automatically jail tests allows many network tests
to run in parallel, giving a drastic speedup. So, let's import the
feature and start using it in main.
Signed-off-by: Igor Ostapenko <pm@igoro.pro>
Reviewed by: markj, kp
Tested by: markj, kp
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D45865
(cherry picked from commit 257e70f1d5ee61037c8c59b116538d3b6b1427a2)
Approved by: kp (mentor), markj (mentor)
Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.
The build of kyua is controlled by WITH(OUT)_TESTS_SUPPORT.
Reviewed by: emaste
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24103
