postgresql/src/test/perl/README

Perl-based TAP tests
====================

src/test/perl/ contains shared infrastructure that's used by Perl-based tests
across the source tree, particularly tests in src/bin and src/test. It's used
to drive tests for backup and restore, replication, etc - anything that can't
really be expressed using pg_regress or the isolation test framework.

The tests are invoked via perl's 'prove' command, wrapped in PostgreSQL
makefiles to handle instance setup etc. See the $(prove_check) and
$(prove_installcheck) targets in Makefile.global. By default every test in the
t/ subdirectory is run. Individual test(s) can be run instead by passing
something like PROVE_TESTS="t/001_testname.pl t/002_othertestname.pl" to make.

By default, to keep the noise low during runs, we do not set any flags via
PROVE_FLAGS, but this can be done on the 'make' command line if desired, eg:

make check-world PROVE_FLAGS='--verbose'

When a test fails, the terminal output from 'prove' is usually not sufficient
to diagnose the problem.  Look into the log files that are left under
tmp_check/log/ to get more info.  Files named 'regress_log_XXX' are log
output from the perl test scripts themselves, and should be examined first.
Other files are postmaster logs, and may be helpful as additional data.

The tests default to a timeout of 180 seconds for many individual operations.
Slow hosts may avoid load-induced, spurious failures by setting environment
variable PG_TEST_TIMEOUT_DEFAULT to some number of seconds greater than 180.
Developers may see faster failures by setting that environment variable to
some lesser number of seconds.

Data directories will also be left behind for analysis when a test fails;
they are named according to the test filename.  But if the environment
variable PG_TEST_NOCLEAN is set, data directories will be retained
regardless of test status.


Writing tests
-------------

You should prefer to write tests using pg_regress in src/test/regress, or
isolation tester specs in src/test/isolation, if possible. If not, check to
see if your new tests make sense under an existing tree in src/test, like
src/test/ssl, or should be added to one of the suites for an existing utility.

Note that all tests and test tools should have perltidy run on them before
patches are submitted, using perltidy --profile=src/tools/pgindent/perltidyrc

Tests are written using Perl's Test::More with some PostgreSQL-specific
infrastructure from src/test/perl providing node management, support for
invoking 'psql' to run queries and get results, etc. You should read the
documentation for Test::More before trying to write tests.

Test scripts in the t/ subdirectory of a suite are executed in alphabetical
order.

Each test script should begin with:

    use strict;
    use warnings;
    use PostgreSQL::Test::Cluster;
    use PostgreSQL::Test::Utils;
    use Test::More;

then it will generally need to set up one or more nodes, run commands
against them and evaluate the results. For example:

    my $node = PostgreSQL::Test::Cluster->new('primary');
    $node->init;
    $node->start;

    my $ret = $node->safe_psql('postgres', 'SELECT 1');
    is($ret, '1', 'SELECT 1 returns 1');

    $node->stop('fast');

Each test script should end with:

	done_testing();

Test::Builder::Level controls how far up in the call stack a test will look
at when reporting a failure.  This should be incremented by any subroutine
which directly or indirectly calls test routines from Test::More, such as
ok() or is():

    local $Test::Builder::Level = $Test::Builder::Level + 1;

Read the documentation for more on how to write tests:

    perldoc Test::More
    perldoc Test::Builder

For available PostgreSQL-specific test methods and some example tests read the
perldoc for the test modules, e.g.:

    perldoc src/test/perl/PostgreSQL/Test/Cluster.pm

Portability
-----------

Avoid using any bleeding-edge Perl features.  We have buildfarm animals
running Perl versions as old as 5.14, so your tests will be expected
to pass on that.

Also, do not use any non-core Perl modules except IPC::Run.  Or, if you
must do so for a particular test, arrange to skip the test when the needed
module isn't present.  If unsure, you can consult Module::CoreList to find
out whether a given module is part of the Perl core, and which module
versions shipped with which Perl releases.

One way to test for compatibility with old Perl versions is to use
perlbrew; see http://perlbrew.pl .  After installing that, do

    export PERLBREW_CONFIGURE_FLAGS='-de -Duseshrplib'
    perlbrew --force install 5.14.0
    perlbrew use 5.14.0
    perlbrew install-cpanm
    cpanm install Test::Simple@0.98
    cpanm install IPC::Run@0.79
    cpanm install ExtUtils::MakeMaker@6.50  # downgrade

TIP: if Test::Simple's utf8 regression test hangs up, try setting a
UTF8-compatible locale, e.g. "export LANG=en_US.utf8".

Then re-run Postgres' configure to ensure the correct Perl is used when
running tests.  To verify that the right Perl was found:

    grep ^PERL= config.log

Due to limitations of cpanm, this recipe doesn't exactly duplicate the
module list of older buildfarm animals.  The discrepancies should seldom
matter, but if you want to be sure, bypass cpanm and instead manually
install the desired versions of Test::Simple and IPC::Run.