postgresql/src
Tomas Vondra c1a5caea82 Do not decode TOAST data for table rewrites
During table rewrites (VACUUM FULL and CLUSTER), the main heap is logged
using XLOG / FPI records, and thus (correctly) ignored in decoding.
But the associated TOAST table is WAL-logged as plain INSERT records,
and so was logically decoded and passed to reorder buffer.

That has severe consequences with TOAST tables of non-trivial size.
Firstly, reorder buffer has to keep all those changes, possibly spilling
them to a file, incurring I/O costs and disk space.

Secondly, ReoderBufferCommit() was stashing all those TOAST chunks into
a hash table, which got discarded only after processing the row from the
main heap.  But as the main heap is not decoded for rewrites, this never
happened, so all the TOAST data accumulated in memory, resulting either
in excessive memory consumption or OOM.

The fix is simple, as commit e9edc1ba already introduced infrastructure
(namely HEAP_INSERT_NO_LOGICAL flag) to skip logical decoding of TOAST
tables, but it only applied it to system tables.  So simply use it for
all TOAST data in raw_heap_insert().

That would however solve only the memory consumption issue - the TOAST
changes would still be decoded and added to the reorder buffer, and
spilled to disk (although without TOAST tuple data, so much smaller).
But we can solve that by tweaking DecodeInsert() to just ignore such
INSERT records altogether, using XLH_INSERT_CONTAINS_NEW_TUPLE flag,
instead of skipping them later in ReorderBufferCommit().

Review: Masahiko Sawada
Discussion: https://www.postgresql.org/message-id/flat/1a17c643-e9af-3dba-486b-fbe31bc1823a%402ndquadrant.com
Backpatch: 9.4-, where logical decoding was introduced
2018-11-28 01:53:29 +01:00
..
backend Do not decode TOAST data for table rewrites 2018-11-28 01:53:29 +01:00
bin Fix translation of special characters in psql's LaTeX output modes. 2018-11-26 17:32:51 -05:00
common Enlarge find_other_exec's meager fgets buffer 2018-04-19 10:45:15 -03:00
include Fix float-to-integer coercions to handle edge cases correctly. 2018-11-24 12:45:50 -05:00
interfaces Second try at fixing numeric data passed through an ECPG SQLDA. 2018-11-14 11:27:31 -05:00
makefiles Prevent accidental linking of system-supplied copies of libpq.so etc. 2018-07-09 17:23:32 -04:00
pl Still further rethinking of build changes for macOS Mojave. 2018-10-18 14:55:23 -04:00
port Stamp 9.4.20. 2018-11-05 16:51:23 -05:00
template Yet further rethinking of build changes for macOS Mojave. 2018-11-02 18:54:00 -04:00
test Update additional float4/8 expected-output files. 2018-11-24 13:53:12 -05:00
timezone Sync our copy of the timezone library with IANA release tzcode2018g. 2018-10-31 09:48:24 -04:00
tools Add valgrind suppressions for wcsrtombs optimizations 2018-11-18 00:10:15 +01:00
tutorial pgindent run for 9.4 2014-05-06 12:12:18 -04:00
.gitignore Convert cvsignore to gitignore, and add .gitignore for build targets. 2010-09-22 12:57:04 +02:00
bcc32.mak Autoconfiscate selection of 64-bit int type for 64-bit large object API. 2012-10-07 21:52:43 -04:00
DEVELOPERS Replace a couple of references to files that no longer exist in the source 2009-05-04 08:08:47 +00:00
Makefile Install TAP test infrastructure so it's available for extension testing. 2016-09-23 15:50:00 -04:00
Makefile.global.in Yet further rethinking of build changes for macOS Mojave. 2018-11-02 18:54:00 -04:00
Makefile.shlib Prevent accidental linking of system-supplied copies of libpq.so etc. 2018-07-09 17:23:32 -04:00
nls-global.mk nls-global.mk: search build dir for source files, too 2016-06-07 18:55:18 -04:00
win32.mak Autoconfiscate selection of 64-bit int type for 64-bit large object API. 2012-10-07 21:52:43 -04:00