Compare commits

...

6 commits

Author SHA1 Message Date
8ce4b93292 [ADD] migration: neutralize the COW views that would break a version bump
key is the only thing pairing a website copy with the module view it came
from. A copy whose key matches nothing is never paired, so it never receives
the new inherit_id, never changes shape, and never joins a view combination.
Renaming the key is therefore enough to take it out of the way:

    UPDATE ir_ui_view SET key = '<prefix>.' || key, active = false

active = false alone would NOT work: an inactive copy keeping the same key
still shadows the module view. Nothing is deleted either, so inherit_id
ondelete='restrict' and the website_page foreign keys are never touched, and
the old arch stays in database as a readable archive. --restore undoes it.

Written in plain psql on purpose: it runs on a database not yet migrated,
where starting an Odoo shell of the target version is not guaranteed. It is a
systematic step driven by the detector, valid for every bump, so it lives in
the loop rather than in a per-version fix_migration file.

Offered rather than forced, since those copies carry real customizations.

Verified on a throwaway copy: dry-run writes nothing; apply renames the key and
deactivates while keeping the 4058-char arch; a second apply is a no-op;
restore returns the row to its exact initial state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-01 01:04:38 -04:00
1e4cc4f471 [ADD] migration: snapshot and diff the website COW views at each version bump
A version bump rewrites website views without announcing any of it, so "the
site looks wrong" after a migration is currently unanswerable.

snapshot_cow_views.py records every website_id view (key, mode, inherit_id,
active, arch and its md5) and diffs two snapshots. Rows come back as JSON
straight from Postgres because an arch holds newlines and pipes; the column
list is intersected with information_schema, since ir_ui_view does not expose
the same columns from 12.0 to 18.0. Snapshots hold customer template content,
so they go under private/ and stay out of git.

todo_upgrade.py takes one before and one after each OpenUpgrade run, then
prints the diff. Both are non-blocking: forensic material must never stop an
upgrade.

Measured on the real 12.0 -> 13.0 jump, the diff shows what no log reported:
71 -> 68 copies, 16 deleted and 13 created. portal.frontend_layout is not
converted but DELETED (id 2670) and RECREATED (id 3397), with its children
re-parented from one to the other. The four theme_technolibre copies,
muk_web_branding, project_agile and erplibre_website_snippets_basic_html are
dropped outright, and website_crm.contactus_thanks is renamed to
website_form.contactus_thanks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-01 00:58:54 -04:00
1ec37defc3 [FIX] check_cow_views: detect on arch shape, not on mode
Two blind spots made the detector miss real breakages.

1. Comparing « mode » is not the right test. What decides the shape an arch
   must have is whether the target declares an inherit_id: with one, the arch
   must be inheritance specs (<data>, <xpath>, position=); without one, it must
   be a standalone template. A view moving from a root template to
   « inherit_id + primary="True" » keeps mode='primary' on BOTH sides, so the
   old test reported nothing while the copy still broke. The comparison is now
   (target declares inherit) vs (stored arch is spec-shaped), and each finding
   carries the reason. A mode change with a matching shape is still reported,
   as a lesser warning.

   arch_db is read as text up to 15.0 and as jsonb from 16.0; both are handled.

2. A module renamed upstream was reported as « module absent », hiding every
   view it owns. renamed_modules is now read from the target OpenUpgrade
   apriori.py (21 entries for 13.0, 56 for 14.0, 39 for 16.0, 20 for 18.0) and
   used before concluding the module is gone.

Also correct the advice printed for a copy at risk: deactivating it is not
enough, an inactive copy keeping the same key still shadows the module view.
Renaming the key is what actually unpairs it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-01 00:54:44 -04:00
045ccc1c85 [ADD] migration: predict the website COW views a version bump will break
A copy-on-write view freezes the structure of the module view it was copied
from. When that module view changes mode between two versions, the copy keeps
an arch written for the old mode and the upgrade dies on
« Element ... cannot be located in parent view », hours into the migration.

Measured on a real 12.0 database: portal.frontend_layout is declared primary in
12.0 (full QWeb template, so a full-template arch is legitimate) and becomes an
extension in 13.0 (inherit_id + xpath). The 2021 COW copy follows the module,
turns into an extension, and keeps its 12.0 arch. The customization is genuine;
the breakage is produced by the migration.

check_cow_views.py compares the mode stored in database with the mode declared
in the target version sources and sorts the views in three buckets: at risk
(mode change), module absent from the target version, and pages made from the
editor (not at risk). Read-only, and called from step 2 so the list is known
before the long version loop rather than during it.

On the reference database it reports exactly the view that broke the upgrade,
plus 7 views whose module is gone in 13.0 (custom theme included).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-31 23:28:54 -04:00
4536c5bc40 [IMP] migration: per-database uninstall list, with a stated reason
Which modules must be uninstalled before a version bump depends on the data of
one specific database, so that list does not belong to a shared versioned file.

- Read private/odoo/migration/<database>/uninstall_module_list_odooXX0_to_odooYY0.txt
  first, then the shared versioned defaults under script/odoo/migration/, and
  merge them (duplicates dropped). private/ mirrors script/, the convention
  already used by script/todo/todo.json -> private/todo/todo_override.json.
- Fix the parser. It was « f.readline().split() »: only the FIRST line was kept,
  so a multi-line list was silently truncated, and a comma-separated list turned
  into one bogus module name. On a file starting with a comment it returned the
  words of that comment as module names. It now reads every line and accepts
  commas, several names per line, blank lines and comments.
- Support a « # reason » justification per module, printed before uninstalling;
  a module with no reason is flagged. Removing a module must stay reviewable.
- Ignore private/odoo/ in git: these lists describe one database.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-31 23:25:43 -04:00
ab9f94cc28 [FIX] migration: stop recording failed steps as successful
The database migration recorded steps as done even when they had failed, so a
resume skipped them and the loop kept going on a broken database.

execute.py
- exec_command_live() left exit_code = None when an exception was raised. None
  is falsy, so « if not status: » marked the step done and
  « if status and wait_at_error » skipped the error prompt: a crashed command
  was reported as a success. Both except blocks now set exit_code = 1.
- The « no Odoo version installed » path returned a bare -1 while callers
  unpack a tuple (status, cmd), raising ValueError instead of surfacing the
  failure. It now returns the same shape the caller asked for.

todo_upgrade.py
- todo_upgrade_execute(): treat a None status as a failure (defence in depth).
- Database migration (OpenUpgrade): the return code was discarded, with an
  explicit « TODO detect error », and the state was written unconditionally.
  It is now captured; on failure the loop stops instead of migrating the next
  version on top of a half-migrated database.
- Neutralization: the flag was set before the « if not status » test, making
  that test dead code. Removed the unconditional assignment.
- Clone and fix-migration hook: their return codes were never captured, so the
  steps were marked done whatever happened. Both are now checked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-31 23:22:02 -04:00
9 changed files with 1056 additions and 21 deletions

5
private/.gitignore vendored
View file

@ -1 +1,6 @@
*.kdbx
# Per-database migration lists (modules to uninstall/install before a version
# bump). They describe one specific database, never a shared default, so they
# must not be versioned. Shared defaults belong to script/odoo/migration/.
odoo/

View file

@ -109,7 +109,16 @@ class Execute:
_logger.error(
f"You cannot execute Odoo command if no version is installed. Command : {command}"
)
return -1
# Return the SAME shape the caller asked for. A bare int here
# made callers doing « status, cmd = exec_command_live(...) »
# crash with ValueError instead of seeing the failure.
if return_status_and_output_and_command:
return 1, command, []
if return_status_and_command:
return 1, command
if return_status_and_output:
return 1, []
return 1
command = f"source ./.venv.{source_odoo}/bin/activate && {command}"
if new_window and self.cmd_source_default:
command = self.cmd_source_default % command
@ -154,7 +163,12 @@ class Execute:
if process.returncode != 0 and not quiet:
print("Command returned error code:" f" {process.returncode}")
# An exception MUST report a failure. exit_code stays None otherwise,
# and None is falsy: callers testing « if not status: » would mark the
# step as done, and « if status and wait_at_error » would skip the error
# prompt. A crashed command was therefore recorded as a success.
except FileNotFoundError:
exit_code = 1
if not quiet:
if "password" in command:
print(
@ -164,6 +178,7 @@ class Execute:
else:
print(f"Error: Command '{command}' not found.")
except Exception as e:
exit_code = 1
if not quiet:
print(f"An error occurred: {e}")
process_end_time = time.time()

View file

@ -21,5 +21,46 @@ source ./.venv.odoo15.0_python3.8.20/bin/activate && cat ./script/odoo/migration
<!-- [en] -->
Check [uninstall_module_list_odoo140_to_odoo150.txt](uninstall_module_list_odoo140_to_odoo150.txt)
## Module lists to uninstall
Before a version bump, the migration uninstalls the modules listed in
`uninstall_module_list_odoo<from>_to_odoo<to>.txt`. Two locations are read, the
private one first, and the results are merged (duplicates dropped):
1. `private/odoo/migration/<database>/uninstall_module_list_odooXX0_to_odooYY0.txt`
— specific to ONE database, not versioned. Which modules must be dropped
depends on the data, so this is where nearly every entry belongs.
2. `script/odoo/migration/uninstall_module_list_odooXX0_to_odooYY0.txt`
— shared defaults, versioned, valid for every database.
Syntax: one module per line, with a justification after `#`. Commas and several
names per line are accepted; blank lines and full-line comments are ignored.
A module without a stated reason is flagged at runtime: removing a module is a
decision someone must be able to review later.
<!-- [fr] -->
Consultez [uninstall_module_list_odoo140_to_odoo150.txt](uninstall_module_list_odoo140_to_odoo150.txt)
## Listes de modules à désinstaller
Avant une montée de version, la migration désinstalle les modules listés dans
`uninstall_module_list_odoo<depuis>_to_odoo<vers>.txt`. Deux emplacements sont
lus, le privé d'abord, puis fusionnés (doublons éliminés) :
1. `private/odoo/migration/<base>/uninstall_module_list_odooXX0_to_odooYY0.txt`
— propre à UNE base de données, non versionné. Les modules à supprimer
dépendent des données : c'est ici que va la quasi-totalité des entrées.
2. `script/odoo/migration/uninstall_module_list_odooXX0_to_odooYY0.txt`
— valeurs par défaut partagées, versionnées, valables pour toute base.
Syntaxe : un module par ligne, avec une justification après `#`. Les virgules et
plusieurs noms par ligne sont acceptés ; lignes vides et commentaires pleine
ligne sont ignorés. Un module sans raison est signalé à l'exécution : supprimer
un module est une décision qui doit pouvoir être relue plus tard.
<!-- [common] -->
```
queue_job # blocks 12->13, trigger queue_job_notify
mgmtsystem_hazard # not ported to 13.0
web_syncer # dropped upstream
```

View file

@ -7,4 +7,27 @@ Exécutez ce script lors de la migration de base de données. Exemple :
source ./.venv.odoo15.0_python3.8.20/bin/activate && cat ./script/odoo/migration/fix_migration_odoo140_to_odoo150.py | ./odoo15.0/odoo/odoo-bin shell -d DATABASE
```
Consultez [uninstall_module_list_odoo140_to_odoo150.txt](uninstall_module_list_odoo140_to_odoo150.txt)
Consultez [uninstall_module_list_odoo140_to_odoo150.txt](uninstall_module_list_odoo140_to_odoo150.txt)
## Listes de modules à désinstaller
Avant une montée de version, la migration désinstalle les modules listés dans
`uninstall_module_list_odoo<depuis>_to_odoo<vers>.txt`. Deux emplacements sont
lus, le privé d'abord, puis fusionnés (doublons éliminés) :
1. `private/odoo/migration/<base>/uninstall_module_list_odooXX0_to_odooYY0.txt`
— propre à UNE base de données, non versionné. Les modules à supprimer
dépendent des données : c'est ici que va la quasi-totalité des entrées.
2. `script/odoo/migration/uninstall_module_list_odooXX0_to_odooYY0.txt`
— valeurs par défaut partagées, versionnées, valables pour toute base.
Syntaxe : un module par ligne, avec une justification après `#`. Les virgules et
plusieurs noms par ligne sont acceptés ; lignes vides et commentaires pleine
ligne sont ignorés. Un module sans raison est signalé à l'exécution : supprimer
un module est une décision qui doit pouvoir être relue plus tard.
```
queue_job # blocks 12->13, trigger queue_job_notify
mgmtsystem_hazard # not ported to 13.0
web_syncer # dropped upstream
```

View file

@ -8,3 +8,26 @@ source ./.venv.odoo15.0_python3.8.20/bin/activate && cat ./script/odoo/migration
```
Check [uninstall_module_list_odoo140_to_odoo150.txt](uninstall_module_list_odoo140_to_odoo150.txt)
## Module lists to uninstall
Before a version bump, the migration uninstalls the modules listed in
`uninstall_module_list_odoo<from>_to_odoo<to>.txt`. Two locations are read, the
private one first, and the results are merged (duplicates dropped):
1. `private/odoo/migration/<database>/uninstall_module_list_odooXX0_to_odooYY0.txt`
— specific to ONE database, not versioned. Which modules must be dropped
depends on the data, so this is where nearly every entry belongs.
2. `script/odoo/migration/uninstall_module_list_odooXX0_to_odooYY0.txt`
— shared defaults, versioned, valid for every database.
Syntax: one module per line, with a justification after `#`. Commas and several
names per line are accepted; blank lines and full-line comments are ignored.
A module without a stated reason is flagged at runtime: removing a module is a
decision someone must be able to review later.
```
queue_job # blocks 12->13, trigger queue_job_notify
mgmtsystem_hazard # not ported to 13.0
web_syncer # dropped upstream
```

View file

@ -0,0 +1,347 @@
#!/usr/bin/env python3
# © 2021-2026 TechnoLibre (http://www.technolibre.ca)
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl)
"""Predict which website COW views will break on the next version bump.
Background
----------
When a website view is customized, Odoo makes a copy-on-write (COW) copy tied
to a website_id. That copy freezes the arch AND the structure of the module
view it was copied from.
A module view can change structure between two Odoo versions. Example measured
on a real 12.0 database: ``portal.frontend_layout`` is declared ``primary`` in
12.0 (a full QWeb template) and becomes an ``extension`` in 13.0
(``inherit_id="web.frontend_layout"`` + xpath). During the upgrade the COW copy
follows the module and becomes an extension, but keeps its 12.0 full-template
arch. Odoo then applies the ``<t t-name=...>`` root as an inheritance spec,
cannot find it in the parent, and the whole upgrade stops on::
ValueError: Element '<t ... t-name="portal.frontend_layout">'
cannot be located in parent view
So the rule is:
a COW view breaks when the shape its arch must have changes between
version N and version N+1.
The discriminant is NOT ``mode``. What decides the required shape is whether
the target declares an ``inherit_id``: if it does, the arch must be inheritance
specs (``<data>``, ``<xpath>``, ``position=``); if it does not, the arch must be
a standalone template. Comparing ``mode`` alone misses a real case: a view
moving from a root template to ``inherit_id`` + ``primary="True"`` keeps
``mode='primary'`` on both sides yet still has to change shape, and the copy
still breaks.
That is predictable *before* starting a multi-hour migration: the stored arch is
in the database, and the required shape is declared in the target version
sources. This script compares the two and reports the views at risk.
It only reads: no database write, no source modification.
"""
import argparse
import glob
import json
import os
import re
import subprocess
import sys
import xml.etree.ElementTree as ET
# A view whose module counterpart cannot be found at all.
MODE_UNKNOWN = "unknown"
# arch_db is text up to 15.0 and jsonb from 16.0 ({"en_US": "<data>..."}).
RE_XML_DECLARATION = re.compile(r"<\?xml.*?\?>", re.DOTALL)
RE_FIRST_TAG = re.compile(r"<\s*([A-Za-z_][\w.:-]*)")
# Tags that carry inheritance specs rather than a standalone template.
SPEC_ROOT_TAG = ("data", "xpath")
def query_cow_views(database):
"""Return [(id, key, mode, website_id, arch)] for every website COW view."""
sql = (
"SELECT id, COALESCE(key, ''), mode, website_id,"
" replace(left(COALESCE(arch_db::text, ''), 400), chr(10), ' ')"
" FROM ir_ui_view WHERE website_id IS NOT NULL ORDER BY id;"
)
result = subprocess.run(
["psql", "-d", database, "-tAF", "|", "-c", sql],
capture_output=True,
text=True,
)
if result.returncode:
raise RuntimeError(
f"Cannot read views from '{database}': {result.stderr.strip()}"
)
lst_view = []
for line in result.stdout.splitlines():
if not line.strip():
continue
view_id, key, mode, website_id, arch = line.split("|", 4)
lst_view.append((int(view_id), key, mode, website_id, arch))
return lst_view
def arch_is_inheritance_spec(arch):
"""True when the arch holds inheritance specs, not a standalone template.
This is the real discriminant, not ``mode``. A view declared with an
``inherit_id`` must hold specs (``<data>``, ``<xpath>``, or an element with
a ``position``); a root view holds a full template (``<t t-name=...>``,
``<form>``, ...). A copy that keeps the wrong form for what the target
version expects is exactly what raises « cannot be located in parent view ».
"""
if not arch:
return None
# From 16.0 arch_db is jsonb: take any translation, the structure is shared.
if arch.lstrip().startswith("{"):
try:
translations = json.loads(arch)
arch = next(iter(translations.values()), "")
except (ValueError, StopIteration):
# Truncated jsonb: fall through and look at the raw text.
pass
arch = RE_XML_DECLARATION.sub("", arch or "")
match = RE_FIRST_TAG.search(arch)
if not match:
return None
if match.group(1).lower() in SPEC_ROOT_TAG:
return True
# <field name="x" position="after"> style specs.
return "position=" in arch[: match.end() + 200]
def load_renamed_modules(odoo_version):
"""Return {old_module: new_module} from the target OpenUpgrade apriori.py.
Without this a module renamed upstream looks absent, and every view of that
module is misreported as « module gone » instead of being checked.
"""
pattern = os.path.join(odoo_version, "**", "apriori.py")
for file_path in sorted(glob.glob(pattern, recursive=True)):
data_vars = {}
try:
with open(file_path, "r", encoding="utf-8") as f:
exec(f.read(), data_vars) # noqa: S102 - upstream data file
except Exception:
# A broken or exotic apriori.py must not stop the whole report.
continue
renamed = data_vars.get("renamed_modules")
if isinstance(renamed, dict) and renamed:
return renamed
return {}
def find_module_dir(odoo_version, module_name):
"""Locate a module directory inside an odoo<version> tree."""
lst_pattern = [
os.path.join(odoo_version, "odoo", "addons", module_name),
os.path.join(odoo_version, "odoo", "odoo", "addons", module_name),
os.path.join(odoo_version, "addons", "*", module_name),
]
for pattern in lst_pattern:
for path in sorted(glob.glob(pattern)):
if os.path.isdir(path):
return path
return None
def declared_view_shape(module_dir, template_id):
"""Return (mode, inherits) for a view declared in the sources, else None.
``inherits`` is what really matters: a declared inherit_id means the arch
must be inheritance specs. ``mode`` is kept because it is still worth
reporting, but it is NOT a reliable discriminant: a view moving from a root
template to « inherit_id + primary="True" » keeps mode='primary' on both
sides while its arch shape has to change.
"""
pattern = os.path.join(module_dir, "**", "*.xml")
for file_path in sorted(glob.glob(pattern, recursive=True)):
try:
root = ET.parse(file_path).getroot()
except ET.ParseError:
continue
for element in root.iter():
if element.get("id") != template_id:
continue
if element.tag == "template":
inherits = bool(element.get("inherit_id"))
if str(element.get("primary", "")).lower() in ("true", "1"):
return "primary", inherits
return ("extension" if inherits else "primary"), inherits
if (
element.tag == "record"
and element.get("model") == "ir.ui.view"
):
mode = None
inherits = False
for field in element.findall("field"):
if field.get("name") == "mode":
mode = (field.text or "").strip()
elif field.get("name") == "inherit_id":
inherits = True
if mode:
return mode, inherits
return ("extension" if inherits else "primary"), inherits
return None
def analyse(database, target_version):
"""Sort COW views into three buckets by comparing with the target sources.
- at_risk : the module view changes mode -> the copy will break
- module_absent : the module itself is gone in the target version
- no_counterpart : no module view with that id, so it is a page or a record
created from the editor. Normal, and not at risk.
"""
lst_at_risk = []
lst_module_absent = []
lst_no_counterpart = []
cache_shape = {}
renamed_modules = load_renamed_modules(target_version)
for view_id, key, mode, website_id, arch in query_cow_views(database):
if not key or "." not in key:
continue
if key not in cache_shape:
module_name, _, template_id = key.partition(".")
module_dir = find_module_dir(target_version, module_name)
if module_dir is None and module_name in renamed_modules:
module_dir = find_module_dir(
target_version, renamed_modules[module_name]
)
if module_dir is None:
cache_shape[key] = MODE_UNKNOWN
else:
cache_shape[key] = declared_view_shape(module_dir, template_id)
shape = cache_shape[key]
if shape == MODE_UNKNOWN:
lst_module_absent.append((view_id, key, mode, website_id))
continue
if shape is None:
lst_no_counterpart.append((view_id, key, mode, website_id))
continue
target_mode, target_inherits = shape
is_spec = arch_is_inheritance_spec(arch)
# The decisive test: the target expects inheritance specs but the copy
# holds a standalone template, or the reverse.
if is_spec is not None and target_inherits != is_spec:
reason = (
"target inherits, copy holds a standalone template"
if target_inherits
else "target is a root view, copy holds inheritance specs"
)
lst_at_risk.append(
(view_id, key, mode, target_mode, website_id, reason)
)
elif target_mode != mode:
# Shape is fine but the mode moves: worth reporting, less severe.
lst_at_risk.append(
(
view_id,
key,
mode,
target_mode,
website_id,
"mode changes, arch shape unchanged",
)
)
return lst_at_risk, lst_module_absent, lst_no_counterpart
def main():
parser = argparse.ArgumentParser(
description=(
"Report website COW views that will break on the next Odoo"
" version bump (read-only)."
)
)
parser.add_argument(
"-d", "--database", required=True, help="database to inspect"
)
parser.add_argument(
"-t",
"--target_version",
required=True,
help="target Odoo source directory, e.g. odoo13.0",
)
parser.add_argument(
"-v",
"--verbose",
action="store_true",
help="also list the editor-made pages, which are not at risk",
)
config = parser.parse_args()
if not os.path.isdir(config.target_version):
print(
f"❌ Target version directory '{config.target_version}' not found."
)
return 1
lst_at_risk, lst_module_absent, lst_no_counterpart = analyse(
config.database, config.target_version
)
if not lst_at_risk:
print(
"✅ -> No website COW view changes mode in"
f" {config.target_version}."
)
else:
print(
f"⚠️ {len(lst_at_risk)} website COW view(s) will break when moving"
f" to {config.target_version}: the copy keeps an arch whose shape"
" no longer matches what the target module view expects."
)
for (
view_id,
key,
mode,
target_mode,
website_id,
reason,
) in lst_at_risk:
print(
f" - id={view_id} website={website_id} {key}"
f" : {mode} -> {target_mode} ({reason})"
)
print(
" Arbitrate BEFORE launching the migration. To neutralize a copy,"
" rename its key (UPDATE ir_ui_view SET key='zz_cow_archive.'||key,"
" active=false): an unmatched key is never paired with the module"
" view, so the copy never receives the new inherit_id. Setting"
" active=false alone is NOT enough -- an inactive copy that keeps"
" the same key still shadows the module view."
)
if lst_module_absent:
print(
f"ℹ {len(lst_module_absent)} COW view(s) belong to a module absent"
f" from {config.target_version}:"
)
for view_id, key, mode, website_id in lst_module_absent:
print(f" - id={view_id} website={website_id} {key} ({mode})")
if lst_no_counterpart:
print(
f"ℹ {len(lst_no_counterpart)} COW view(s) are pages or records made"
" from the website editor (no module view of that name): not at"
" risk." + ("" if config.verbose else " Use -v to list them.")
)
if config.verbose:
for view_id, key, mode, website_id in lst_no_counterpart:
print(f" - id={view_id} website={website_id} {key} ({mode})")
# Informative only: never fail the migration on a warning.
return 0
if __name__ == "__main__":
sys.exit(main())

View file

@ -0,0 +1,158 @@
#!/usr/bin/env python3
# © 2021-2026 TechnoLibre (http://www.technolibre.ca)
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl)
"""Neutralize the website COW views that would break a version bump.
How it works
------------
``key`` is the only thing that pairs a website copy with the module view it
came from. A copy whose key matches nothing is never paired, so it never
receives the new ``inherit_id``, never changes shape, and never takes part in
any view combination. Renaming the key is therefore enough to take a copy out
of the way::
UPDATE ir_ui_view SET key = '<prefix>.' || key, active = false WHERE id = ?
Setting ``active = false`` alone would NOT work: an inactive copy that keeps
the same key still shadows the module view.
Nothing is deleted, so ``inherit_id ondelete='restrict'`` and the
``website_page`` foreign keys are never touched, and the 12.0 arch stays in
database as a readable archive. ``--restore`` puts everything back.
Plain psql on purpose: this must run on a database that has not been migrated
yet, where starting an Odoo shell of the target version is not guaranteed.
"""
import argparse
import os
import subprocess
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from check_cow_views import analyse # noqa: E402
DEFAULT_PREFIX = "zz_cow_archive"
def run_psql(database, sql):
"""Run a statement and return stdout, raising on failure."""
result = subprocess.run(
["psql", "-d", database, "-tAc", sql],
capture_output=True,
text=True,
)
if result.returncode:
raise RuntimeError(
f"Query failed on '{database}': {result.stderr.strip()}"
)
return result.stdout.strip()
def neutralize(database, lst_view_id, prefix):
"""Rename the key of the given views and deactivate them."""
if not lst_view_id:
return 0
ids = ",".join(str(view_id) for view_id in lst_view_id)
output = run_psql(
database,
"WITH updated AS ("
f" UPDATE ir_ui_view SET key = '{prefix}.' || key, active = false"
f" WHERE id IN ({ids}) AND key NOT LIKE '{prefix}.%'"
" RETURNING 1) SELECT count(*) FROM updated;",
)
return int(output or 0)
def restore(database, prefix):
"""Undo a neutralization: strip the prefix and reactivate."""
output = run_psql(
database,
"WITH updated AS ("
f" UPDATE ir_ui_view SET key = substring(key from {len(prefix) + 2}),"
" active = true"
f" WHERE key LIKE '{prefix}.%'"
" RETURNING 1) SELECT count(*) FROM updated;",
)
return int(output or 0)
def main():
parser = argparse.ArgumentParser(
description=(
"Neutralize the website COW views that would break a version"
" bump, by renaming their key. Dry-run unless --apply."
)
)
parser.add_argument("-d", "--database", required=True)
parser.add_argument(
"-t",
"--target_version",
help="target Odoo source directory, e.g. odoo13.0",
)
parser.add_argument(
"--prefix",
default=DEFAULT_PREFIX,
help=f"archive prefix for the key (default: {DEFAULT_PREFIX})",
)
parser.add_argument(
"--apply", action="store_true", help="actually write to the database"
)
parser.add_argument(
"--restore",
action="store_true",
help="undo a previous neutralization and reactivate the copies",
)
config = parser.parse_args()
if config.restore:
count = restore(config.database, config.prefix)
print(f"✅ -> {count} COW view(s) restored on '{config.database}'.")
return 0
if not config.target_version:
parser.error("--target_version is required unless --restore is used")
if not os.path.isdir(config.target_version):
print(
f"❌ Target version directory '{config.target_version}' not found."
)
return 1
lst_at_risk, _, _ = analyse(config.database, config.target_version)
if not lst_at_risk:
print("✅ -> No website COW view to neutralize.")
return 0
print(
f"⚠️ {len(lst_at_risk)} website COW view(s) would break the bump to"
f" {config.target_version}:"
)
lst_view_id = []
for view_id, key, mode, target_mode, website_id, reason in lst_at_risk:
lst_view_id.append(view_id)
print(
f" - id={view_id} website={website_id} {key}"
f" : {mode} -> {target_mode} ({reason})"
)
if not config.apply:
print(
"ℹ Dry-run. Add --apply to rename their key to"
f" '{config.prefix}.<key>' and deactivate them. Reversible with"
" --restore; the arch stays in database."
)
return 0
count = neutralize(config.database, lst_view_id, config.prefix)
print(
f"✅ -> {count} COW view(s) neutralized (key prefixed with"
f" '{config.prefix}.', deactivated). The arch is kept as an archive;"
" use --restore to undo."
)
return 0
if __name__ == "__main__":
sys.exit(main())

View file

@ -0,0 +1,213 @@
#!/usr/bin/env python3
# © 2021-2026 TechnoLibre (http://www.technolibre.ca)
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl)
"""Snapshot the website COW views, and diff two snapshots.
Why
---
A version bump rewrites website views in ways nobody announces: OpenUpgrade
converts Bootstrap markup on every ``website_id IS NOT NULL`` view, modules
rewrite the copies they own, and some copies simply disappear. Without a
before/after record, "the site looks wrong" is unanswerable.
Taking a snapshot before and after each jump turns that into a diff: which copy
lost its arch, which changed mode, which was renamed, which vanished.
Snapshots hold customer template content, so they belong under ``private/``
and are never versioned.
Usage::
snapshot_cow_views.py -d <database> --label before_13
snapshot_cow_views.py -d <database> --label after_13
snapshot_cow_views.py --diff <before.json> <after.json>
"""
import argparse
import datetime
import hashlib
import json
import os
import subprocess
import sys
# Columns worth recording. ir_ui_view does not expose the same set across 12.0
# to 18.0, so the query keeps only those that actually exist.
WANTED_COLUMN = [
"id",
"key",
"name",
"type",
"mode",
"active",
"priority",
"website_id",
"inherit_id",
"arch_fs",
"arch_updated",
]
DEFAULT_DIR = os.path.join("private", "odoo", "migration")
def run_psql(database, sql):
"""Run a read-only query and return stdout."""
result = subprocess.run(
["psql", "-d", database, "-tAc", sql],
capture_output=True,
text=True,
)
if result.returncode:
raise RuntimeError(
f"Query failed on '{database}': {result.stderr.strip()}"
)
return result.stdout
def existing_columns(database):
"""Column names of ir_ui_view present in this database."""
output = run_psql(
database,
"SELECT column_name FROM information_schema.columns"
" WHERE table_name = 'ir_ui_view';",
)
return {line.strip() for line in output.splitlines() if line.strip()}
def collect(database):
"""Return the list of COW views as plain dicts, arch included.
The rows come back as JSON straight from Postgres: an arch holds newlines
and pipes, so no hand-made separator survives it.
"""
available = existing_columns(database)
lst_column = [name for name in WANTED_COLUMN if name in available]
select = ", ".join(lst_column) + ", arch_db::text AS arch_db"
output = run_psql(
database,
"SELECT COALESCE(json_agg(row_to_json(t)), '[]'::json) FROM ("
f" SELECT {select} FROM ir_ui_view"
" WHERE website_id IS NOT NULL ORDER BY id) t;",
)
lst_view = json.loads(output or "[]")
for view in lst_view:
arch = view.pop("arch_db", None) or ""
view["arch_md5"] = hashlib.md5(arch.encode("utf-8")).hexdigest()
view["arch_len"] = len(arch)
view["arch_db"] = arch
return lst_view
def save(database, label, output_dir):
"""Write a snapshot and return its path."""
directory = output_dir or os.path.join(
DEFAULT_DIR, database, "cow_snapshots"
)
os.makedirs(directory, exist_ok=True)
lst_view = collect(database)
payload = {
"database": database,
"label": label,
"taken_at": datetime.datetime.now().isoformat(timespec="seconds"),
"count": len(lst_view),
"views": lst_view,
}
file_path = os.path.join(directory, f"{label}.json")
with open(file_path, "w", encoding="utf-8") as f:
json.dump(payload, f, indent=2, ensure_ascii=False)
print(f"✅ -> {len(lst_view)} COW view(s) recorded in {file_path}")
return file_path
def load(file_path):
with open(file_path, "r", encoding="utf-8") as f:
return json.load(f)
def diff(path_before, path_after):
"""Print what changed between two snapshots."""
before = load(path_before)
after = load(path_after)
map_before = {view["id"]: view for view in before["views"]}
map_after = {view["id"]: view for view in after["views"]}
removed = sorted(set(map_before) - set(map_after))
added = sorted(set(map_after) - set(map_before))
common = sorted(set(map_before) & set(map_after))
print(
f"📊 {before.get('label')} ({before.get('count')} views)"
f" -> {after.get('label')} ({after.get('count')} views)"
)
if removed:
print(f"❌ {len(removed)} COW view(s) disappeared:")
for view_id in removed:
view = map_before[view_id]
print(f" - id={view_id} {view.get('key')}")
if added:
print(f"➕ {len(added)} COW view(s) appeared:")
for view_id in added:
view = map_after[view_id]
print(f" - id={view_id} {view.get('key')} ({view.get('mode')})")
lst_changed = []
for view_id in common:
old, new = map_before[view_id], map_after[view_id]
lst_field = []
for field in ("key", "mode", "inherit_id", "active", "arch_md5"):
if old.get(field) != new.get(field):
if field == "arch_md5":
lst_field.append(
f"arch rewritten ({old.get('arch_len')} ->"
f" {new.get('arch_len')} chars)"
)
else:
lst_field.append(
f"{field}: {old.get(field)} -> {new.get(field)}"
)
if lst_field:
lst_changed.append((view_id, new.get("key"), lst_field))
if lst_changed:
print(f"✏️ {len(lst_changed)} COW view(s) changed:")
for view_id, key, lst_field in lst_changed:
print(f" - id={view_id} {key}")
for change in lst_field:
print(f" {change}")
if not (removed or added or lst_changed):
print("✅ -> No change on the website COW views.")
return 0
def main():
parser = argparse.ArgumentParser(
description="Snapshot website COW views, or diff two snapshots."
)
parser.add_argument("-d", "--database", help="database to snapshot")
parser.add_argument(
"-l", "--label", help="snapshot name, e.g. before_13 or after_13"
)
parser.add_argument(
"-o", "--output_dir", help="where to write (default: private/...)"
)
parser.add_argument(
"--diff",
nargs=2,
metavar=("BEFORE", "AFTER"),
help="compare two snapshot files instead of taking one",
)
config = parser.parse_args()
if config.diff:
return diff(*config.diff)
if not config.database or not config.label:
parser.error("--database and --label are required to take a snapshot")
save(config.database, config.label, config.output_dir)
return 0
if __name__ == "__main__":
sys.exit(main())

View file

@ -44,6 +44,12 @@ FILENAME_ODOO_VERSION = ".odoo-version"
LOCAL_MANIFEST = os.path.join(
".repo", "local_manifests", "erplibre_manifest.xml"
)
# Module lists for a version bump. The shared, versioned defaults live under
# script/; the per-database lists live under private/ (mirroring script/, like
# script/todo/todo.json -> private/todo/todo_override.json). Which modules must
# be dropped depends on the database, so that choice is never versioned.
PATH_MIGRATION_GLOBAL = os.path.join("script", "odoo", "migration")
PATH_MIGRATION_PRIVATE = os.path.join("private", "odoo", "migration")
class TodoUpgrade:
@ -810,8 +816,9 @@ class TodoUpgrade:
# ./script/database/migrate/process_backup_file.py --path_backup_zip image_db/db.zip --path_output_zip image_db/dbFIX.zip --word_to_delete discuss_channel_channel_type_not_null
# Puis faire un retry de la commande, sinon rien
self.dct_progression["state_1_neutralize_database"] = True
self.write_config()
# Only record the step when it actually succeeded. The previous
# unconditional assignment made the test below dead code: a failed
# neutralization was remembered as done and skipped on resume.
if not status:
self.dct_progression["state_1_neutralize_database"] = True
self.write_config()
@ -827,18 +834,24 @@ class TodoUpgrade:
)
if not is_state_4_reach_open_upgrade:
lst_module_to_uninstall = []
uninstall_module_list_file = os.path.join(
"script",
"odoo",
"migration",
f"uninstall_module_list_odoo{start_version * 10}_to_odoo{(start_version + 1) * 10}.txt",
lst_module_to_uninstall, lst_uninstall_reason = (
self.read_uninstall_module_list(start_version, database_name)
)
if os.path.exists(uninstall_module_list_file):
with open(uninstall_module_list_file, "r") as f:
lst_module_to_uninstall = [
a.strip() for a in f.readline().split()
]
if lst_uninstall_reason:
# Show WHY each module goes away: which modules must be dropped
# depends on the database, and a removal without a stated reason
# is a decision nobody can review later.
print("✨ Modules to uninstall before migration :")
for name, reason, origin in lst_uninstall_reason:
print(
f" - {name}"
+ (
f" — {reason}"
if reason
else " — ⚠️ no reason given"
)
+ f" [{origin}]"
)
if config_state_1_uninstall_module:
lst_module_to_uninstall = (
@ -882,7 +895,10 @@ class TodoUpgrade:
print(f"🔷 {msg}")
self.add_comment_progression(msg)
if not self.dct_progression.get("state_2_update_all") and not already_update_state_1:
if (
not self.dct_progression.get("state_2_update_all")
and not already_update_state_1
):
status, cmd_executed = self.todo_upgrade_execute(
f"./script/addons/update_addons_all.sh {database_name}",
single_source_odoo=True,
@ -891,6 +907,20 @@ class TodoUpgrade:
self.dct_progression["state_2_update_all"] = True
self.write_config()
# Predict the website COW views that the NEXT version bump will break.
# A copy-on-write view freezes the structure of the module view it was
# copied from; when that module view changes mode between two versions,
# the copy keeps an arch written for the old mode and the upgrade dies
# on « Element ... cannot be located in parent view ». Reporting it here
# -- before hours of migration -- leaves time to arbitrate.
# Only the next bump can be predicted: the modes in database describe
# the current version.
self.todo_upgrade_execute(
f"{PYTHON_BIN} ./script/odoo/migration/check_cow_views.py"
f" -d {database_name} -t odoo{start_version + 1}.0",
wait_at_error=False,
)
msg = "3 - Clean up database before data migration"
print(f"🔷 {msg}")
self.add_comment_progression(msg)
@ -1029,7 +1059,20 @@ class TodoUpgrade:
# Duplicate database
cmd_clone_database = f"./odoo_bin.sh db --clone --from_database {last_database_name} --database {database_name_upgrade}"
self.todo_upgrade_execute(cmd_clone_database)
status, cmd_executed = self.todo_upgrade_execute(
cmd_clone_database
)
# Everything downstream runs against this clone: if it failed,
# do not mark it done (a rerun would skip the clone and migrate
# a missing or truncated database).
if status:
print(
f"❌ -> Clone to Odoo{next_version} FAILED (status"
f" {status}). Stopping: '{database_name_upgrade}' is"
" not usable."
)
return
lst_clone_odoo[index] = True
self.dct_progression["state_4_clone_odoo_lst"] = lst_clone_odoo
@ -1508,11 +1551,22 @@ class TodoUpgrade:
f"fix_migration_odoo{(next_version-1)*10}_to_odoo{next_version*10}.py",
)
if os.path.exists(file_path_fix_migration):
self.todo_upgrade_execute(
status, cmd_executed = self.todo_upgrade_execute(
f"cat ./{file_path_fix_migration} | ./odoo{next_version}.0/odoo/odoo-bin shell -d {database_name_upgrade}",
single_source_odoo=True,
)
# A fix that did not run must not be recorded as applied,
# otherwise the rerun skips it and OpenUpgrade hits the very
# problem the fix exists to prevent.
if status:
print(
f"❌ -> Fix migration Odoo{next_version} FAILED"
f" (status {status}):"
f" {file_path_fix_migration}"
)
return
lst_fix_migration_odoo[index] = file_path_fix_migration
self.dct_progression["state_4_fix_migration_odoo_lst"] = (
lst_fix_migration_odoo
@ -1571,13 +1625,49 @@ class TodoUpgrade:
cmd_upgrade = f"./run.sh --upgrade-path=./odoo{next_version}.0/OCA_OpenUpgrade/openupgrade_scripts/scripts --update all -c config.conf --stop-after-init --no-http --load=base,web,openupgrade_framework -d {database_name_upgrade}"
lst_upgrade_odoo[index] = cmd_upgrade
self.todo_upgrade_execute(
# Record the website COW views before the data migration. The
# upgrade silently deletes and recreates copies (measured on
# 12->13: 16 copies dropped, 13 created, children re-parented),
# and rewrites the arch of many others. Without a before/after
# record, "the site looks wrong" cannot be investigated.
self.snapshot_cow_views(
database_name_upgrade, f"before_{next_version}"
)
# Take the copies that cannot survive this bump out of the way,
# otherwise the data migration dies on them. Offered, not
# forced: their arch is a real customization.
self.neutralize_cow_views(database_name_upgrade, next_version)
status, cmd_executed = self.todo_upgrade_execute(
cmd_upgrade,
new_env={
"OPENUPGRADE_TARGET_VERSION": f"{next_version}.0"
},
)
# TODO detect error
# This is THE data migration. Recording it as done when it
# failed used to send the loop to the next version on top of a
# half-migrated database. Stop here instead: the state stays
# unset, so a rerun replays this version.
if status:
print(
f"❌ -> Database migration to Odoo{next_version} FAILED"
f" (status {status}). Stopping before version"
f" {next_version + 1} to avoid migrating a broken"
" database. Fix the cause, then relaunch: this version"
" will be replayed."
)
return
self.snapshot_cow_views(
database_name_upgrade, f"after_{next_version}"
)
self.diff_cow_views(
database_name_upgrade,
f"before_{next_version}",
f"after_{next_version}",
)
self.dct_progression["state_4_upgrade_odoo_lst"] = (
lst_upgrade_odoo
@ -1764,6 +1854,123 @@ class TodoUpgrade:
}
return dct_module
def snapshot_cow_views(self, database_name, label):
"""Record the website COW views of a database under private/.
Never blocks the migration: a snapshot is forensic material, its
absence must not stop an upgrade.
"""
self.todo_upgrade_execute(
f"{PYTHON_BIN} ./script/odoo/migration/snapshot_cow_views.py"
f" -d {database_name} -l {label}",
wait_at_error=False,
)
def neutralize_cow_views(self, database_name, next_version):
"""Offer to neutralize the COW views that would break this bump.
Renaming their key unpairs them from the module view, so the upgrade
stops choking on them. Nothing is deleted and the operation is
reversible (neutralize_cow_views.py --restore), but the choice belongs
to the user: those copies carry real customizations.
"""
cmd = (
f"{PYTHON_BIN} ./script/odoo/migration/neutralize_cow_views.py"
f" -d {database_name} -t odoo{next_version}.0"
)
status, cmd_executed, output = self.todo_upgrade_execute(
cmd, get_output=True, wait_at_error=False
)
if "No website COW view to neutralize" in "\n".join(output or []):
return
answer = (
input(
"💬 Neutralize these copies so the upgrade can proceed?"
" Their arch is kept and the change is reversible."
" (Y/n) : "
)
.strip()
.lower()
)
if answer == "n":
print(
"⚠️ -> Skipped. The data migration will very likely stop on"
" these views."
)
return
self.todo_upgrade_execute(f"{cmd} --apply", wait_at_error=False)
def diff_cow_views(self, database_name, label_before, label_after):
"""Print what the version bump did to the website COW views."""
directory = os.path.join(
PATH_MIGRATION_PRIVATE, database_name, "cow_snapshots"
)
path_before = os.path.join(directory, f"{label_before}.json")
path_after = os.path.join(directory, f"{label_after}.json")
if not (os.path.exists(path_before) and os.path.exists(path_after)):
return
self.todo_upgrade_execute(
f"{PYTHON_BIN} ./script/odoo/migration/snapshot_cow_views.py"
f" --diff {path_before} {path_after}",
wait_at_error=False,
)
@staticmethod
def parse_module_list_file(file_path):
"""Read a module list file, return [(module, reason), ...].
Accepted syntax, one module per line with an optional justification:
queue_job # blocks 12->13, trigger queue_job_notify
mgmtsystem_hazard # not ported to 13.0
Commas and several names per line are also accepted, so a list copied
from a command line works as-is. Blank lines and full-line comments are
ignored.
The previous parser was « f.readline().split() »: it kept only the FIRST
line, so a multi-line list was silently truncated, and a comma-separated
list collapsed into one bogus module name.
"""
lst_module = []
with open(file_path, "r", encoding="utf-8") as f:
for line in f:
content, _, reason = line.partition("#")
for module_name in content.replace(",", " ").split():
lst_module.append((module_name, reason.strip()))
return lst_module
def read_uninstall_module_list(self, start_version, database_name):
"""Modules to uninstall before migrating start_version -> next.
Reads the per-database private list first, then the shared versioned
defaults; duplicates are dropped, keeping the first occurrence.
Returns (lst_module, lst_detail) where lst_detail carries
(module, reason, origin_file) so the caller can justify each removal.
"""
file_name = (
f"uninstall_module_list_odoo{start_version * 10}"
f"_to_odoo{(start_version + 1) * 10}.txt"
)
lst_path = [
os.path.join(PATH_MIGRATION_PRIVATE, database_name, file_name),
os.path.join(PATH_MIGRATION_GLOBAL, file_name),
]
lst_module = []
lst_detail = []
for file_path in lst_path:
if not os.path.exists(file_path):
continue
for module_name, reason in self.parse_module_list_file(file_path):
if module_name in lst_module:
continue
lst_module.append(module_name)
lst_detail.append((module_name, reason, file_path))
return lst_module, lst_detail
def uninstall_from_database(
self, lst_module_to_uninstall, database_name, actual_version
):
@ -1920,7 +2127,10 @@ class TodoUpgrade:
self.lst_command_executed.append(cmd_executed)
self.dct_progression["command_executed"] = self.lst_command_executed
self.write_config()
if status and wait_at_error:
# None means « the command never reported a status » -> treat it as a
# failure, never as a success (defence in depth: exec_command_live now
# always sets one, but a silent None must not skip this prompt).
if (status is None or status) and wait_at_error:
print("[1] to redo the command")
wait_status = (
input(