We publish three sets of VM images on download.freebsd.org:
* Generic VM images
* BASIC-CLOUDINIT VM images
* BASIC-CI VM images
Of these, the first had names like
FreeBSD-14.2-RELEASE-amd64-ufs.raw.xz,
the second had names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CLOUDINIT.ufs.qcow2.xz,
and the third had names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CI.ufs.raw.xz
but were listed in the CHECKSUM files with names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CI-ufs.raw.xz
Standardize these to consistently use a hyphen before the filesystem
type rather than a period.
Note: On FreeBSD 14 we ship images without the -${FS} component in
their names; these are hardlinks to the -ufs images for compatibility
purposes. On FreeBSD 13 we only have UFS images and don't include the
filesystem in the name.
MFC after: 1 week
Reported by: jmg
Sponsored by: Amazon
(cherry picked from commit 35623ccc375846648f4dcdfc47c7e84fc9db4852)
On some platforms (e.g. powerpc) we don't have packages, so we can't
install them onto the ISOs. Proceed with building the images anyway.
Reported by: Weekly snapshot builds
Fixes: 7e2996c1f5b4 ("release: install wireless firmware onto disc1 and dvd")
MFC after: 1 minute
(cherry picked from commit e8263ace39c8ecf11233c0a10d0b1839e6813046)
For "release" builds (as opposed to "snapshot" builds -- in this
context BETAs and RCs are "releases") ${SNAP_SUFFIX} is empty; but it
stuck into some ociimages filenames via a copy-and-paste error.
The final filenames on the download mirrors were not affected, so
this does not need to be merged to releng/14.2.
MFC after: 3 days
Sponsored by: Amazon
(cherry picked from commit d54fdd16636bbe6a38a5b0a636dd83cf64aa6e06)
Wireless driver firmware is no longer added to the src tree.
In order to have wireless support in the installer for the new drivers
we install the firmware packages onto disc1 (and memstick) and dvd
if built on FreeBSD and NOPKG is not defined (to not break cross-builds
from Linux or OSX and to allow people to opt-out).
Sponsored by: The FreeBSD Foundation
Submitted by: cperciva (the orig. commands and where to place them)
Reviewed by: jrtc27
Differential Revision: https://reviews.freebsd.org/D47407
(cherry picked from commit 7e2996c1f5b4e684cae40c2418b68061df9997d9)
These are installed into /ociimages/ and stage onto the download site
at /OCI-IMAGES/; they don't belong in the same directory as the
"distribution" .txz files.
Fixes: d03c82c28da8 ("release: add optional OCI images")
MFC after: 1 minute
Sponsored by: Amazon
(cherry picked from commit 976e1d36be688166e140cbaded01d433c61e6d47)
We use a *.txz glob to get all of the "distributions" which comprise
the FreeBSD release, but we now (optionally) also build container
images which are .txz files. Grep those out from the distribution
lists.
A better long-term fix would probably be to generate an explicit list
of the .txz files we want rather than using an overbroad glob and
filtering out the files we *don't* want.
Fixes: d03c82c28da8 ("release: add optional OCI images")
MFC after: 1 minute
Sponsored by: Amazon
(cherry picked from commit 47866cdcc7278faa27250dcea16374231c676050)
Add the flavored port for all the
supported wireless drivers we recently added support for.
Sponsored by: The FreeBSD Foundation
Pointed out by: cperciva (as part of the set of changes)
Reviewed by: cperciva (#releng)
Differential Revision: https://reviews.freebsd.org/D47406
(cherry picked from commit 2483a2d36803896d27861c03e1f53dd9546b1e73)
If WITH_OCIIMAGES is set, "make ftp-stage" will now publish those bits
into a directory under /OCI-IMAGES/, similar to how we publish CI and
VM images.
MFC after: 1 minute
(cherry picked from commit e95e3e6eb7ef723aa5dd08de48a9ad3fec647c7d)
This adds three OCI archive format files to the release containing
FreeBSD base images suitable for static linked, dynamic linked and shell
workloads. The shell image also contains pkg-bootstrap and can be easily
extended by installing packages (including pkgbase packages).
Reviewed by: dch, cpersiva, jlduran, zlei
Differential Revision: https://reviews.freebsd.org/D46759
MFC after: 2 days
(cherry picked from commit d03c82c28da86e0812b98b051d24ae5980804ad7)
In 2016, commit 8834318685 reworked the code for excluding -dbg
files from install media, and in the process accidentally broke it
for dvd images ('grep -v' should have been 'grep -vE').
FreeBSD Update builds later began to depend on this, and in any case
since DVD images are intended as "include everything" images there's
no point excluding those; so remove the (broken) filtering from DVD
image builds.
Sponsored by: Amazon
MFC after: 2 days
(cherry picked from commit cdfd0600dc8882f0a0d0e6d9a1cdcf926edba6d6)
AWS Graviton [1234] systems have a bug in their ACPI where they mark
the PL061's GPIO pins as needing to be configured in PullUp mode (in
fact the PL061 has no pullup/pulldown resistors); this flag needs to
be removed in order for _AEI objects to be handled on these systems.
Reviewed by: Ali Saidi
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D47239
(cherry picked from commit 2f3f867ac6dd7ff3769366b828b79c44b38828e1)
While OK for 14.x, both 13.x and 15.0 are failing in snapshot
builds.
Reported by: cperciva
Sponsored by: SkunkWerks, GmbH
Reviewed by: emaste
Approved by: cperciva
Differential Revision: https://reviews.freebsd.org/D47285
(cherry picked from commit 89311e6f987ebb1a968eee6fe594b11bfb91977a)
- use raw image disk type and enable zfs, this yields smaller
images for upload after using native qcow2 + zstd compression
Reviewed by: lwhsu, emaste
Differential Revision: https://reviews.freebsd.org/D47055
MFC after: 3 days
Approved by: emaste
(cherry picked from commit c611041660845e8ee8bf3009b32327d1f783a47b)
This allows future releng tooling to use OCI for the industry
standard Open Container Initiative tooling, reducing potential
for confusion.
Approved by: cperciva
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D46975
MFC after: 3 days
Sponsored by: SkunkWerks, GmbH
(cherry picked from commit 1be84d745bcc11b766ac1fa03afd2c8ea8771737)
- Add the firstboot-freebsd-update package, as long as we do not have
pkgbase, this is needed
- Support SLAAC by default to complement DHCPv4 (use SYNCDHP instead)
Signed-off-by: Jose Luis Duran <jlduran@gmail.com>
(cherry picked from commit 120740221fd4a4577e63e6c279f9873cabe449d0)
These are the same as the standard "base" images except:
* They don't have kernel or world debug symbols,
* They don't have FreeBSD tests,
* They don't have 32-bit libraries,
* They don't have LLDB,
* They don't have the Amazon SSM Agent pre-installed,
* They don't default to installing the awscli at first boot.
This reduces the amount of disk space in use when the EC2 instance
finishes booting from ~5 GB to ~1 GB.
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46509
(cherry picked from commit 647299caa06e38622dc05a4358f9407b62a9bdda)
Move it from VM_EXTRA_PACKAGES in ec2.conf to VM_EXTRA_PACKAGES in
ec2-{base,cloud-init}.conf
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46508
(cherry picked from commit 40ff0753abb9c00b5f1e5df9ea00c9fdded55ac4)
Having the "base" FreeBSD network configuration (aka. what is used
when not using cloud-init) in ec2.conf will allow us to reuse it in
other AMIs.
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46507
(cherry picked from commit f961ddb28d6909d4c67e3e0b6b60498bbcbf64cb)
This makes it possible for a VM build configuration file to pass
options to make installworld/installkernel/distribution, e.g.
WITHOUT_DEBUG_FILES=YES in order to produce smaller images.
Note that these options are only applied at install time, not at
build time (since the same build is installed into many different
VM images), so not all src.conf options are usable here.
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46506
(cherry picked from commit 81d3df02bcf5ed6e41a91fd9fbc3b81cfe809ff3)
Allow the cloudware *_FLAVOURS and *_FSLIST values to be overridden
at the command line, to assist users who want to e.g. build only one
of the many EC2 AMIs available.
(cherry picked from commit 863975b6840b2833b0f772648ba2532806ffece8)
Stop shipping a log file for etcupdate. This is a source of
non-reproducability as it uses mktemp thereby guaranteeing the output is
different each run.
Differential Revision: https://reviews.freebsd.org/D46317
(cherry picked from commit e972e408d19a58c4f2855eface487d06ef73b799)
Without this, "make clean ec2ami" won't build a new AMI.
MFC after: 3 days
Sponsored by: Amazon
(cherry picked from commit e06022e1bfc263e8b1393c7b948707a250f16f83)
8 GB root disk images make FreeBSD/EC2 boot much faster than 6 GB
root disk images.
MFC after: 2 days
Sponsored by: Amazon
(cherry picked from commit 2dac89aee3304dd6eda9b267a0ad1cc6621a7094)
The DHCP server in EC2 knows exactly which system should be using
which IP address (and in fact EC2 has source IP filtering on by
default) so there's no point ARPing an address before using it.
The preceding commits (changing the ARP wait time from 2 s to 250 ms)
and this one (eliminating the wait entirely in EC2) reduce the time
required for a newly launched FreeBSD/EC2 instance to launch by 2
seconds.
Discussed with: icattard
MFC after: 10 days
Sponsored by: Amazon
(cherry picked from commit 54a543d5ea3a58aee2f001498376127efea24bd2)
[MFC note: This is not needed on stable/14 right now since the riscv64
images still fit within the default 5 GB size, but I want it in the
tree in case I need to bump VM sizes on a per-arch basis later.]
The default VM size may depend on the architecture. In particular,
it is currently larged on riscv64 due to a toolchain issue which
results in bloated binaries.
MFC after: 3 days
Fixes: 59c21ed6e811 "release: Bump default VM size for riscv64 to 6 GB"
Sponsored by: Amazon
(cherry picked from commit ed807f7bcaecbf32b0b81a7d18cee489a3666157)
If WITH_CLOUDWARE is not set, we didn't build BASIC-CLOUDWARE; don't try
to install it into the /VM-IMAGES/ directory as part of ftp-stage.
Fixes: 4771c2e9d1c7 "release: Publish non-uploaded cloudware bits"
MFC after: 6 days
(cherry picked from commit 836f00ebb96ac343e0f8a68ea264d2d330198488)
While most "cloudware" (AWS, Azure, GCP, Vagrant) images get uploaded
to their respective clouds, we have two images -- BASIC-CI and
BASIC-CLOUDINIT -- which are published via the FTP mirrors. We have
been handling these using a manual and error-prone process of copying
them from the internal build directories into the FTP staging tree,
but they should be handled using the "make install ftp-stage"
mechanism which is used for all the other published images.
This commit 1. Adds a new 'make cloud-install' target which is run as
part of 'make install' and installs BASIC-CI and BASIC-CLOUDINIT images,
and 2. Extends 'make ftp-stage' in Makefile.mirrors to copy those bits
into the FTP staging area.
As a side benefit, this will also make BASIC-CI and BASIC-CLOUDINIT
images available for arm64; until now they have only been available
for amd64.
MFC after: 1 week
(cherry picked from commit 4771c2e9d1c7db949a82dfe4f2b9878bb358a50e)
Remove archivers/unzip (now in base) and emulators/linux_base-c7 (old
and unlikely to be useful without other linux packages being installed),
ports-mgmt/portmaster (now largely obsolete and discouraged in favour
of using pkg and binary packages) and x11-drivers/xf86-video-vmware
(questionably useful).
Replace devel/git with devel/git@lite (sufficient for most purposes),
and adjust the "ensure the ports exist to sanitize the list" code to
ignore the @lite part when checking that /usr/ports/devel/git exists.
Add sysutils/seatd and x11-wm/sway for wayland support.
MFC after: 1 minute
Differential Revision: https://reviews.freebsd.org/D45278
(cherry picked from commit d31ed58b9f9360cecd9acfbd12999190edca6e56)
For historical reasons, the vm-release and cloudware-release targets
were written as recursive makes of ${VMTARGETS} and ${CLOUDTARGETS};
this worked fine until we started running `make release -jN` and
had both VM and cloud targets depending on ${QEMUTGT}, at which
point things exploded when we tried to build the port twice at the
same time.
Switch vm-release and cloudware-release to have make dependencies on
their individual components; this way a single make process runs and
is aware of the duplicate dependency (and only runs it once).
MFC after: 5 days
(cherry picked from commit ce7756fdca1f0a89a74bc406cd3d3ac1d1be1ffb)
The vm_extra_pre_umount function in vmimage.subr served two purposes:
It removed /etc/resolv.conf and /qemu (if cross-building), and it
provided a function for cloudware to override in order to make cloud
specific changes to the filesystem before constructing a disk image.
This resulted in a number of bugs:
1. When cross-building, the emulator binary was left as /qemu in the
Azure, GCE, Openstack and Vagrant images.
2. The build host's resolv.conf was left as /etc/resolv.conf in the
basic-ci and basic-cloudinit images.
3. When building GCE images, a Google-specific resolv.conf file was
constructed, and then deleted before the disk image was created.
Move the bits needed for running code inside a VM staging directory
from vm_install_base into a new vm_emulation_setup routine, and move
the corresponding cleanup bits from vm_extra_pre_umount to a new
vm_emulation_cleanup routine.
Remove the /qemu and /etc/resolv.conf cleanups from the cloudware
configuration files (where they exist) since we will now be running
vm_emulation_cleanup to remove those even when vm_extra_pre_umount
has been overridden.
Override vm_emulation_cleanup in gce.conf since in that one case (and
*only* that one case) we don't want to clean up resolv.conf (since it
was constructed for the VM image rather than copied from the host).
releng/14.1 candidate.
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva
(cherry picked from commit 0b1c5628c74a37e2aa2aa3769c606d3e826302f8)
For a bit over 5 years, we have used qemu when cross-building cloudware
images; in particular, it's necessary when installing packages which
might include post-install scripts.
Use qemu in the vm-images target too; while "generic" vm images don't
install packages, they still run newaliases and /etc/rc.d/ldconfig,
both of which fail without appropriate emulation.
MFC after: 1 week
(cherry picked from commit 9287d1853c00267a0118401530debf9c0554ff28)
When the VM image building code was updated to support building
non-UFS images, the vm-images-stage target was not updated to
install those newly built images to the FTP site. As a result, we
have been sending weekly snapshot announcements since August claiming
that ZFS VM images are available when they are not in fact present
anywhere publicly accessible.
Fixes: 32ae9a6b39 "release: Build UFS and ZFS VM images"
Reported by: Michael Dexter
MFC after: 5 days
(cherry picked from commit f4b08097d8e274e1a8526d864c31462ea42d9e9f)
This missing dependency is only visible with -jN (which was otherwise
broken until recently) and only when building DVD images (which aren't
included in weekly snapshots and thereby slipped past testing).
MFC after: 3 days
(cherry picked from commit 567cbd8ff1c4edc1f2015ec4f93608aaf5a3c546)
With multiple filesystems and disk image formats, we now use different
staging directories; we need to include them all in CLEANDIRS.
Fixes: 97bd53ef4d20b "Makefile.vm: Fix duplicate rc.conf files"
MFC after: 3 days
(cherry picked from commit 7ffa1f14e3c19f0061ec9f38bb74c564a49ffc17)
- Point --rootdir at the installed destdir in the dvd tree. This
causes pkg to determine the ABI from the installed destdir instead
of the host's binaries. Previously the result was that packages
for the host's ABI were always downloaded breaking cross-releases
(e.g. arm64 releases built on an amd64 host included amd64
packages on the DVD ISO image rather than arm64 packages). This
also handles version mismatches, and I tested this by cross-building
a 15.x arm64 release on a 14.x amd64 host.
- As a result, pkg now does a chdir(3) to the rootdir before running,
so the -o argument to fetch needs to be updated to be relative to
rootdir instead of the CWD as make runs.
- Add a new ROOTDIR variable to limit references to "dvd" to one
place. Ideally ROOTDIR would be an argument to this script so
that it didn't really know about the dvd layout at all.
- While here, simplify creation of symlinks by just using a longer
path to the link name instead of using 'cd' in the shell before
invoking ln(1). Also use ln -sf to create the pkg.pkg symlink
rather than rm + ln.
PR: 278273
Reported by: gatekeeper <tiago.gasiba@gmail.com>
Reviewed by: imp, delphij
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D44749
(cherry picked from commit ea2663040d524efd84b12edcbe83884437fdceea)
Make basic-cloudinit available both in qcow2 and raw formats
MFC After: 3 days
Co-Authored-By: bapt
Sponsored by: OVHCloud (original patch from bapt)
Differential Revision: https://reviews.freebsd.org/D44747
(cherry picked from commit 5c213bf01a5dca7c4030efd2ad9c9f84852e7b32)
Confusingly, uuencode writes to stdout by default, but uudecode does not.
Fixes: 6c3e01bf8578 "release: Randomize powerpc boot block file name"
MFC after: 3 days
(cherry picked from commit 4d7174b1d91ff3651ef78987d93989a40a4facd2)
With parallel builds enabled, we can end up building multiple ISOs at
once, which causes problems if mkisoimages.sh uses a consistent file
name "/tmp/hfs-boot-block".
MFC after: 3 days
(cherry picked from commit 6c3e01bf8578de54abc019832d799de80da7378c)
Set WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles when building tools
needed for uploading cloudware images.
While I'm here, adjust the bsdec2-image-upload build target to match
the style used by the Azure/GCE/Vagrant Makefiles.
MFC after: 3 days
(cherry picked from commit 6c87aed39c3053cd338c1ec18fba5b1d773beca9)
Build qemu (if needed) with WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles
so that we can have a read-only /usr/ports and don't contaminate it. This
became an issue when I enabled parallel release building, since one image
might be creating its ports.txz file at the same time as we're building
qemu as a prerequisite for building another image.
MFC after: 5 days
(cherry picked from commit 6f454c211b295ad13e80fc5b858efc9bad33e33b)
With the latest changes to release/Makefile, it is now possible to
run `make release -jN` without the build failing (at least in my
latest tests).
This reverts commit 7b707e797b2cd6265ba8f6215e59445e9efb9e97.
MFC after: 1 week
(cherry picked from commit 2c01e189ff190cf1618dedd01f196bca68298c44)
The disc1 and bootonly directories have files distributed into them
for use in "full" and "mini" images; the former are disc1.iso and
memstick.img, and the latter is bootonly.iso and mini-memstick.img.
Unfortunately the scripts which package a directory tree into an ISO
or memory stick image also modify the directory, for example to
create an appropriate /etc/fstab file; so creating two images at the
same time breaks.
Resolve this by copying disc1 to disc1-disc1 and disc1-memstick,
and copying bootonly to bootonly-bootonly and bootonly-memstick,
before using those directories for constructing the ISO+memstick
images.
MFC after: 1 week
(cherry picked from commit a1bff97300abe4b46528357e39c83da764df1fd8)
