- use raw image disk type and enable zfs, this yields smaller
images for upload after using native qcow2 + zstd compression
Reviewed by: lwhsu, emaste
Differential Revision: https://reviews.freebsd.org/D47055
MFC after: 3 days
Approved by: emaste
(cherry picked from commit c611041660)
This allows future releng tooling to use OCI for the industry
standard Open Container Initiative tooling, reducing potential
for confusion.
Approved by: cperciva
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D46975
MFC after: 3 days
Sponsored by: SkunkWerks, GmbH
(cherry picked from commit 1be84d745b)
These are the same as the standard "base" images except:
* They don't have kernel or world debug symbols,
* They don't have FreeBSD tests,
* They don't have 32-bit libraries,
* They don't have LLDB,
* They don't have the Amazon SSM Agent pre-installed,
* They don't default to installing the awscli at first boot.
This reduces the amount of disk space in use when the EC2 instance
finishes booting from ~5 GB to ~1 GB.
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46509
(cherry picked from commit 647299caa0)
Allow the cloudware *_FLAVOURS and *_FSLIST values to be overridden
at the command line, to assist users who want to e.g. build only one
of the many EC2 AMIs available.
(cherry picked from commit 863975b684)
While most "cloudware" (AWS, Azure, GCP, Vagrant) images get uploaded
to their respective clouds, we have two images -- BASIC-CI and
BASIC-CLOUDINIT -- which are published via the FTP mirrors. We have
been handling these using a manual and error-prone process of copying
them from the internal build directories into the FTP staging tree,
but they should be handled using the "make install ftp-stage"
mechanism which is used for all the other published images.
This commit 1. Adds a new 'make cloud-install' target which is run as
part of 'make install' and installs BASIC-CI and BASIC-CLOUDINIT images,
and 2. Extends 'make ftp-stage' in Makefile.mirrors to copy those bits
into the FTP staging area.
As a side benefit, this will also make BASIC-CI and BASIC-CLOUDINIT
images available for arm64; until now they have only been available
for amd64.
MFC after: 1 week
(cherry picked from commit 4771c2e9d1)
For historical reasons, the vm-release and cloudware-release targets
were written as recursive makes of ${VMTARGETS} and ${CLOUDTARGETS};
this worked fine until we started running `make release -jN` and
had both VM and cloud targets depending on ${QEMUTGT}, at which
point things exploded when we tried to build the port twice at the
same time.
Switch vm-release and cloudware-release to have make dependencies on
their individual components; this way a single make process runs and
is aware of the duplicate dependency (and only runs it once).
MFC after: 5 days
(cherry picked from commit ce7756fdca)
For a bit over 5 years, we have used qemu when cross-building cloudware
images; in particular, it's necessary when installing packages which
might include post-install scripts.
Use qemu in the vm-images target too; while "generic" vm images don't
install packages, they still run newaliases and /etc/rc.d/ldconfig,
both of which fail without appropriate emulation.
MFC after: 1 week
(cherry picked from commit 9287d1853c)
With multiple filesystems and disk image formats, we now use different
staging directories; we need to include them all in CLEANDIRS.
Fixes: 97bd53ef4d "Makefile.vm: Fix duplicate rc.conf files"
MFC after: 3 days
(cherry picked from commit 7ffa1f14e3)
Make basic-cloudinit available both in qcow2 and raw formats
MFC After: 3 days
Co-Authored-By: bapt
Sponsored by: OVHCloud (original patch from bapt)
Differential Revision: https://reviews.freebsd.org/D44747
(cherry picked from commit 5c213bf01a)
Build qemu (if needed) with WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles
so that we can have a read-only /usr/ports and don't contaminate it. This
became an issue when I enabled parallel release building, since one image
might be creating its ports.txz file at the same time as we're building
qemu as a prerequisite for building another image.
MFC after: 5 days
(cherry picked from commit 6f454c211b)
Provide both zfs and ufs images which a 1MB partition reserved for the
config drive wearing a GPT Label "config-drive" to allow consumer to
know where they should push the config drive on the provided image.
This has been tested on OVHCloud baremetal via "bring your own image"
Also tested on openstack
Reviewed by: emaste
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44369
(cherry picked from commit 78b6117dac)
(cherry picked from commit a977444652)
Let Azure use the image directly built by the release engineering team.
Reviewed by: emaste
Approved by: cperciva (re)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41906
(cherry picked from commit 63007e4611)
The emulator-portinstall target now unconditionally ensures that qemu
is installed; but is only invoked if needed (aka. when cross building
VM images).
MFC After: 3 days
MFC With: 97bd53ef4d ("Fix duplicate rc.conf files")
(cherry picked from commit 7d0ee5ebd0)
Two bugs in Makefile.vm resulted in disk images being "built" multiple
times, resulting in lines added to /etc/rc.conf being duplicated:
1. The vm-image target reused the same "staging tree" directory for all
of its builds (multiple disk image types and multiple filesystem types).
2. The cw-type-flavour-fs target depends on emulator-portinstall, which
did not have a 'touch ${.TARGET}' and thus re-ran every time -- and
caused the cw-type-flavour-fs target to be re-run. This was triggered
by release builds running `make cloudware-release` (creating the disk
images) followed by `make ec2amis` (which re-created the disk images
prior to uploading them).
MFC After: 1 week
Sponsored by: https://www.patreon.com/cperciva
(cherry picked from commit 97bd53ef4d)
Known issues:
1. The ec2-user user is created with a homedir of /usr/home/ec2-user
instead of /home/ec2-user; this appears to be a bug in cloud-init's
FreeBSD support.
2. Cloud-init configures IPv4 networking but not IPv6 networking.
releng/14.0 candidate.
Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Relnotes: yes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41793
(cherry picked from commit 1f4ce7a39f)
Using the recently-added "cloudware flavours" mechanism, turn the
existing EC2 AMIs into a new "base" flavour. The only user-visible
change is that AMI names now include the word "base".
releng/14.0 candidate.
Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41791
(cherry picked from commit 7494fb6b7d)
There are two "Vagrant" images right now: "Vagrant Image for VirtualBox"
and "Vagrant Image for VMWare". Rather than listing these separately in
a CLOUDWARE list, place "VAGRANT" into a CLOUDWARE_TYPES list and then
use a VAGRANT_FLAVOURS variable to identify the two versions. Add make
logic to allow defaults (in this case, image format and filesystem) to
be specified once for VAGRANT and inherited by both flavours.
This will make future work to add flavoured EC2 images simpler.
releng/14.0 candidate.
Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41790
(cherry picked from commit e6eeac9b71)
The cw*-package targets were introduced in February 2015 as part of
adding support for building GCE images; but GCE support was reworked
in June 2015 and the cw*-package targets were in fact never used.
Remove them.
The *_DISK variables were introduced in February 2015 as part of
adding the cloudware-install target; this was removed in May 2016 as
the cloudware images are published via the respective cloud systems
and not published as disk images via the FreeBSD FTP site. As such,
the *_DISK variables are not unused; remove them.
releng/14.0 candidate.
Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41789
(cherry picked from commit fcbeeb5ff4)
The pre-existing "ec2ami" target builds and uploads a single AMI
(with filesystem determined by ${VMFS}) as before; a new "ec2amis"
target does both UFS and ZFS.
Reviewed by: gjb
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41343
New ${CW}_FSLIST settings control the list of filesystem types with which
each cloudware image will be built; currently these are all set to "ufs",
i.e. no change from previous.
The cloudware images have their filesystem type as part of their file
name; for backwards compatibilty the ${VMFS} image is linked to the
previously used file name. This compatibility can be removed once all
the cloudware uploading/publishing code has been updated to use the new
image names (possibly more than one of them).
Reviewed by: gjb
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41341
A new option 'VMFSLIST' controls the list of filesystems for which VM
images will be built; the default is to build both UFS and ZFS.
The vm-install target installs these as ${OSRELEASE}-${FS}.${FORMAT},
e.g. FreeBSD-14.0-CURRENT-amd64-zfs.vmdk. For backwards compatibility,
the ${VMFS} image is linked to the previously used ${OSRELEASE}.${FORMAT}
name.
Cloudware building will be updated in a later commit.
Reviewed by: gjb
Reviewed by: emaste, markj (previous version)
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41340
This just stages the kernel and builds a stripped-down rootfs for use
with the Firecracker VMM. At some point in the future the release
engineering team might start publishing these, but initially it's
just here to simplify FreeBSD/Firecracker development and testing.
Note that the rootfs generated:
* Uses an IP address of 10.0.0.2 with a gateway of 10.0.0.1,
* Has sshd enabled,
* Has user "freebsd" with password "freebsd" and a root password
of "root", and
* Is 1 GB in size (but has growfs enabled).
All of those are subject to change without notice; anyone intending to
use FreeBSD/Firecracker in anything remotely resembling a production
environment should talk to cperciva first.
Reviewed by: gjb
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D40956
Provides an OCI (Oracle Cloud Infrastructure) release target for
Oracle's KVM-based VM implementation. Tested using 13.1-RELEASE,
primarily on Ampere CPU on A1.Flex VM shapes, but also works on
amd64 shapes.
- supports cloud-init and custom scripts
- provides a freebsd@ sudo-enabled user
- root user disabled over ssh & console
Approved by: gjb
Reviewed by: emaste
MFS after: 1 week
Sponsored by: The FreeBSD Foundation
Sponsored by: SkunkWerks, GmbH
Technical assistance from: Oracle
Differential Revision: https://reviews.freebsd.org/D34746
The change extends vmimage.subr to handle a new parameter, VMFS, which
should be equal to either "ufs" or "zfs". When it is set to ZFS, we use
makefs to create a bootable pool populated using the same dataset layout
as bsdinstall and "poudriere image" use. The pool can be grown using
the growfs rc.d script, just as in UFS images.
This will make it easy to provide VM and cloud images with ZFS as the
root filesystem. So far I did not do extensive testing of cloud images;
I merely verified that creation of ZFS-based AWS AMIs works and allows
me to create amd64 and arm64 EC2 instances with ZFS as the root
filesystem.
Reviewed by: emaste, gjb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34426
Imports the changes for building official images on Azure Marketplace,
which fulfill the requirements of Azure and FreeBSD cloud images like
disk layout and UEFI for Gen2 VM, along with some minor improvements like
configurations to speed up booting.
"CLOUDWARE" list will be updated after some more collaborations with re
completed.
Reviewed by: re (gjb)
Sponsored by: The FreeBSD Foundation
Technical assistance from: Microsoft
Differential Revision: https://reviews.freebsd.org/D23804
truncate(1) is not case-sensitive with regard to setting the size
of a file. makefs(8), however, does not honor upper-case values.
Update release-specific files and the release(7) manual page to
reflect this.
MFC with: 1ca8842f3a
Submitted by: ehem_freebsd_m5p.com (original)
Differential Review: https://reviews.freebsd.org/D28979
Sponsored by: Rubicon Communications, LLC ("Netgate")
In followup to 80ab50e1de,
export UNAME_r in Makefile.inc1 instead of Makefile.vm.
MFC after: 3 days
MFC with: 80ab50e1de
Sponsored by: Rubicon Communications, LLC ("Netgate")
A lot of projects CI can't do FreeBSD tests currently.
The main reason is that the project CI infrastructure is runned on Linux
and that our images aren't modifiable from a Linux hosts.
Add a basic image specific for this case (called BASIC-CI for a lack of a
better name).
The image have no package pre-installed.
It only have a few modification to have dhcp client runned on the default
interface and sshd started with option to be able to log on without a password
as root.
Sponsored by: The FreeBSD Foundation
Reviewed by: re (gjb@)
Differential Revision: https://reviews.freebsd.org/D25598
30GB to 3GB. The raw images can be resized using truncate(1), and
other formats can be resized with tools included with other tools
included with other hypervisors.
Enable the growfs(8) rc(8) at firstboot if the disk was resized
prior to booting the virtual machine for the first time.
Discussed with: several
PR: 232313 (requested in other context)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
If MACHINE_ARCH doesn't match TARGET_ARCH, and we're not in the special
case of building i386 images on an amd64 host, we need to pull in the
qemu-user-static package; this allows us to run some commands inside
the VM disk image chroot, most notably to install packages.
Reviewed by: gjb
MFC after: 2 weeks
Sponsored by: FreeBSD/EC2 patreon (https://www.patreon.com/cperciva)
providing more space for a local buildworld to succeed without
attaching separate disks for /usr/src and /usr/obj.
Reported by: mckusick
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
otherwise with 'CLOUDWARE=vagrant-virtualbox', the path to the
configuration file may be incorrect.
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
All operations are performed on the original output file,
and nothing is ever done with the copied file (as in, no
modifications and/or publication from the copied file).
Sponsored by: The FreeBSD Foundation
This was reported to not work as expected, and feedback
following the report was never received. In addition, we
do not publish these anywhere publicly.
The configuration file is not (yet) removed, in hopes that
the remaining details on the original issues can be resolved,
and we can publish the resulting images in the future.
Sponsored by: The FreeBSD Foundation
Microsoft has graciously overtaken publication of FreeBSD
on Azure since 10.3-RELEASE.
Many thanks to Microsoft for their support of FreeBSD on
Azure.
The configuration file and Makefile.azure are not (yet)
planned to be removed, as they may be useful in the future.
Sponsored by: The FreeBSD Foundation
This was discussed during the 10.2-RELEASE cycle, however
since we were nearing the end of the cycle, we decided to
defer this change until after 10.2-RELEASE.
Reminded by: so (delphij), jmg
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
