Add a bit_count function, which efficiently counts the number of bits set in
a bitstring.
sys/sys/bitstring.h
tests/sys/sys/bitstring_test.c
share/man/man3/bitstring.3
Add bit_alloc
sys/kern/subr_unit.c
Use bit_count instead of a naive counting loop in check_unrhdr, used
when INVARIANTS are enabled. The userland test runs about 6x faster
in a generic build, or 8.5x faster when built for Nehalem, which has
the POPCNT instruction.
sys/sys/param.h
Bump __FreeBSD_version due to the addition of bit_alloc
UPDATING
Add a note about the ABI incompatibility of the bitstring(3)
changes, as suggested by lidl.
Suggested by: gibbs
Reviewed by: gibbs, ngie
MFC after: 9 days
X-MFC-With: 299090, 300538
Relnotes: yes
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6255
It turns out we need to leave this in place for a while so that people
running self-hosting armv6hf systems can do the builds necessary to update
to armv6 (which is now hardfloat by default).
This will still build the compiler for the target but will not build the
bootstrap cross-compiler in the cross-tools phase. Other toolchain
bootstrapping, such as elftoolchan and binutils, currently still occurs.
This will utilize the default CC (cc, /usr/bin/cc) as an external compiler.
This is planned to be on-by-default eventually.
This will utilize the __FreeBSD_cc_version compiler macro defined in the
source tree and compare it to CC's version. If they match then the
cross-compiler is skipped. If [X]CC is an external compiler (absolute
path) or WITHOUT_CROSS_COMPILER is already set, then this logic is skipped.
If the expected bootstrap compiler type no longer matches the found CC
compiler type (clang vs gcc), then the logic is skipped. As an extra
safety check the version number is also compared from the compiler to
the tree version.
Clang:
The macro FREEBSD_CC_VERSION is defined in:
lib/clang/include/clang/Basic/Version.inc
For clang -target will be used if TARGET_ARCH != MACHINE_ARCH. This
is from the current external toolchain logic. There is currently an
assumption that the host compiler can build the TARGET_ARCH. This
will usually be the case since we don't conditionalize target arch
support in clang, but it will break when introducing new
architectures. This problem is mitigated by incrementing the version
when adding new architectures.
GCC:
The macro FBSD_CC_VER is defined in:
gnu/usr.bin/cc/cc_tools/freebsd-native.h
For GCC there is no simple -target support when TARGET_ARCH !=
MACHINE_ARCH. In this case the opportunistic skip is not done. If we
add proper support for this case in external toolchain logic then it
will be fine to enable.
This relies on the macros being incremented whenever any change occurs
to these compilers that warrant rebuilding files. It also should never
repeat earlier values.
Reviewed by: brooks, bapt, imp
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D6357
See r298220 for more explanation. We don't want to prevent installing
if a cookie exists for the install target.
Sponsored by: EMC / Isilon Storage Division
This is the same problem as r290629. With META_MODE we do not generate
.depend files, so there is no proper dependency to lookup. Guessed
dependencies must be used. If this proves to be a problem then we will
have to generate and use .depend files even with META_MODE.
Sponsored by: EMC / Isilon Storage Division
This is a follow-up to r299289. If the user did not run bootstrap-tools
for this directory then just build the tool as normal. It assumes that
TARGET == MACHINE, but that was already the case before r299289.
Sponsored by: EMC / Isilon Storage Division
sglist_count_vmpages() determines the number of segments required for
a buffer described by an array of VM pages. sglist_append_vmpages()
adds the segments described by such a buffer to an sglist. The latter
function is largely pulled from sglist_append_bio(), and
sglist_append_bio() now uses sglist_append_vmpages().
Reviewed by: kib
Sponsored by: Chelsio Communications
Add a pair of bus methods that can be used to "map" resources for direct
CPU access using bus_space(9). bus_map_resource() creates a mapping and
bus_unmap_resource() releases a previously created mapping. Mappings are
described by 'struct resource_map' object. Pointers to these objects can
be passed as the first argument to the bus_space wrapper API used for bus
resources.
Drivers that wish to map all of a resource using default settings
(for example, using uncacheable memory attributes) do not need to change.
However, drivers that wish to use non-default settings can now do so
without jumping through hoops.
First, an RF_UNMAPPED flag is added to request that a resource is not
implicitly mapped with the default settings when it is activated. This
permits other activation steps (such as enabling I/O or memory decoding
in a device's PCI command register) to be taken without creating a
mapping. Right now the AGP drivers don't set RF_ACTIVE to avoid using
up a large amount of KVA to map the AGP aperture on 32-bit platforms.
Once RF_UNMAPPED is supported on all platforms that support AGP this
can be changed to using RF_UNMAPPED with RF_ACTIVE instead.
Second, bus_map_resource accepts an optional structure that defines
additional settings for a given mapping.
For example, a driver can now request to map only a subset of a resource
instead of the entire range. The AGP driver could also use this to only
map the first page of the aperture (IIRC, it calls pmap_mapdev() directly
to map the first page currently). I will also eventually change the
PCI-PCI bridge driver to request mappings of the subset of the I/O window
resource on its parent side to create mappings for child devices rather
than passing child resources directly up to nexus to be mapped. This
also permits bridges that do address translation to request suitable
mappings from a resource on the "upper" side of the bus when mapping
resources on the "lower" side of the bus.
Another attribute that can be specified is an alternate memory attribute
for memory-mapped resources. This can be used to request a
Write-Combining mapping of a PCI BAR in an MI fashion. (Currently the
drivers that do this call pmap_change_attr() directly for x86 only.)
Note that this commit only adds the MI framework. Each platform needs
to add support for handling RF_UNMAPPED and thew new
bus_map/unmap_resource methods. Generally speaking, any drivers that
are calling rman_set_bustag() and rman_set_bushandle() need to be
updated.
Discussed on: arch
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D5237
The DTraceToolkit is part of the Open DTrace effort and is supported
on FreeBSD as a port (sysutils/DTraceToolkit) which has been updated
to properly track toolkit development upstream.
Sponsored by: DARPA, AFRL
intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013.
A robust mutex is guaranteed to be cleared by the system upon either
thread or process owner termination while the mutex is held. The next
mutex locker is then notified about inconsistent mutex state and can
execute (or abandon) corrective actions.
The patch mostly consists of small changes here and there, adding
neccessary checks for the inconsistent and abandoned conditions into
existing paths. Additionally, the thread exit handler was extended to
iterate over the userspace-maintained list of owned robust mutexes,
unlocking and marking as terminated each of them.
The list of owned robust mutexes cannot be maintained atomically
synchronous with the mutex lock state (it is possible in kernel, but
is too expensive). Instead, for the duration of lock or unlock
operation, the current mutex is remembered in a special slot that is
also checked by the kernel at thread termination.
Kernel must be aware about the per-thread location of the heads of
robust mutex lists and the current active mutex slot. When a thread
touches a robust mutex for the first time, a new umtx op syscall is
issued which informs about location of lists heads.
The umtx sleep queues for PP and PI mutexes are split between
non-robust and robust.
Somewhat unrelated changes in the patch:
1. Style.
2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared
pi mutexes.
3. Removal of the userspace struct pthread_mutex m_owner field.
4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls
the lifetime of the shared mutex associated with a vnode' page.
Reviewed by: jilles (previous version, supposedly the objection was fixed)
Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects)
Tested by: pho
Sponsored by: The FreeBSD Foundation
translate the pci rid to a controller ID. The translation could be based
on the 'msi-map' OFW property, a similar ACPI option, or hard-coded for
hardware lacking the above options.
Reviewed by: wma
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Add a new get_id interface to pci and pcib. This will allow us to both
detect failures, and get different PCI IDs.
For the former the interface returns an int to signal an error. The ID is
returned at a uintptr_t * argument.
For the latter there is a type argument that allows selecting the ID type.
This only specifies a single type, however a MSI type will be added
to handle the need to find the ID the hardware passes to the ARM GICv3
interrupt controller.
A follow up commit will be made to remove pci_get_rid.
Reviewed by: jhb, rstone (previous version)
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6239
We want to avoid .text relocations in shared objects. libcrypto was the
only consumer and it is now fixed (as of r299389). Remove the now-unused
support for turning off the linker warning.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D6323
detect failures, and get different PCI IDs.
For the former the interface returns an int to signal an error. The ID is
returned at a uintptr_t * argument.
For the latter there is a type argument that allows selecting the ID type.
This only specifies a single type, however a MSI type will be added
to handle the need to find the ID the hardware passes to the ARM GICv3
interrupt controller.
A follow up commit will be made to remove pci_get_rid.
Reviewed by: jhb, rstone
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6239
Kernel installs always override KMODDIR when installing modules, so
this default setting is only used for standalone module builds. Many
out-of-tree modules manually override KMODDIR already to avoid placing
modules in /boot/kernel. This now makes that behavior the default.
Discussed on: arch@
Reviewed by: imp
Relnotes: yes
the bio to a pristine state should you wish to re-use it for another
I/O without freeing it. In the bast, a simple bzero was done to do
this, but that may not be sufficient in the future when the bio may
contain state that's not part of the documented API. Besides, it makes
the code clearer as to the intent...
Noticed by: smh@
As result of this, a new examples package is now created.
Note, this is only effective with 'SHARED=copies' (the default),
as the 'SHARED=symlinks' mechanism will create a symlink to the
source tree version of the file(s).
Sponsored by: The FreeBSD Foundation
bus_get_cpus() returns a specified set of CPUs for a device. It accepts
an enum for the second parameter that indicates the type of cpuset to
request. Currently two valus are supported:
- LOCAL_CPUS (on x86 this returns all the CPUs in the package closest to
the device when DEVICE_NUMA is enabled)
- INTR_CPUS (like LOCAL_CPUS but only returns 1 SMT thread for each core)
For systems that do not support NUMA (or if it is not enabled in the kernel
config), LOCAL_CPUS fails with EINVAL. INTR_CPUS is mapped to 'all_cpus'
by default. The idea is that INTR_CPUS should always return a valid set.
Device drivers which want to use per-CPU interrupts should start using
INTR_CPUS instead of simply assigning interrupts to all available CPUs.
In the future we may wish to add tunables to control the policy of
INTR_CPUS (e.g. should it be local-only or global, should it ignore
SMT threads or not).
The x86 nexus driver exposes the internal set of interrupt CPUs from the
the x86 interrupt code via INTR_CPUS.
The ACPI bus driver and PCI bridge drivers use _PXM to return a suitable
LOCAL_CPUS set when _PXM exists and DEVICE_NUMA is enabled. They also and
the global INTR_CPUS set from the nexus driver with the per-domain set from
_PXM to generate a local INTR_CPUS set for child devices.
Compared to the r298933, this version uses 'struct _cpuset' in
<sys/bus.h> instead of 'cpuset_t' to avoid requiring <sys/param.h>
(<sys/_cpuset.h> still requires <sys/param.h> for MAXCPU even though
<sys/_bitset.h> does not after recent changes).
The slowstart_flightsize and local_slowstart_flightsize sysctl's
were removed from the TCP code in 226447 several years ago.
PR: 209376
MFC after: 1 week
This is an initial work in progress to use the replacement bhnd
bus code for devices which support it.
* Add manpage updates for bhnd, bhndb, siba
* Add kernel options for bhnd, bhndbus, etc
* Add initial support in if_bwn_pci / if_bwn_mac for using bhnd
as the bus transport for suppoted NICs
* if_bwn_pci will eventually be the PCI bus glue to interface to bwn,
which will use the right backend bus to attach to, versus direct
nexus/bhnd attachments (as found in embedded broadcom devices.)
The PCI glue defaults to probing at a lower level than the bwn glue,
so bwn should still attach as per normal without a boot time tunable set.
It's also not fully fleshed out - the bwn probe/attach code needs to be
broken out into platform and bus specific things (just like ath, ath_pci,
ath_ahb) before we can shift the driver over to using this.
Tested:
* BCM4311, STA mode
* BCM4312, STA mode
Submitted by: Landon Fuller <landonf@landonf.org>
Differential Revision: https://reviews.freebsd.org/D6191
after r298107
Summary of changes:
- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)
Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.
MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division
Two new functions are provided, bit_ffs_at() and bit_ffc_at(), which allow
for efficient searching of set or cleared bits starting from any bit offset
within the bit string.
Performance is improved by operating on longs instead of bytes and using
ffsl() for searches within a long. ffsl() is a compiler builtin in both
clang and gcc for most architectures, converting what was a brute force
while loop search into a couple of instructions.
All of the bitstring(3) API continues to be contained in the header file.
Some of the functions are large enough that perhaps they should be uninlined
and moved to a library, but that is beyond the scope of this commit.
sys/sys/bitstring.h:
Convert the majority of the existing bit string implementation from
macros to inline functions.
Properly protect the implementation from inadvertant macro expansion
when included in a user's program by prefixing all private
macros/functions and local variables with '_'.
Add bit_ffs_at() and bit_ffc_at(). Implement bit_ffs() and
bit_ffc() in terms of their "at" counterparts.
Provide a kernel implementation of bit_alloc(), making the full API
usable in the kernel.
Improve code documenation.
share/man/man3/bitstring.3:
Add pre-exisiting API bit_ffc() to the synopsis.
Document new APIs.
Document the initialization state of the bit strings
allocated/declared by bit_alloc() and bit_decl().
Correct documentation for bitstr_size(). The original code comments
indicate the size is in bytes, not "elements of bitstr_t". The new
implementation follows this lead. Only hastd assumed "elements"
rather than bytes and it has been corrected.
etc/mtree/BSD.tests.dist:
tests/sys/Makefile:
tests/sys/sys/Makefile:
tests/sys/sys/bitstring.c:
Add tests for all existing and new functionality.
include/bitstring.h
Include all headers needed by sys/bitstring.h
lib/libbluetooth/bluetooth.h:
usr.sbin/bluetooth/hccontrol/le.c:
Include bitstring.h instead of sys/bitstring.h.
sbin/hastd/activemap.c:
Correct usage of bitstr_size().
sys/dev/xen/blkback/blkback.c
Use new bit_alloc.
sys/kern/subr_unit.c:
Remove hard-coded assumption that sizeof(bitstr_t) is 1. Get rid of
unrb.busy, which caches the number of bits set in unrb.map. When
INVARIANTS are disabled, nothing needs to know that information.
callapse_unr can be adapted to use bit_ffs and bit_ffc instead.
Eliminating unrb.busy saves memory, simplifies the code, and
provides a slight speedup when INVARIANTS are disabled.
sys/net/flowtable.c:
Use the new kernel implementation of bit-alloc, instead of hacking
the old libc-dependent macro.
sys/sys/param.h
Update __FreeBSD_version to indicate availability of new API
Submitted by: gibbs, asomers
Reviewed by: gibbs, ngie
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6004
It will be used for the upcoming LRO hash table initialization.
And probably will be useful in other cases, when M_WAITOK can't
be used.
Reviewed by: jhb, kib
Sponsored by: Microsoft OSTC
Differential Revision: https://reviews.freebsd.org/D6138
bus_get_cpus() returns a specified set of CPUs for a device. It accepts
an enum for the second parameter that indicates the type of cpuset to
request. Currently two valus are supported:
- LOCAL_CPUS (on x86 this returns all the CPUs in the package closest to
the device when DEVICE_NUMA is enabled)
- INTR_CPUS (like LOCAL_CPUS but only returns 1 SMT thread for each core)
For systems that do not support NUMA (or if it is not enabled in the kernel
config), LOCAL_CPUS fails with EINVAL. INTR_CPUS is mapped to 'all_cpus'
by default. The idea is that INTR_CPUS should always return a valid set.
Device drivers which want to use per-CPU interrupts should start using
INTR_CPUS instead of simply assigning interrupts to all available CPUs.
In the future we may wish to add tunables to control the policy of
INTR_CPUS (e.g. should it be local-only or global, should it ignore
SMT threads or not).
The x86 nexus driver exposes the internal set of interrupt CPUs from the
the x86 interrupt code via INTR_CPUS.
The ACPI bus driver and PCI bridge drivers use _PXM to return a suitable
LOCAL_CPUS set when _PXM exists and DEVICE_NUMA is enabled. They also and
the global INTR_CPUS set from the nexus driver with the per-domain set from
_PXM to generate a local INTR_CPUS set for child devices.
Reviewed by: wblock (manpage)
Differential Revision: https://reviews.freebsd.org/D5519
By default set to 'YES' so it does not change the current behaviour for users,
this variable allows to decide to not extract crach dumps from the dump
device at boot time by setting it to "NO" in rc.conf.
Sponsored by: Gandi.net
Fix a related typo while here.
Note, this change results in the Kyuafile inclusion in the runtime
package, which needs to be fixed, however addresses the PR as far
as I can tell in my tests.
PR: 209114
Submitted by: ngie
Sponsored by: The FreeBSD Foundation
Specifically, mention that rman_get_bustag/handle/virtual are valid after
a resource is activated. Also, mention the wrapper API that accepts a
struct resource instead of a bus tag and handle.
The BUS_RESCAN() method rescans a single bus device checking for devices
that have been added or removed from the bus. A new 'rescan' command is
added to devctl(8) to trigger a rescan.
Differential Revision: https://reviews.freebsd.org/D6016
This is a minor follow-up to r297422, prompted by a Coverity warning. (It's
not a real defect, just a code smell.) OSD slot array reservations are an
array of pointers (void **) but were cast to void* and back unnecessarily.
Keep the correct type from reservation to use.
osd.9 is updated to match, along with a few trivial igor fixes.
Reported by: Coverity
CID: 1353811
Sponsored by: EMC / Isilon Storage Division
The mfi(4) manpage doesn't explain the modules yet, but at least we direct
users to the right place.
PR: 205925
Submitted by: dvl
MFC after: 1 week
Sponsored by: Essen Hackathon 2016
ufs/ffs/fs.h and not 10%. The newfs(8) and tunefs(8)
man pages had this change already, but fs(5) did not.
This change makes it consistent again.
Bump Dd.
PR: 204929
Submitted by: amutu@amutu.com
MFC after: 3 days
Sponsored by: Essen Linuxhotel Hackathon 2016
Otherwise the build command changes every build. META_MODE will
only remove it if something changes to warrant rebuilding
the file.
Sponsored by: EMC / Isilon Storage Division
dirdeps.mk: move logic to handle -f dirdeps.mk to inside check
for first read of dirdeps.mk
Also fix handling of WITHOUT_DIRDEPS_BELOW
gendirdeps.mk: pass M2D_EXCLUDES to meta2deps
meta.autodep.mk: if we build with nofilemon, leave a cookie to
prevent updating dependencies until cleaned.
Reviewed by: bdrewery
This is limited to src-tree builds, meaning not extended to ports or other
out-of-tree builds.
This will help ensure that read-only OBJDIRS will be respected at install-time
by causing a more consistent failure for those who don't use a read-only
OBJDIR. It also will cause Jenkins to yell. This is a better solution than
trying to see CC=false as has been attempted and discussed before.
Of course this is only relevant for files generated by CC.
Disable this for META_MODE since it will detect the CFLAGS/command
change and force a rebuild.
Sponsored by: EMC / Isilon Storage Division
Prefix with dashes (unordered list) and put one variable on each
line (to avoid future conflicts)
Done via the following one-liner:
> sh -c 'for i in $(make -C tests/sys/aio PROG=foo -VPROG_VARS:O); do printf "\t\t- $i\n"; done'
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
This will allow the variables [*] to be overridden on a per-PROG basis,
which is useful when controlling "stripping" behavior for some tests
that require debug symbols or to be unstripped
DEBUG_FLAGS (similar to CFLAGS) supports appending, whereas STRIP is
an override
*: Due to how STRIP is defined in bsd.own.mk (in addition to
bsd.lib.mk and bsd.prog.mk), and the fact that bsd.test.mk pulls in
bsd.own.mk first, overriding STRIP doesn't work today.
A follow up commit is pending to "rectify" this after additional
testing is done.
Discussed with: bdrewery
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
When using meta mode with filemon, the build is reliably incremental
safe. Bmake will use the meta files, along with filemon information,
to rebuild targets when their dependencies change, commands change,
or files they generate are missing.
Sponsored by: EMC / Isilon Storage Division
This avoids 'build command changed' due to CFLAGS/CC changes during the
normal build. Without this the build-tools targets end up rebuilding
for the *target* rather than keeping the native versions built in
build-tools.
Sponsored by: EMC / Isilon Storage Division
This happened to work for not prepending .OBJDIR twice but broke the
other case of prepending it when needed.
Pointyhat to: bdrewery
Sponsored by: EMC / Isilon Storage Division
This will only be done if the target is defined, so if the target is
defined after bsd.sys.mk is included then it needs to manually add
${META_DEPS} still.
Sponsored by: EMC / Isilon Storage Division
MAKEOBJDIRPREFIX is set to blank and exported from MAKELEVEL0 along
with OBJROOT exported. In sub-makes OBJROOT is recalculated with
an empty MAKEOBJDIRPREFIX though.
Sponsored by: EMC / Isilon Storage Division
This corrects an error in r296947 in that it is not possible to assert
which thread holds a shared (or read) lock, but it is possible to assert
that one is held. Just not very useful.
MFC after: 1 week
Submitted by: wblock, jhb
Reviewed by: kib (earlier version), jhb, wblock
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5659
sounds like some kind of horrific theme park. "Hey kids, want to go to
User Land?" "No! We'll be good!"
The obvious replacement is "userland", a compound word replete with
term-of-art meaning and just a hint of cautionary tale. The alternate
terms "flugelhorn" and "bullfrog", while also good, are less well-known
and were voted down in committee.
MFC after: 1 week
This overrides the cross-compiler's default sysroot to use the WORLD32's
sysroot for building the lib32 libraries. Previously the cross-compiler
would default the sysroot to the 64bit WORLDTMP and -B/-L/-isystem flags
were used to build using the lib32 files. This leads to multiple issues
discussed later. Some extra headers are now needed to be staged since the
64bit WORLDTMP is not referenced at all for headers. The 64bit WORLDTMP
is still used via PATH for build tools. Overriding the default
target/arch is retained in the CC/CXX overrides.
This allows reverting the LDSCRIPT rewriting in installworld from r296921 and
r235122, thus allowing read-only objdirs to work for installing again.
This removes the need for _LDSCRIPTROOT.
This allows progressing the change to always use --sysroot for the build
rather than only relying on the cross-compiler's default sysroot. The
work for that is in D3970 and needed to resolve WITHOUT_CROSS_COMPILER
not using a --sysroot [1].
PR: 196193 [1]
Sponsored by: EMC / Isilon Storage Division
Skip the log entry as there is nothing good to write out. Don't fail
the syscall though since it already succeeded. There's no reason
filemon's tracing failure should fail the already-succeeded syscall.
Record the error for later to return from close(2) on the filemon devfs
file descriptor.
Discussed with: markj, sjg, kib (briefly with kib)
Reported by: mjg
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
The mbuf provider is made up of a set of Statically Defined Tracepoints
which help us look into mbufs as they are allocated and freed. This can be
used to inspect the buffers or for a simplified mbuf leak detector.
New tracepoints are:
mbuf:::m-init
mbuf:::m-gethdr
mbuf:::m-get
mbuf:::m-getcl
mbuf:::m-clget
mbuf:::m-cljget
mbuf:::m-cljset
mbuf:::m-free
mbuf:::m-freem
There is also a translator for mbufs which gives some visibility into the structure,
see mbuf.d for more details.
Reviewed by: bz, markj
MFC after: 2 weeks
Sponsored by: Rubicon Communications (Netgate)
Differential Revision: https://reviews.freebsd.org/D5682
None of lstat(2), fstat(2), fstatat(2) were tracked either.
The other filemon implementations also do not track stat(2), nor
does bmake utilize it. The act of opening a file for read should
be enough to decide that a file is a dependency. There could be
rare cases where just having a file would cause a dependency but it
is unlikely.
MFC after: 2 weeks
Also noted by: sjg
Sponsored by: EMC / Isilon Storage Division
- proc.p_filemon is added which is protected by PROC_LOCK. This improves
performance and avoids double-fork issues, taking allproc_lock
while in syscalls, and walking the process tree in syscalls. A
particular proc.p_filemon can only be changed to NULL or another
filemon, or the filemon inherited, while the filemon->lock is held.
- Filemon are reference counted. On the last reference the log will be closed.
- When closing the devfs file handle, the filemon will be detached from all
processes and inheritance prevented.
- Disallow attaching to a process already being traced since filemon is
typically intended to be used on children only. This is allowed for
curproc as bmake relies on this behavior for rare cases when combining
.MAKE with .META.
- Detach any previously tracked process on ioctl(FILEMON_SET_PID).
- Handle error from devfs_set_cdevpriv() in filemon_open().
- The global filemon lock and lists are removed.
- A free list is no longer kept. Previously this list was
forever-expanding and never garbage cleaned.
- No longer loses track of double-forks. If the process holding the filemon
handle closes it will close the log rather than wait on a daemonized process,
but it will log all activity until it closes its handle. The filemon
will be removed from the process and not inherited.
- A separate process count is kept only as an optimization for
forced detachment to avoid taking allproc_lock and walking the entire
process tree.
- struct filemon access is protected by sx(9) filemon->lock as it was before.
- Add more comments and KASSERTS.
MFC after: 2 weeks
Reviewed by: kib, mjg, markj (all on previous versions)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5520
Archive member handling works again
meta mode, treat missing reads as for writes.
Update dirdeps.mk - much improved startup time.
Update meta.stage.mk - avoid ln when chmod required.
SYSCTL_COUNTER_U64_ARRAY() macro.
- Add proper asserts to the SYSCTL_COUNTER_U64_ARRAY() macro that checks
the size of the first element of the array.
- Add an example to the counter(9) manual page how to use the
SYSCTL_COUNTER_U64_ARRAY() macro.
- Add some missing symbolic links for counter(9) while at it.
This is several year's worth of fail point upgrades done at EMC Isilon. They
are interdependent enough that it makes sense to put a single diff up for them.
Primarily, we added:
- Changing all mainline execution paths to be lockless, which lets us use fail
points in more sleep-sensitive areas, and allows more parallel execution
- A number of additional commands, including 'pause' that lets us do some
interesting deterministic repros of race conditions
- The ability to dump the stacks of all threads sleeping on a fail point
- A number of other API changes to allow marking up the fail point's context in
the code, and firing callbacks before and after execution
- A man page update
Submitted by: Matthew Bryan <matthew.bryan@isilon.com>
Reviewed by: cem (earlier version), jhb, kib, pho
With feedback from: bdrewery
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5427
Specifically this fixes /usr/lib/libc.so stripping the paths to the
libraries. The reason for this in r266227 was both likely because ld(1) did
not fully respect --sysroot until r291226 and because of the lib32
build. The lib32 build does not use --sysroot into the /usr/lib32 path,
rather it only uses -L and -B into the /usr/lib32 path and --sysroot
into the normal (64bit) /usr/lib. The _LDSCRIPTROOT was added with
the ldscript support in bsd.lib.mk so that it builds a 32-bit-sysrooted pathed
ldscript in the object directory and then installs a normal unprefixed
version in installworld. This commit also fixes the rebuild during
install which was broken in r266227. This commit would break DIRDEPS_BUILD
build of lib32 but it does not currently have a way to build it anyhow.
For example, before this change we had in /usr/lib/libc.so:
GROUP ( libc.so.7 libc_nonshared.a libssp_nonshared.a )
Now it is restored to pre-r266227:
GROUP ( /lib/libc.so.7 /usr/lib/libc_nonshared.a /usr/lib/libssp_nonshared.a )
The motivation for this is in testing of lld.
From emaste:
lld does not have built-in search paths (e.g. /lib, /usr/lib) and relies on
-L arguments passed by the caller. As the linker is nearly always invoked
from the clang driver this is fine other than the fact that /usr/lib/libc.so
is an ldscript that refers to libc.so.7 which is in /lib, not /usr/lib.
PR: 207980
Reported by: emaste
Submitted by: emaste (based on)
Differential Revision: https://reviews.freebsd.org/D5637
No functional change.
This prevents adding empty targets to the main called target which is
confusing for debugging.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
This also fixes META MODE rebuilding these because the 'number of build commands'
changed from the previous build.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
Extend it to other cases of meta mode cookies so they get the proper rm
cookie behavior when a .meta file detects it needs to rebuild and fails.
Sponsored by: EMC / Isilon Storage Division
This obsoletes the _SKIP_BUILD check but keeps it for now until it
proves to be enough.
In the dirdeps build the first 'make all' or 'make' ran would invoke
'make dirdeps' which builds dependencies and then builds the current
directory in a sub-make (when BUILD_AT_LEVEL0 is no, which for us it
is). This behavior causes things attached to 'all:' to build in the
dirdeps phase AND the sub-make phase which creates all kinds of problems
for staging, meta file tracking, and races.
Sponsored by: EMC / Isilon Storage Division
This also fixes meta tracking for the beforeinstall since it had been
marked .PHONY before (in bsd.sys.mk).
Sponsored by: EMC / Isilon Storage Division
This is a follow-up to r291561 which reworked the bootstrap tool PATH
handling.
An example of this is when building lib/clang/libclangedit. At first
clang-tblgen will not be staged in the host tree so it will have
CLANG_TBLGEN=clang-tblgen set and exported. During the build though it
will stage clang-tblgen and then find it via the PATH. On the next
build it finds clang-tblgen in the stage directory and would set
CLANG_TBLGEN=<stagedir/usr/bin>/clang-tblgen thus causing the build
command to change. In both cases the same exact tool was used though so
there is no need to rebuild. If the tool did change the normal
meta/filemon handling would pick that up via timestamp comparisons and
rebuild.
Sponsored by: EMC / Isilon Storage Division
An example of where this is needed is in share/examples which for
'etc-examples' runs 'make -C SRCTOP/etc etc-examples' which installs
to the default DESTDIR otherwise.
Sponsored by: EMC / Isilon Storage Division
This came in r239572 for META_MODE handling but it doesn't make sense
since the staging is always done in make(all); make(buildincludes)
is never actually ran in the META_MODE build.
Reported by: bapt
Sponsored by: EMC / Isilon Storage Division
The meta file may decide the target is out of date but nothing
ensures that the *next* build will build this target if it
fails this time for some reason; it is still out-of-date
until it succeeds.
Convert the include/ cookie usage to the global versions.
Sponsored by: EMC / Isilon Storage Division
The inclusion of .MAKE.DEPENDFILE (.depend) has special logic in make
to ignore stale/missing dependencies. bmake 20160220 added a '.dinclude'
directive that uses the special logic for .depend when including the file.
This fixes a build error when a file is moved or deleted that exists in a
.depend.OBJ file. This happened in r292782 when sha512c.c "moved" and an
incremental build of lib/libmd would fail with:
make: don't know how to make /usr/src/lib/libcrypt/../libmd/sha512c.c. Stop
Now this will just be seen as a stale dependency and cause a rebuild:
make: /usr/obj/usr/src/lib/libmd/.depend.sha512c.o, 13: ignoring stale .depend for /usr/src/lib/libcrypt/../libmd/sha512c.c
--- sha512c.o ---
...
This rebuild will only be done once since the .depend.sha512c.o will
be updated on the build with the -MF flags.
This also removes -MP being passed for the .depend.OBJ generation (which
would create fake targets for system headers) since the logic is no
longer needed to protect from missing files.
Sponsored by: EMC / Isilon Storage Division
One example is in cddl/usr.sbin/dtrace/tests/common/aggs. It could be
fixed but other uses of this would break, especially in the
DIRDEPS_BUILD which uses the group names for stage cookies.
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
This will allow Makefile.depend to properly capture all dependencies.
It is not 100% optimal but works. Other options would be to use *.meta
here which would include too much or to keep a Makefile.depend per PROG
and include it from the main Makefile.depend which would not be
straight forward.
Sponsored by: EMC / Isilon Storage Division
Code may still be executing from the wrappers at unload time and thus is
not generally safe to unload. Converting the wrappers to use
EVENTHANDLER(9) will allow this to safely drain on active threads in
hooks. More work on EVENTHANDLER(9) is needed first.
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
There is no good way to guess if any of these will be needed but
they commonly are and add no extra overhead so just stage them.
Sponsored by: EMC / Isilon Storage Division
Currently the base.txz distribution does not get the BSD.debug.dist mtree
extracted into it. So if you start from that and then try to build a 3rd-party
application outside of buildworld it will by-default try installing the
debug files into a missing directory if they are being installed into /usr/lib.
Check for the existence before forcing the directory to be created rather than
the older way of running a shell command with test -d || mkdir -p always.
Reported by: HardenedBSD (https://github.com/HardenedBSD/secadm/issues/23)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5411
This was a regression in r295985.
bsd.dep.mk adds to SRCS for dtrace probes, yacc grammars and some
others.
The code that is moving is planned to be removed once FAST_DEPEND is
default (and the only option) though since FAST_DEPEND doesn't use this.
Pointyhat to: bdrewery
Sponsored by: EMC / Isilon Storage Division
Since m_cat() may copy data from the second mbuf chain into the last mbuf
of the first chain, it may free the first mbuf of the second chain. Thus,
the second chain is not guaranteed to be valid after m_cat() returns.
Reviewed by: glebius
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D5497
This is done to prevent not having CCACHE_DIR causing meta mode with filemon to
see stat changes in the ccache dir and cause rebuilds.
Sponsored by: EMC / Isilon Storage Division
improve cancellation robustness.
Introduce a new file operation, fo_aio_queue, which is responsible for
queueing and completing an asynchronous I/O request for a given file.
The AIO subystem now exports library of routines to manipulate AIO
requests as well as the ability to run a handler function in the
"default" pool of AIO daemons to service a request.
A default implementation for file types which do not include an
fo_aio_queue method queues requests to the "default" pool invoking the
fo_read or fo_write methods as before.
The AIO subsystem permits file types to install a private "cancel"
routine when a request is queued to permit safe dequeueing and cleanup
of cancelled requests.
Sockets now use their own pool of AIO daemons and service per-socket
requests in FIFO order. Socket requests will not block indefinitely
permitting timely cancellation of all requests.
Due to the now-tight coupling of the AIO subsystem with file types,
the AIO subsystem is now a standard part of all kernels. The VFS_AIO
kernel option and aio.ko module are gone.
Many file types may block indefinitely in their fo_read or fo_write
callbacks resulting in a hung AIO daemon. This can result in hung
user processes (when processes attempt to cancel all outstanding
requests during exit) or a hung system. To protect against this, AIO
requests are only permitted for known "safe" files by default. AIO
requests for all file types can be enabled by setting the new
vfs.aio.enable_usafe sysctl to a non-zero value. The AIO tests have
been updated to skip operations on unsafe file types if the sysctl is
zero.
Currently, AIO requests on sockets and raw disks are considered safe
and are enabled by default. aio_mlock() is also enabled by default.
Reviewed by: cem, jilles
Discussed with: kib (earlier version)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D5289
taskqueue_enqueue() was changed to support both fast and non-fast
taskqueues 10 years ago in r154167. It has been a compat shim ever
since. It's time for the compat shim to go.
Submitted by: Howard Su <howard0su@gmail.com>
Reviewed by: sephe
Differential Revision: https://reviews.freebsd.org/D5131
Freescale's QorIQ line includes a new ethernet controller, based on their
Datapath Acceleration Architecture (DPAA). This uses a combination of a Frame
manager, Buffer manager, and Queue manager to improve performance across all
interfaces by being able to pass data directly between hardware acceleration
interfaces.
As part of this import, Freescale's Netcomm Software (ncsw) driver is imported.
This was an attempt by Freescale to create an OS-agnostic sub-driver for
managing the hardware, using shims to interface to the OS-specific APIs. This
work was abandoned, and Freescale's primary work is in the Linux driver (dual
BSD/GPL license). Hence, this was imported directly to sys/contrib, rather than
going through the vendor area. Going forward, FreeBSD-specific changes may be
made to the ncsw code, diverging from the upstream in potentially incompatible
ways. An alternative could be to import the Linux driver itself, using the
linuxKPI layer, as that would maintain parity with the vendor-maintained driver.
However, the Linux driver has not been evaluated for reliability yet, and may
have issues with the import, whereas the ncsw-based driver in this commit was
completed by Semihalf 4 years ago, and is very stable.
Other SoC modules based on DPAA, which could be added in the future:
* Security and Encryption engine (SEC4.x, SEC5.x)
* RAID engine
Additional work to be done:
* Implement polling mode
* Test vlan support
* Add support for the Pattern Matching Engine, which can do regular expression
matching on packets.
This driver has been tested on the P5020 QorIQ SoC. Others listed in the
dtsec(4) manual page are expected to work as the same DPAA engine is included in
all.
Obtained from: Semihalf
Relnotes: Yes
Sponsored by: Alex Perez/Inertial Computing
Summary:
Many instances of bus_alloc_resource() simply use 0 and ~0 for start and end to
denote 'anywhere' with a given count. To clean this up, introduce a
bus_alloc_resource_anywhere() convenience function.
Bump __FreeBSD_version for the new API.
Reviewed By: jhb
Differential Revision: https://reviews.freebsd.org/D5370
This allows 'make analyze' or 'make OBJ.clang-analyzer' to run the
Clang static analyzer and present results on stdout.
Obtained from: NetBSD (CVS Rev. 1.3)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5449
- bsd.progs.mk is safe to include regardless of PROGS/PROGS_CXX/SCRIPTS
being set.
- bsd.progs.mk includes bsd.prog.mk always and will bring in
bsd.files.mk and bsd.obj.mk.
- DIRDEPS_BUILD: There's no need for _SKIP_BUILD or _SKIP_STAGING as the
PROGS were implicitly being built by the staging dependency anyway. This
was also preventing other objects from building if they were not part of
the staging sets.
- DIRDEPS_BUILD: This fixes PROGS without bsd.test.mk not staging.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
For PROGS this was recursing twice since MAKELEVEL0 is for 'dirdeps'
which then really builds in a sub-make.
Sponsored by: EMC / Isilon Storage Division
r96164 added them to avoid recursing twice with _SUBDIR. That issue was
fixed in bsd.subdir.mk in r291635 for all targets.
Sponsored by: EMC / Isilon Storage Division
As of r295661 elfcopy supports PE format for EFI boot binaries and is a
viable objcopy implementation for the base system and ports.
The (temporary) src.conf knob WITHOUT_ELFCOPY_AS_OBJCOPY knob may be set
to obtain the GNU version if necessary.
PR: 207091 [exp-run]
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
The OBJS_DEPEND_GUESS mechanism required moving the bsd.dep.mk inclusion
to after the checks, but left DEPENDFILE not-yet-set. Move it to
bsd.own.mk to resolve this.
Pointyhat to: bdrewery
Reported by: antoine (ports failures)
Sponsored by: EMC / Isilon Storage Division
After calling the cap_init(3) function Casper will fork from it's original
process, using pdfork(2). Forking from a process has a lot of advantages:
1. We have the same cwd as the original process.
2. The same uid, gid and groups.
3. The same MAC labels.
4. The same descriptor table.
5. The same routing table.
6. The same umask.
7. The same cpuset(1).
From now services are also in form of libraries.
We also removed libcapsicum at all and converts existing program using Casper
to new architecture.
Discussed with: pjd, jonathan, ed, drysdale@google.com, emaste
Partially reviewed by: drysdale@google.com, bdrewery
Approved by: pjd (mentor)
Differential Revision: https://reviews.freebsd.org/D4277
