Changed AttributeDescription.{ad_cname,ad_lang} to struct berval everywhere
Deleted ad_free() everywhere
Added ad_mutex to init.c
The AttributeDescriptions are in a linked list hanging off of the
corresponding AttributeType.
error reporting to client and syslog. And indexing, of course.
Now, the problem is that matching rules get called from different
places that are inconsistent in what an assertedValue is. When doing
a modify, a full certificate value is passed (to verify it isn't
already there). When doing a search or compare, the passed value is
in the syntax of the matching rule.
Consistency would require that the caller extracts an asserted value
from the full value before calling smr_match. It can do this by
calling smr_convert (it was unused, was it meant to be used for
this?).
Unfortunately, the caller is typically value_find, value_match, etc.
that have themselves little knowledge of what they are dealing with,
so their interface needs to be extended, new flag values or new
arguments, so that they know if they have a value in attribute type
syntax or in matching rule syntax.
to activate. The bdb_decode works in-place on the db data. Add/Search are
OK, I think Modify needs to be tweaked. Don't use this yet unless you want
to help finish it.
LDAP/UDP messages. Slapd marks received CLDAP messages as LDAP_VERSION2.
The client library can generate CLDAP queries if -Protocol 2 is chosen,
otherwise not. LDAPv2 CLDAP cannot query the slapd rootDSE, gets no reply.
Compile with -DLDAP_CONNECTIONLESS to use this code.
For slapd, use "-h cldap://" to listen on UDP.
For ldapsearch, use "-H cldap://" to query on UDP.
Client-side support is very minimal:
no automatic timeout/retries
no basedn wildcard expansion on results
no support for specifying multiple servers at once.
Summary of changes is cited below.
The patch still needs some cosmetic changes to be made, but is ready for testing.
-----Original Message-----
From: Sam Drake [mailto:drake@timesten.com]
Sent: Saturday, April 07, 2001 10:40 PM
To: 'mitya@seismic.ru'
Cc: openldap-devel@OpenLDAP.org
Subject: RE: Slapd frontend performance issues
FYI, here is a short description of the changes I made. I'll package up the
changes asap, but it may take a couple of days.
The performance numbers quoted in this report were seen at my location with
a 100,000 object database ... the slower numbers I mentioned earlier were
reported by a customer with a 1,000,000 object database.
I also can't explain the very poor performance I saw with OpenLDAP and LDBM
with a 100,000 object database.
...Sam Drake / TimesTen Performance Software
----------
Work Performed
OpenLDAP 2.0.9, including back-sql, was built successfully on Solaris
8 using gcc. The LDAP server itself, slapd, passed all tests bundled
with OpenLDAP. OpenLDAP was built using Sleepycat LDBM release 3.1.17
as the "native" storage manager.
The experimental back-sql facility in slapd was also built
successfully. It was built using Oracle release 8.1.7 and the Oracle
ODBC driver and ODBC Driver Manager from Merant. Rudimentary testing
was performed with the data and examples provided with back-sql, and
back-sql was found to be functional.
Slapd and back-sql were then tested with TimesTen, using TimesTen
4.1.1. Back-sql was not immediately functional with TimesTen due to a
number of SQL limitations in the TimesTen product.
Functional issues encountered were:
1. Back-sql issued SELECT statements including the construct,
"UPPER(?)". While TimesTen supports UPPER, it does not support the
use of parameters as input to builtin functions. Back-sql was
modified to convert the parameter to upper case prior to giving it
to the underlying database ... a change that is appropriate for all
databases.
2. Back-sql issued SELECT statements using the SQL CONCAT function.
TimesTen does not support this function. Back-sql was modified to
concatentate the necessary strings itself (in "C" code) prior to
passing the parameters to SQL. This change is also appropriate for
all databases, not just TimesTen.
Once these two issues were resolved, back-sql could successfully
process LDAP searches using the sample data and examples provided with
back-sql.
While performance was not measured at this point, numerous serious
performance problems were observed with the back-sql code and the
generated SQL. In particular:
1. In the process of implementing an LDAP search, back-sql will
generate and execute a SQL query for all object classes stored in
back-sql. During the source of generating each SQL query, it is
common for back-sql to determine that a particular object class can
not possibly have any members satisfying the search. For example,
this can occur if the query searches an attribute of the LDAP
object that does not exist in the SQL schema. In this case,
back-sql would generate and issue the SQL query anyway, including a
clause such as "WHERE 1=0" in the generated SELECT. The overhead
of parsing, optimizing and executing the query is non-trivial, and
the answer (the empty set) is known in advance. Solution: Back-sql
was modified to stop executing a SQL query when it can be
predetermined that the query will return no rows.
2. Searches in LDAP are fundamentally case-insensitive ("abc" is equal
to "aBc"). However, in SQL this is not normally the case.
Back-sql thus generated SQL SELECT statements including clauses of
the form, "WHERE UPPER(attribute) = 'JOE'". Even if an index is
defined on the attribute in the relational database, the index can
not be used to satisfy the query, as the index is case sensitive.
The relational database then is forced to scan all rows in the
table in order to satisfy the query ... an expensive and
non-scalable proposition. Solution: Back-sql was modified to allow
the schema designer to add additional "upper cased" columns to the
SQL schema. These columns, if present, contain an upper cased
version of the "standard" field, and will be used preferentially
for searching. Such columns can be provided for all searchable
columns, some columns, or no columns. An application using
database "triggers" or similar mechanisms can automatically
maintain these upper cased columns when the standard column is
changed.
3. In order to implement the hierarchical nature of LDAP object
hierarchies, OpenLDAP uses suffix searches in SQL. For example, to
find all objects in the subtree "o=TimesTen,c=us", a SQL SELECT
statement of the form, "WHERE UPPER(dn) LIKE '%O=TIMESTEN,C=US'"
would be employed. Aside from the UPPER issue discussed above, a
second performance problem in this query is the use of suffix
search. In TimesTen (and most relational databases), indexes can
be used to optimize exact-match searches and prefix searches.
However, suffix searches must be performed by scanning every row in
the table ... an expensive and non-scalable proposition. Solution:
Back-sql was modified to optionally add a new "dn_ru" column to the
ldap_entries table. This additional column, if present, contains a
byte-reversed and upper cased version of the DN. This allows
back-sql to generate indexable prefix searches. This column is
also easily maintained automatically through the use of triggers.
Results
A simple database schema was generated holding the LDAP objects and
attributes specified by our customer. An application was written to
generate test databases. Both TimesTen and Oracle 8.1.7 were
populated with 100,000 entry databases.
Load Times
Using "slapadd" followed by "slapindex", loading and indexing 100,000
entries in an LDBM database ran for 19 minutes 10 seconds.
Using a C++ application that used ODBC, loading 100,000 entries into
a disk based RDBMS took 17 minutes 53 seconds.
Using a C++ application that used ODBC, loading 100,000 entries into
TimesTen took 1 minute 40 seconds.
Search Times
The command, "timex timesearch.sh '(cn=fname210100*)'" was used to
test search times. This command issues the same LDAP search 4000
times over a single LDAP connection. Both the client and server
(slapd) were run on the same machine.
With TimesTen as the database, 4000 queries took 14.93 seconds, for a
rate of 267.9 per second.
With a disk based RDBMS as the database, 4000 queries took 77.79 seconds,
for a
rate of 51.42 per second.
With LDBM as the database, 1 query takes 76 seconds, or 0.076 per
second. Something is clearly broken.
Now operations that set the status of an entry to CREATING (add.c, modrdn.c)
need to set it to COMMIT, by calling cache_entry_commit, before returning
the entry itself, otherwise the entry is removed from the cache
and its private data is freed.
Should fix crashes due to add failures as in ITS#1245
slapadd core-dumps when destroying db's env (Sleepycat 3.2.9) (ITS#1239)
Only call ldbm_shutdown_env if the database has been opened, ie. when
li->li_dbenv != NULL. Would appear any time a shutdown occurred and
not all LDBM databases were opened.
The old monitoring stuff has been removed; the new backend is
enabled by using --enable-monitor at configure time and requires
database monitor
in slapd.conf to be activated. At present it implements a subset
of the old monitoring options, and it should be extendable to
a number of different subsystems. The search operation has been
implementd; it does not honor abandon or size/time limits, though.
The compare and the abandon operations are planned.
Copyright Pierangelo Masarati <ando@sys-net.it>; the code is provided
AS IS with NO GUARANTEE. It can be used and distributed under the
conditions stated by the OpenLDAP Public License.
David A. Cooper <david.cooper@nist.gov> (ITS#1232)
according to draft-ietf-ldapbis-dn-05.txt
A copyright statement follows:
The functions normalize_unicode(), get_hexpair(), write_hex_pair(),
get_next_byte(), get_next_char(), get_ber_length(),
ber_parse_primitive_string(), ber_parse_string(), String_normalize(),
DirectoryString_normalize(), PrintableString_normalize(),
IA5String_normalize(), ber_parse_primitive_bitstring(),
ber_parse_bitstring(), getNext8bits(), bitString_normalize(), match_oid(),
match_key(), get_validated_av_in_dn(), get_validated_rdn_in_dn(),
and get_validated_dn() in this file were developed at the National Institute
of Standards and Technology by employees of the Federal Government in the
course of their official duties. Pursuant to title 17 Section 105 of the
United States Code the code in these functions is not subject to copyright
protection and is in the public domain. The copyright for all other code in
this file is as specified below.
by slapd/tools/*; slap_mods_free is needed by ldbm_back_modrdn after
fixing ITS#1184 (at present -DMULTIATTRVAL_RDN is needed when compiling
back-ldbm/modrdn.c to trigger the compilation of new code).
for LDBM backends called "dbsync", which takes minimum of one argument up
to 3 args which are sync frequency, # of delays, and delay periods. See
man page update for "dbsync" configuration for more details.
with minor changes by committer
---
Copyright 2001, F5 Networks, Inc, All rights reserved.
This software is not subject to any license of F5 Networks.
This is free software; you can redistribute and use it
under the same terms as OpenLDAP itself.
- librewrite, for string rewriting; it may be used in back-ldap
by configuring with '--enable-rewrite'. It must be used in
back-meta. There's a text file, 'libraries/librewrite/RATIONALE',
that explains the usage and the features. More comprehensive
documentation will follow.
- enhancements of back-ldap (ITS#989,ITS#998,ITS#1002,ITS#1054 and ITS#1137)
including dn rewriting, a fix to group acl matching and so
- back-meta: a new backend that proxies a set of remote servers
by spawning queries. It uses portions of back-ldap and the rewrite
capabilities of librewrite. It can be compiled by configuring
with `--enable-ldap --enable-rewrite --enable-meta'.
There's a text file, 'servers/slapd/back-meta/Documentation', that
describes the main features and config statements.
Note: someone (Kurt?) should run 'autoconf' and commit 'configure' as
my autoconf version must be different: my configures contain a number
of differences and I didn't feel comfortable in adding them :)
caseIgnoreSubstringsIndexer(), caseIgnoreSubstringsFilter(),
caseExactIndexer(), caseExactFilter(), caseExactSubstringsIndexer() and
caseExactSubstringsFilter() use UTF8normalize
Copyright 2000, Caldera Systems, Inc All rights reserved.
This software is not subject to any license of Caldera Systems, Inc
This is free software; you can redistribute and use it
under the same terms as OpenLDAP itself
---
Copyright 2000 Stig Venaas, UNINETT
All rights reserved.
Redistribution and use in source and binary forms are permitted
without restriction or fee of any kind as long as this notice is
preserved. The name of UNINETT may not be used to endorse or promote
products derived from this software without specific prior written
permission. This software is provided ``as is'' without express or
implied warranty.
