tree 38523690945aacc4fd3c2f6254aca29ddffac1f9
parent 00a784e9a4ab33ebb0a42e35444c8b25a3a36bda
author Jonathan Nieder <jrnieder@gmail.com> 1339844262 -0500
committer Jonathan Nieder <jrnieder@gmail.com> 1548627799 -0800

liblzma: make dlopen()-based liblzma2 compatibility optional

Suppose I want to build a statically linked program:

	gcc -static -o app app.c -lrpm -llzma

Suppose further that librpm.a was built against a pre-5.0 version of
liblzma so it does not allocate as much space for reserved fields at
the end of lzma_stream as the current API requires.

(This is a hypothetical scenario --- Debian librpm does not provide a
static library.)

If liblzma uses unpatched lzma_code() from XZ Utils >= 5.0, then
during calls to librpm that try to compress or decompress an
xz-compressed RPM, lzma_code’s reserved field checks will overflow the
buffer and segfault.

If liblzma uses the modified version of lzma_code() which asks libdl
if liblzma.so.2 is resident and refrains from checking reserved fields
past the end of the old lzma_stream struct when the answer is "yes",
the behavior is no better.  The dynamic library liblzma.so.2 is _not_
resident, so lzma_code() dutifully reads reserved fields past the end
of the buffer --- segfault.

So the only safe behavior in the static library is to unconditionally
disable checks that might break for callers we want to continue to
support.

The new "./configure --enable-liblzma2-compat" option implements all
three sets of semantics:

 - "./configure --disable-liblzma2-compat" means to check the full set
   of reserved fields unconditionally.  You can use this to check how
   your application would behave with the unpatched library.

 - "./configure --enable-liblzma2-compat=auto" means to skip checks of
   reserved fields past the old end of struct lzma_stream when
   liblzma.so.2 is resident.  If a DSO built against liblzma2 shares
   the process image, the ABI-incompatible checks are skipped for
   safety, whereas in the usual case when no such DSO is resident, the
   full set of checks is run to help application developers remember
   to zero all reserved fields.

 - "./configure --enable-liblzma2-compat" makes liblzma skip the
   ABI-incompatible checks unconditionallty.  You can use this if you
   want your copy of liblzma to be usable by static libraries that
   were built against the old library.

Patch-Name: liblzma-make-dlopen-based-liblzma2-compatibility-opt.patch
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>