Move the initialization code to a general place instead of keeping it
specific to file-backed streams.
Fixes: 596a61cf6b (libio: Start to return errors when flushing fwrite's buffer [BZ #29459], 2025-01-28)
Reported-by: Florian Weimer <fweimer@redhat.com>
Reviewed-by: Arjun Shankar <arjun@redhat.com>
__LP64__ is a GCC extension and shouldn't be used in an installed
header.
Fixes: 596a61cf6b (libio: Start to return errors when flushing fwrite's buffer [BZ #29459], 2025-01-28)
Reported-by: Florian Weimer <fweimer@redhat.com>
Reviewed-by: Arjun Shankar <arjun@redhat.com>
As discussed in bug 32535, fflush fails on files opened for reading
using mmap after ungetc. Fix the logic to handle this case and still
compute the file offset correctly.
Tested for x86_64.
As discussed in bug 32529, fseek fails on files opened for reading
using mmap after ungetc. The implementation of fseek for such files
has an offset computation that's also incorrect after fflush. A
combined fix addresses both problems (with tests for both included as
well) and it seems reasonable to consider them a single bug.
Tested for x86_64.
As discussed in bug 32369 and required by POSIX, the POSIX feature
fflush (NULL) should flush input files, not just output files. The
POSIX requirement is that "fflush() shall perform this flushing action
on all streams for which the behavior is defined above", and the
definition for input files is for "a stream open for reading with an
underlying file description, if the file is not already at EOF, and
the file is one capable of seeking".
Implement this requirement in glibc. (The underlying flushing
implementation is what deals with avoiding errors for seeking on an
unseekable file.)
Tested for x86_64.
As discussed in bug 12724 and required by POSIX, before an input file
(based on an underlying seekable file descriptor) is closed, fclose is
sometimes required to seek that file descriptor to the correct offset,
so that any other file descriptors sharing the underlying open file
description are left at that offset (as a motivating example, a script
could call a sequence of commands each of which processes some data
from (seekable) stdin using stdio; fclose needs to do this so that
each successive command can read exactly the data not handled by
previous commands), but glibc fails to do this.
The precise POSIX wording has changed a few times; in the 2024 edition
it's "If the file is not already at EOF, and the file is one capable
of seeking, the file offset of the underlying open file description
shall be set to the file position of the stream if the stream is the
active handle to the underlying file description.".
Add appropriate logic to _IO_new_file_close_it to handle this case. I
haven't made any attempt to test or change things in this area for the
"old" functions.
Note that there was a previous attempt to fix bug 12724, reverted in
commit eb6cbd249f. The fix version here
addresses the original test in that bug report without breaking the
one given in a subsequent comment in that bug report (which works with
glibc before the patch, but maybe was broken by the original fix that
was reverted).
The logic here tries to take care not to seek the file, even to its
newly computed current offset, if at EOF / possibly not the active
handle; even seeking to the current offset would be problematic
because of a potential race (fclose computes the current offset,
another thread or process with the active handle does its own seek,
fclose does a seek (not permitted by POSIX in this case) that loses
the effect of the seek on the active handle in another thread or
process). There are tests included for various cases of being or not
being the active handle, though there aren't tests for the potential
race condition.
Tested for x86_64.
As discussed in bug 5994 (plus duplicates), POSIX requires fflush
after ungetc to discard pushed-back characters but preserve the file
position indicator. For this purpose, each ungetc decrements the file
position indicator by 1; it is unspecified after ungetc at the start
of the file, and after ungetwc, so no special handling is needed for
either of those cases.
This is fixed with appropriate logic in _IO_new_file_sync. I haven't
made any attempt to test or change things in this area for the "old"
functions; the case of files using mmap is addressed in a subsequent
patch (and there seem to be no problems in this area with files opened
with fmemopen).
Tested for x86_64.
When an error happens, fwrite is expected to return a value that is less
than nmemb. If this error happens while flushing its internal buffer,
fwrite is in a complex scenario: all the data might have been written to
the buffer, indicating a successful copy, but the buffer is expected to
be flushed and it was not.
POSIX.1-2024 states the following about errors on fwrite:
If an error occurs, the resulting value of the file-position indicator
for the stream is unspecified.
The fwrite() function shall return the number of elements successfully
written, which may be less than nitems if a write error is encountered.
With that in mind, this commit modifies _IO_new_file_write in order to
return the total number of bytes written via the file pointer. It also
modifies fwrite in order to use the new information and return the
correct number of bytes written even when sputn returns EOF.
Add 2 tests:
1. tst-fwrite-bz29459: This test is based on the reproducer attached to
bug 29459. In order to work, it requires to pipe stdout to another
process making it hard to reuse test-driver.c. This code is more
specific to the issue reported.
2. tst-fwrite-pipe: Recreates the issue by creating a pipe that is shared
with a child process. Reuses test-driver.c. Evaluates a more generic
scenario.
Co-authored-by: Florian Weimer <fweimer@redhat.com>
Reviewed-by: DJ Delorie <dj@redhat.com>
Check if TEST_CC supports -Wno-restrict before using it to avoid Clang
error:
error: unknown warning option '-Wno-restrict' [-Werror,-Wunknown-warning-option]
Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: Sam James <sam@gentoo.org>
Suppress Clang -Wgnu-folding-constant warnings, like
tst-freopen.c:44:13: error: variable length array folded to constant array as an extension [-Werror,-Wgnu-folding-constant]
44 | char temp[strlen (test) + 1];
| ^~~~~~~~~~~~~~~~~
Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: Sam James <sam@gentoo.org>
The C standard requires that ungetc guarantees at least one pushback,
but the malloc call to allocate the pushback buffer could fail, thus
violating that requirement. Fix this by adding a single byte pushback
buffer in the FILE struct that the pushback can fall back to if malloc
fails.
The side-effect is that if the initial malloc fails and the 1-byte
fallback buffer is used, future resizing (if it succeeds) will be
2-bytes, 4-bytes and so on, which is suboptimal but it's after a malloc
failure, so maybe even desirable.
A future optimization here could be to have the pushback code use the
single byte buffer first and only fall back to malloc for subsequent
calls.
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Reviewed-by: Maciej W. Rozycki <macro@redhat.com>
This was missed in a recent global change.
Fixes: 53fcdf5f74 (2024-11-25, "Silence most -Wzero-as-null-pointer-constant diagnostics")
Reported-by: "Maciej W. Rozycki" <macro@redhat.com>
Cc: Siddhesh Poyarekar <siddhesh@sourceware.org>
Cc: Bruno Haible <bruno@clisp.org>
Cc: Martin Uecker <uecker@tugraz.at>
Cc: Xi Ruoyao <xry111@xry111.site>
Cc: Florian Weimer <fweimer@redhat.com>
Cc: Joseph Myers <josmyers@redhat.com>
Signed-off-by: Alejandro Colomar <alx@kernel.org>
Reviewed-by: Maciej W. Rozycki <macro@redhat.com>
Trivial cleanup to limit _IO_least_marker so that it's clear that it is
unused outside of genops.
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
This fixes a buffer overflow in wide character string output, reproducing
when output fails, such as if the output fd is closed or is redirected
to a full device.
Wide character output data attempts to maintain the invariant that
`_IO_buf_base <= _IO_write_base <= _IO_write_end <= _IO_buf_end` (that is,
that the write region is a sub-region of `_IO_buf`). Prior to this commit,
this invariant is violated by the `_IO_wfile_overflow` function as so:
1. `_IO_wsetg` is called, assigning `_IO_write_base` to `_IO_buf_base`
2. `_IO_doallocbuf` is called, which jumps to `_IO_wfile_doallocate` via
the _IO_wfile_jumps vtable. This function then assigns the wide data
`_IO_buf_base` and `_IO_buf_end` to a malloc'd buffer.
Thus the invariant is violated. The fix is simply to reverse the order:
malloc the `_IO_buf` first and then assign `_IO_write_base` to it.
We also take this opportunity to defensively guard the initialization of
the number of unwritten characters via pointer arithmetic. We now check
that the buffer end is not before the buffer beginning; this matches a
similar defensive check in the narrow analogue `fileops.c`.
Add a test which fails without the fix.
Signed-off-by: Peter Ammon <corydoras@ridiculousfish.com>
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
tst-popen-fork failed to build for Hurd due to not being linked with
libpthread. This commit fixes that.
Tested with build-many-glibcs.py for i686-gnu.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
popen modifies its file handler book-keeping under a lock that wasn't
being taken during fork. This meant that a concurrent popen and fork
could end up copying the lock in a "locked" state into the fork child,
where subsequently calling popen would lead to a deadlock due to the
already (spuriously) held lock.
This commit fixes the deadlock by appropriately taking the lock before
fork, and releasing/resetting it in the parent/child after the fork.
A new test for concurrent popen and fork is also added. It consistently
hangs (and therefore fails via timeout) without the fix applied.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Since _IO_vtable_offset is used to detect the old binaries, set it
in _IO_old_file_init_internal before calling _IO_link_in which checks
_IO_vtable_offset. Add a glibc 2.0 test with copy relocation on
_IO_stderr_@GLIBC_2.0 to verify that fopen won't cause memory corruption.
This fixes BZ #32148.
Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: Noah Goldstein <goldstein.w.n@gmail.com>
Add new file libio/tst-fclose-unopened2.c that tests whether fclose on an
unopened file returns EOF.
This test differs from tst-fclose-unopened.c by ensuring the file's buffer
is allocated prior to double-fclose. A comment in tst-fclose-unopened.c
now clarifies that it is testing a file with an unallocated buffer.
Calling fclose on unopened files normally causes a use-after-free bug,
however the standard streams are an exception since they are not
deallocated by fclose.
Tested for x86_64.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
As reported in bug 23675 and shown up in the recently added tests of
different cases of freopen (relevant part of the test currently
conditioned under #if 0 to avoid a failure resulting from this bug),
freopen wrongly forces the stream to unoriented even when a mode with
,ccs= is specified, though such a mode is supposed to result in a
wide-oriented stream. Move the clearing of _mode to before the actual
reopening occurs, so that the main fopen implementation can leave a
wide-oriented stream in the ,ccs= case.
Tested for x86_64.
Add new file libio/tst-fclosed-unopened.c that tests whether fclose on
an unopened file returns EOF.
Calling fclose on unopened files normally causes a use-after-free bug,
however the standard streams are an exception since they are not
deallocated by fclose.
fclose returning EOF for unopened files is not part of the external
contract but there are dependancies on this behaviour. For example,
gnulib's close_stdout in lib/closeout.c.
Tested for x86_64.
Signed-off-by: Aaron Merey <amerey@redhat.com>
As reported in bug 32140, freopen leaks the FILE object when it
returns NULL: there is no valid use of the FILE * pointer (including
passing to freopen again or to fclose) after such an error return, so
the underlying object should be freed. Add code to free it.
Note 1: while I think it's clear from the relevant standards that the
object should be freed and the FILE * can't be used after the call in
this case (the stream is closed, which ends the lifetime of the FILE),
it's entirely possible that some existing code does in fact try to use
the existing FILE * in some way and could be broken by this change.
(Though the most common case for freopen may be stdin / stdout /
stderr, which _IO_deallocate_file explicitly checks for and does not
deallocate.)
Note 2: the deallocation is only done in the _IO_IS_FILEBUF case.
Other kinds of streams bypass all the freopen logic handling closing
the file, meaning a call to _IO_deallocate_file would neither be safe
(the FILE might still be linked into the list of all open FILEs) nor
sufficient (other internal memory allocations associated with the file
would not have been freed). I think the validity of freopen for any
other kind of stream will need clarifying with the Austin Group, but
if it is valid in any such case (where "valid" means "not undefined
behavior so required to close the stream" rather than "required to
successfully associate the stream with the new file in cases where
fopen would work"), more significant changes would be needed to ensure
the stream gets fully closed.
Tested for x86_64.
As reported in bug 32134, freopen does not clear the flags set in
fp->_flags2 by the "e", "m" or "c" mode characters. Clear these so
that they can be set or not as appropriate from the mode string passed
to freopen. The relevant test for "e" in tst-freopen2-main.c is
enabled accordingly; "c" is expected to be covered in a separately
written test (and while tst-freopen2-main.c does include transitions
to and from "m", that's not really a semantic flag intended to result
in behaving in an observably different way).
Tested for x86_64.
_wide_data and _mode are not available in legacy code, so do not attempt
to free the wide backup buffer in legacy code.
Resolves: BZ #32137 and BZ #27821
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
If a file descriptor is left unclosed and is cleaned up by _IO_cleanup
on exit, its backup buffer remains unfreed, registering as a leak in
valgrind. This is not strictly an issue since (1) the program should
ideally be closing the stream once it's not in use and (2) the program
is about to exit anyway, so keeping the backup buffer around a wee bit
longer isn't a real problem. Free it anyway to keep valgrind happy
when the streams in question are the standard ones, i.e. stdout, stdin
or stderr.
Also, the _IO_have_backup macro checks for _IO_save_base,
which is a roundabout way to check for a backup buffer instead of
directly looking for _IO_backup_base. The roundabout check breaks when
the main get area has not been used and user pushes a char into the
backup buffer with ungetc. Fix this to use the _IO_backup_base
directly.
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
When ungetc is called on an unused stream, the backup buffer is
allocated without the main get area being present. This results in
every subsequent ungetc (as the stream remains in the backup area)
checking uninitialized memory in the backup buffer when trying to put a
character back into the stream.
Avoid comparing the input character with buffer contents when in backup
to avoid this uninitialized read. The uninitialized read is harmless in
this context since the location is promptly overwritten with the input
character, thus fulfilling ungetc functionality.
Also adjust wording in the manual to drop the paragraph that says glibc
cannot do multiple ungetc back to back since with this change, ungetc
can actually do this.
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The conditionals for several mtrace-based tests in catgets, elf, libio,
malloc, misc, nptl, posix, and stdio-common were incorrect leading to
test failures when bootstrapping glibc without perl.
The correct conditional for mtrace-based tests requires three checks:
first checking for run-built-tests, then build-shared, and lastly that
PERL is not equal to "no" (missing perl).
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
These comments were written in 2003 (added in 2c008571c3), predating
the addition of getdelim(3)/getline(3) in POSIX.1-2008.
Reviewed-by: Sam James <sam@gentoo.org>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Deallocate the memory for the FILE structure when seeking to the end fails
in append mode.
Fixes: ea33158c96 ("Fix offset caching for streams and use it for ftell (BZ #16680)")
Since Glibc never provides symbol binary compatibility for relocatable
files, fix BZ #31766 by changing _IO_stderr_/_IO_stdin_/_IO_stdout to
compat symbols.
Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: Sunil K Pandey <skpgkp2@gmail.com>
This patch fixes BZ #27777 "fclose does a linear search, takes ages when
many FILE* are opened". Simply put, the master list of opened (FILE*),
namely _IO_list_all, is a singly-linked list. As a consequence, the
removal of a single element is in O(N), which cripples the performance
of fclose(). The patch switches to a doubly-linked list, yielding O(1)
removal. The one padding field in struct _IO_FILE, __pad5, is renamed
to _prevchain for a doubly-linked list. Since fields in struct _IO_FILE
after the _lock field are internal to glibc and opaque to applications.
We can change them as long as the size of struct _IO_FILE is unchanged,
which is checked as the part of glibc ABI with sizes of _IO_2_1_stdin_,
_IO_2_1_stdout_ and _IO_2_1_stderr_.
NB: When _IO_vtable_offset (fp) == 0, copy relocation will cover the
whole struct _IO_FILE. Otherwise, only fields up to the _lock field
will be copied to applications at run-time. It is used to check if
the _prevchain field can be safely accessed.
After opening 2 million (FILE*), the fclose() of 100 of them takes quite
a few seconds without the patch, and under 2 seconds with it on a loaded
machine.
No test is added since there are no functional changes.
Co-Authored-By: H.J. Lu <hjl.tools@gmail.com>
Signed-off-by: Alexandre Ferrieux <alexandre.ferrieux@orange.com>
Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
This test case calls `fopen':
FILE *fp = fopen (temp_file, "r");
however if that fails it reports `fdopen' being the origin of the error.
Adjust the message to say `fopen' then.
Starting from glibc 2.1, crt1.o contains _IO_stdin_used which is checked
by _IO_check_libio to provide binary compatibility for glibc 2.0. Add
crt1-2.0.o for tests against glibc 2.0. Define tests-2.0 for glibc 2.0
compatibility tests. Add and update glibc 2.0 compatibility tests for
stderr, matherr and pthread_kill.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
The gnulib version contains an important change (9ce573cde), which
fixes some problems with multithreading, entropy loss, and ASLR leak
nfo. It also fixes an issue where getrandom is not being used
on some new files generation (only for __GT_NOCREATE on first try).
The 044bf893ac removed __path_search, which is now moved to another
gnulib shared files (stdio-common/tmpdir.{c,h}). Tthis patch
also fixes direxists to use __stat64_time64 instead of __xstat64,
and move the include of pathmax.h for !_LIBC (since it is not used
by glibc). The license is also changed from GPL 3.0 to 2.1, with
permission from the authors (Bruno Haible and Paul Eggert).
The sync also removed the clock fallback, since clock_gettime
with CLOCK_REALTIME is expected to always succeed.
It syncs with gnulib commit 323834962817af7b115187e8c9a833437f8d20ec.
Checked on x86_64-linux-gnu.
Co-authored-by: Bruno Haible <bruno@clisp.org>
Co-authored-by: Paul Eggert <eggert@cs.ucla.edu>
Reviewed-by: Bruno Haible <bruno@clisp.org>
We would like to avoid statically defining any specific page size on
aarch64-gnu, and instead make sure that everything uses the dynamic
page size, available via vm_page_size and GLRO(dl_pagesize).
There are currently a few places in glibc that require EXEC_PAGESIZE
to be defined. Per Roland's suggestion [0], drop the static
GLRO(dl_pagesize) initializers (for now, only if EXEC_PAGESIZE is not
defined), and don't require EXEC_PAGESIZE definition for libio to
enable mmap usage.
[0]: https://mail.gnu.org/archive/html/bug-hurd/2011-10/msg00035.html
Signed-off-by: Sergey Bugaev <bugaevc@gmail.com>
Message-ID: <20240323173301.151066-4-bugaevc@gmail.com>
It improve fortify checks for sprintf, vsprintf, vsnsprintf, fprintf,
dprintf, asprintf, __asprintf, obstack_printf, gets, fgets,
fgets_unlocked, fread, and fread_unlocked. The runtime checks have
similar support coverage as with GCC.
For function with variadic argument (sprintf, snprintf, fprintf, printf,
dprintf, asprintf, __asprintf, obstack_printf) the fortify wrapper calls
the va_arg version since clang does not support __va_arg_pack.
Checked on aarch64, armhf, x86_64, and i686.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
WG14 decided to use the name C23 as the informal name of the next
revision of the C standard (notwithstanding the publication date in
2024). Update references to C2X in glibc to use the C23 name.
This is intended to update everything *except* where it involves
renaming files (the changes involving renaming tests are intended to
be done separately). In the case of the _ISOC2X_SOURCE feature test
macro - the only user-visible interface involved - support for that
macro is kept for backwards compatibility, while adding
_ISOC23_SOURCE.
Tested for x86_64.
The multibyte character needs to fit into the remaining buffer space,
not the already-written buffer space. Without the fix, we were never
moving the write pointer from the start of the buffer, always using
the single-character fallback buffer.
Fixes commit 04b76b5aa8 ("Don't error out writing
a multibyte character to an unbuffered stream (bug 17522)").
Samuel Zeter
Tulio Magno Quites Machado Filho
Joseph Myers
Paul Eggert
Florian Weimer
Adhemerval Zanella
H.J. Lu
Siddhesh Poyarekar
Alejandro Colomar
Peter Ammon
Arjun Shankar
Aaron Merey
Frédéric Bérat
Andreas Schwab
Carlos O'Donell
Philip Kaludercic
Alexandre Ferrieux
Maciej W. Rozycki
Samuel Thibault
Sergey Bugaev