Bug #915: Please help(nedbatchelder.com)
nedbatchelder.com
Bug #915: Please help
https://nedbatchelder.com/blog/202001/bug_915_please_help.html
13 comments
OK, I found the root cause.
The test harness uses mock to cause NamedTemporaryFile to fail in a few places:
I don't know who's fault this ultimately is, but it does illustrate a possible danger of mock testing when messing with the internals of other libraries.
The test harness uses mock to cause NamedTemporaryFile to fail in a few places:
with patch('tempfile._TemporaryFileWrapper') as mock_ntf:
mock_ntf.side_effect = OSError()
try:
file = _io.open(fd, mode, buffering=buffering,
newline=newline, encoding=encoding)
return _TemporaryFileWrapper(file, name, delete)
except BaseException:
_os.unlink(name)
_os.close(fd)
raise
I don't know who's fault this ultimately is, but it does illustrate a possible danger of mock testing when messing with the internals of other libraries.
That would do it. Calling _os.close() on an fd that is now owned by a file object is a big no-no.
Both pieces of code are broken. "except BaseException" is wrong because it'll catch KeyboardInterrupt and such, after the file object exists. The mock is wrong because it's introducing an exception that the original code can never raise and was written to assume wouldn't be raised, thus turning a corner case rare bug (fd reuse when a user ^Cs a program) into a bigger problem.
The tempfile code should change to only catch Exceptions, not BaseExceptions; it may opt to additionally catch all BaseExceptions and only unlink the name, but that's probably not sufficient anyway since a KeyboardInterrupt may be raised before the whole try block. Really, the unlink should happen in a wider scope try block, and the _os.close should be more tightly constrained to the _io.open call. And the exception matching should be narrower, because it's safer to leak an fd than to accidentally close it twice.
Then either the test harness should find some other way to implement that, or they can ask the tempfile people to move the _TemporaryFileWrapper out of the try block so they can keep using that hook point, if it hasn't been already by that point.
Both pieces of code are broken. "except BaseException" is wrong because it'll catch KeyboardInterrupt and such, after the file object exists. The mock is wrong because it's introducing an exception that the original code can never raise and was written to assume wouldn't be raised, thus turning a corner case rare bug (fd reuse when a user ^Cs a program) into a bigger problem.
The tempfile code should change to only catch Exceptions, not BaseExceptions; it may opt to additionally catch all BaseExceptions and only unlink the name, but that's probably not sufficient anyway since a KeyboardInterrupt may be raised before the whole try block. Really, the unlink should happen in a wider scope try block, and the _os.close should be more tightly constrained to the _io.open call. And the exception matching should be narrower, because it's safer to leak an fd than to accidentally close it twice.
Then either the test harness should find some other way to implement that, or they can ask the tempfile people to move the _TemporaryFileWrapper out of the try block so they can keep using that hook point, if it hasn't been already by that point.
It's still like that in CPython: https://github.com/python/cpython/blob/master/Lib/tempfile.p...
If there's a strong enough case that this is broken (and the KeyboardInterrupt example is a good one), I think we should file a bug about it.
EDIT: Created issue39318: https://bugs.python.org/issue39318. I think the possibility of this failing due to a KeyboardInterrupt, even if remote, is sufficient to consider this a bug (and with KeyboardInterrupt, it would be extremely subtle and hard to reproduce - more reason to get it fixed).
If there's a strong enough case that this is broken (and the KeyboardInterrupt example is a good one), I think we should file a bug about it.
EDIT: Created issue39318: https://bugs.python.org/issue39318. I think the possibility of this failing due to a KeyboardInterrupt, even if remote, is sufficient to consider this a bug (and with KeyboardInterrupt, it would be extremely subtle and hard to reproduce - more reason to get it fixed).
Impressive debugging expertise!
relevant history:
https://bugs.python.org/issue21058
https://bugs.python.org/issue26385
relevant history:
https://bugs.python.org/issue21058
https://bugs.python.org/issue26385
Wacky: the patch given in the initial submission of issue21058 is correct, but the patch that was actually committed incorrectly placed the wrapper inside the try block.
Interesting, that bug/patch was written by a guy I worked with at weta. He was always pushing code past its breaking point. I remember reading this bug before.
Looks like his fix didn’t have this bug. It may have been modified later, his patch was from 2014.
Looks like his fix didn’t have this bug. It may have been modified later, his patch was from 2014.
Very nice work.
> I don't know who's fault this ultimately is, but it does illustrate a possible danger of mock testing when messing with the internals of other libraries.
It doesn't look like there's any way that _TemporaryFileWrapper() could throw an OSError. I'd consider that mock a bug.
> I don't know who's fault this ultimately is, but it does illustrate a possible danger of mock testing when messing with the internals of other libraries.
It doesn't look like there's any way that _TemporaryFileWrapper() could throw an OSError. I'd consider that mock a bug.
It's also why you keep your try/catch blocks as small as possible. In this case, if the return was outside, which it probably should've been, then the error would've bubbled up instead of causing I/O failure.
Here's Ned's blog post describing the solution: https://nedbatchelder.com/blog/202001/bug_915_solved.html
Wow, awesome find.
macOS and probably iOS has an undocumented set of filesystem calls which require a "cookie." You call open() and provide a large integer of your choice (probably random); to close the file you need to provide the same integer. This fixed a lot of crashes in CoreData, where some rando was closing its SQLite fd.
macOS and probably iOS has an undocumented set of filesystem calls which require a "cookie." You call open() and provide a large integer of your choice (probably random); to close the file you need to provide the same integer. This fixed a lot of crashes in CoreData, where some rando was closing its SQLite fd.
> macOS and probably iOS has an undocumented set of filesystem calls which require a "cookie." You call open() and provide a large integer of your choice (probably random); to close the file you need to provide the same integer.
I want to learn more. Is there anything I can read about this? (I tried searching but I just keep on getting stuff about HTTP cookies not this.)
I want to learn more. Is there anything I can read about this? (I tried searching but I just keep on getting stuff about HTTP cookies not this.)
guarded_open_np guarded_close_np change_fdguard_np
https://opensource.apple.com/source/xnu/xnu-4903.241.1/bsd/k...
https://opensource.apple.com/source/xnu/xnu-4903.241.1/bsd/k...
I too am very interested in this, so tried searches with "macos", then "mach" and "xnu".
This led me to tentatively conclude that it's not in XNU.
On the one hand, http://newosxbook.com/src.jl?tree=xnu-1504.15.3&file=/bsd/sy... seems interesting.
On the other hand, https://github.com/apple/darwin-xnu/blob/0a798f6738bc1db0128... is suggested to be a red herring because of https://github.com/apple/darwin-xnu/blob/0a798f6738bc1db0128.... I have no idea if these are connected but they might be.
I'm not sure if my GitHub and/or Google queries aren't finding the relevant bits of magic though; had (only) a couple minutes and got curious.
This led me to tentatively conclude that it's not in XNU.
On the one hand, http://newosxbook.com/src.jl?tree=xnu-1504.15.3&file=/bsd/sy... seems interesting.
On the other hand, https://github.com/apple/darwin-xnu/blob/0a798f6738bc1db0128... is suggested to be a red herring because of https://github.com/apple/darwin-xnu/blob/0a798f6738bc1db0128.... I have no idea if these are connected but they might be.
I'm not sure if my GitHub and/or Google queries aren't finding the relevant bits of magic though; had (only) a couple minutes and got curious.
No, those are libc functions that let you use stdio (fprintf, fwrite, fscanf, etc.) with arbitrary "file" implementations. glibc has these too, but these have nothing to do with the underlying syscalls.
Then the problem isn't the fd leaking, the problem is that the fd is being closed twice. If SQLite owns that fd then it's obviously no longer owned by the NamedTemporaryFile, so something closed it. So either NamedTemporaryFile is closing an underlying fd twice (that sounds like a horrible Python bug, it should never be allowed) or some piece of code somewhere did one of those horrible "close all fds from 3 to 1024" things at one point, and SQLite then reused an fd that was still assigned to a live File object.
Very curious about this wonderful magic. How were you able to get the call stack from the strace line?
I basically set up my GDB with commands to stop on a specific pattern of "lseek, then close", and if the pattern isn't met it just automatically continues the program.
This is what the gdb script looks like:
It generates a few false positives but otherwise fairly quickly stops on the right syscall, at which point I could backtrace and prod the live program.
This is what the gdb script looks like:
set height 0
catch syscall close
catch syscall read
catch syscall lseek
disable 1 2
commands 2
disable 1 2
continue
end
commands 3
if $rdi == 31
enable 1 2
continue
else
continue
end
end
It generates a few false positives but otherwise fairly quickly stops on the right syscall, at which point I could backtrace and prod the live program.
This is fantastic. I thought I was pretty good at gdb, but this is next level (and it isn't complicated).
I've never seen any resource on "how to gdb a really gnarly bug"; this is a great example.
I've never seen any resource on "how to gdb a really gnarly bug"; this is a great example.
Debuggers are great! They're printf debugging, except that your printf can respond to the program state almost arbitrarily at runtime; not only that, they can modify the program state and can do so from a lot more lenient environment than your compiler would usually allow you to do. An advanced user can end up doing more programming in the debugger than outside of it.
Glad to see people writing in C-like languages finally get a taste of lisp programs ;)
This is very nice and interesting. Does that also work for lldb? Would sth like this also be possible with dtrace?
Can you recommend any good resources to learn more about this? Should I just study the LLDB/GDB references? Or maybe if you think there are not much good resources on this, maybe it would be a good idea if you could write a blog post about this?
Can you recommend any good resources to learn more about this? Should I just study the LLDB/GDB references? Or maybe if you think there are not much good resources on this, maybe it would be a good idea if you could write a blog post about this?
> Does that also work for lldb?
LLDB doesn't appear to have a way to set a breakpoint at syscalls, unfortunately. If it's dynamically linked, however, it will probably go through your platform's libc so you can probably achieve the same effect by breaking on all of the syscall wrapper functions.
LLDB doesn't appear to have a way to set a breakpoint at syscalls, unfortunately. If it's dynamically linked, however, it will probably go through your platform's libc so you can probably achieve the same effect by breaking on all of the syscall wrapper functions.
I am also interested in this. So far this video [0] from CPPCon'18 is the most complete and fresh I managed to find. Other than that, GDB manual is okay (LLDB's is not complete, as seems to be the case with most LLVM-based tools).
[0] <https://youtu.be/V1t6faOKjuQ>
[0] <https://youtu.be/V1t6faOKjuQ>
and you can do it in a rr trace on linux I think if its hard to reproduce which sometimes happen with similar programs
... wow. For other people, RR is a debugging tool for recording and playing back program execution. During the replay phase, you get great gdb debugger support, including running code backwards to find bugs.
It's not a toy: it can debug Google Chrome, Libre Office, and QEMU. However, RR is Linux-only.
Thanks alehander42!
Ref: https://en.wikipedia.org/wiki/Rr_(debugging)
It's not a toy: it can debug Google Chrome, Libre Office, and QEMU. However, RR is Linux-only.
Thanks alehander42!
Ref: https://en.wikipedia.org/wiki/Rr_(debugging)
Would https://www.python.org/dev/peps/pep-0446/ have anything to do with this?
We recently upgraded to celery 4 which had a side effect of file descriptors leaking between master and child processes, and database fds were being used for redis. Still no idea how it's happening, but the linked PEP seems to suggest there may be issues that I don't quite understand.
We recently upgraded to celery 4 which had a side effect of file descriptors leaking between master and child processes, and database fds were being used for redis. Still no idea how it's happening, but the linked PEP seems to suggest there may be issues that I don't quite understand.
> If you disable this block of code everything runs fine (although the tests fail). It's kinda creepy that the NamedTemporaryFile is leaking - it really shouldn't...
My guess is the tempfile is leaking due to the attribute caching here https://github.com/python/cpython/blob/master/Lib/tempfile.p... . But that doesn't really explain why the file is getting closed twice. (Presuming it's getting closed twice as otherwise the file descriptor wouldn't be reallocated)
Edit: realised it's just attaching the closer object to the function not the full object so avoiding this.
My guess is the tempfile is leaking due to the attribute caching here https://github.com/python/cpython/blob/master/Lib/tempfile.p... . But that doesn't really explain why the file is getting closed twice. (Presuming it's getting closed twice as otherwise the file descriptor wouldn't be reallocated)
Edit: realised it's just attaching the closer object to the function not the full object so avoiding this.
This whole wrapper nonsense to get with working with NamedTemporaryFile seems really suspect in general.
My gut instinct would be to rewrite the section OP found as a standard NamedTemporaryFile instantiation, then close it manually. The __getattr__ hack feels like a real code smell.
My gut instinct would be to rewrite the section OP found as a standard NamedTemporaryFile instantiation, then close it manually. The __getattr__ hack feels like a real code smell.
Nice find. Is there a good way to find out which line of code does a particular system call in the strace log? E.g. how did you find out that the close(31) is caused by the GC of the mock?
See my answer above (https://news.ycombinator.com/item?id=22031581) - the gist is that I set up GDB to break on a syscall matching the pattern of the offending close(31) (without also breaking on the zillions of legitimate close(31) calls generated by SQLite). Then I backtraced at that point and observed that this particular close wasn't from SQLite (which I'd already suspected), and the backtrace had several lines indicating that the close was from the GC finalization of a FileIO object.
The second step was to figure out which FileIO object was being GCed. I ran the program again and broke inside the FileIO __init__ function (which is written in C), conditioned on the fd being 31. From that breakpoint I did a Python backtrace (using the tools from https://github.com/python/cpython/blob/master/Misc/gdbinit) to figure out the Python code that was responsible for the allocation.
The third step was futzing around with the NamedTemporaryFile code until I could determine why it was leaking a file. This took a little while because I did not initially realize that it had been mocked. Once I did, the solution presented itself.
The second step was to figure out which FileIO object was being GCed. I ran the program again and broke inside the FileIO __init__ function (which is written in C), conditioned on the fd being 31. From that breakpoint I did a Python backtrace (using the tools from https://github.com/python/cpython/blob/master/Misc/gdbinit) to figure out the Python code that was responsible for the allocation.
The third step was futzing around with the NamedTemporaryFile code until I could determine why it was leaking a file. This took a little while because I did not initially realize that it had been mocked. Once I did, the solution presented itself.
I suppose adding f.close() right after the flush might help.
There was a similar bug in unraid that they ended up fixing in 6.8.0 where it turned out that SQLite wasn't handling some sort of error condition in read-ahead I/O? I wonder if this is related.
https://forums.unraid.net/bug-reports/prereleases/sqlite-dat...
From that report:
====== 8< ======
> In the Linux block layer each READ or WRITE can have various modifier bits set. In the case of a read-ahead you get READ|REQ_RAHEAD which tells I/O driver this is a read-ahead. In this case, if there are insufficient resources at the time this request is received, the driver is permitted to terminate the operation with BLK_STS_IOERR status. Here is an example in Linux md/raid5 driver.
> In case of Unraid it can definitely happen under heavy load that a read-ahead comes along and there are no 'stripe buffers' immediately available. In this case, instead of making calling process wait, it terminated the I/O. This has worked this way for years.
[...]
> What I suspect is that this is a bug in SQLite - I think SQLite is using direct-I/O (bypassing page cache) and issuing it's own read-aheads and their logic to handle failing read-ahead is broken. But I did not follow that rabbit hole - too many other problems to work on :/
====== 8< ======
Some related bugs they brought up:
https://bugzilla.kernel.org/show_bug.cgi?id=201685
https://patchwork.kernel.org/patch/10712695/
Edit: it appears that there's _something_ causing corruption when drivers fail read-ahead I/O. Whether it's SQLite or something else in Linux is another question.
https://forums.unraid.net/bug-reports/prereleases/sqlite-dat...
From that report:
====== 8< ======
> In the Linux block layer each READ or WRITE can have various modifier bits set. In the case of a read-ahead you get READ|REQ_RAHEAD which tells I/O driver this is a read-ahead. In this case, if there are insufficient resources at the time this request is received, the driver is permitted to terminate the operation with BLK_STS_IOERR status. Here is an example in Linux md/raid5 driver.
> In case of Unraid it can definitely happen under heavy load that a read-ahead comes along and there are no 'stripe buffers' immediately available. In this case, instead of making calling process wait, it terminated the I/O. This has worked this way for years.
[...]
> What I suspect is that this is a bug in SQLite - I think SQLite is using direct-I/O (bypassing page cache) and issuing it's own read-aheads and their logic to handle failing read-ahead is broken. But I did not follow that rabbit hole - too many other problems to work on :/
====== 8< ======
Some related bugs they brought up:
https://bugzilla.kernel.org/show_bug.cgi?id=201685
https://patchwork.kernel.org/patch/10712695/
Edit: it appears that there's _something_ causing corruption when drivers fail read-ahead I/O. Whether it's SQLite or something else in Linux is another question.
> I think SQLite is using direct-I/O (bypassing page cache)
Should be pretty easy to confirm/deny that suspicion and see if it's worth going down the rabbit hole, no?
Should be pretty easy to confirm/deny that suspicion and see if it's worth going down the rabbit hole, no?
strace -e open -o >(grep O_DIRECT) ./myprogram -o >(grep O_DIRECT)
Its cousin
diff <(cmd a) <(cmd b)
It also has a great unofficial name: the penguin operator.
<()
See? Penguin!
<()
See? Penguin!
Not a command line guru, but this looks interesting. Does it compare outputs of two commands? How does it do it?
it does[0] you are redirecting stdout of both of the subprocesses (the things in brackets) to the stdin of the diff command (I think it creates two different input streams, that diff knows how to compare).
[0] http://tldp.org/LDP/abs/html/process-sub.html
[0] http://tldp.org/LDP/abs/html/process-sub.html
<(command) is being replaced by a path to a file which contains the output of the command:
$ ls <(echo A)
/dev/fd/63
$ cat <(echo A)
A
diff <(command1) <(command2)
diff /dev/fd/62 /dev/fd/63It is called process substitution and specific to Bash (not POSIX shell compliant). The output of the commands in parenthesis will become runtime (file) inputs for the diff command.
More: https://www.gnu.org/software/bash/manual/html_node/Process-S...
More: https://www.gnu.org/software/bash/manual/html_node/Process-S...
It's not specific to bash -- it's in ksh as well. And zsh.
I'm also pretty sure it originated in ksh, not bash.
I'm also pretty sure it originated in ksh, not bash.
diff compares files line by line, here we just redirect the output of cmd1 and cmd2 via <, so they serve as 'proxies' for the file-args diff expects
you might want to read into redirection on the commandline, it's amazingly powerful
you might want to read into redirection on the commandline, it's amazingly powerful
It might be easiest to view it like so:
The command above with >(grep something) does the same thing, just with the other end of a pipe.
$ echo <(echo 1) <(echo 2)
/dev/fd/63 /dev/fd/62
The command above with >(grep something) does the same thing, just with the other end of a pipe.
It's not actually a pipe. >, < and | are IO redirection commands; but of those only | is a pipe. Pipes specifically "pipe" the STDOUT of one command as the STDIN of another. The other two instead point a command to the file descriptor of the output of another.
This is explained in more detail here:
https://askubuntu.com/questions/172982/what-is-the-differenc...
This is explained in more detail here:
https://askubuntu.com/questions/172982/what-is-the-differenc...
It is a pipe, just for a different definition of pipe.
| is the pipe operator in sh for doing a shell "pipeline". However, the parent comment is referring to a linux pipe as in pipe(7) [0].
One easy way to see this is with the following:
Those file descriptors were created using the pipe(2)[1] call by the shell, so it seems fine to refer to them to pipes.
I'll also note that <() / >() are _not_ using the "redirection operator". They're actually distinct operators for "process substitution"[2] in bash terminology. They look similar to redirects, but they're not the same operator, so that stack overflow answer isn't really relevant.
[0]: http://man7.org/linux/man-pages/man7/pipe.7.html
[1]: http://man7.org/linux/man-pages/man2/pipe.2.html
[2]: https://tldp.org/LDP/abs/html/process-sub.html
| is the pipe operator in sh for doing a shell "pipeline". However, the parent comment is referring to a linux pipe as in pipe(7) [0].
One easy way to see this is with the following:
$ ls -l <(echo foo)
lr-x------ 1 user group Jan 12 01:23 /proc/self/fd/16 -> 'pipe:[2937585]'
Those file descriptors were created using the pipe(2)[1] call by the shell, so it seems fine to refer to them to pipes.
I'll also note that <() / >() are _not_ using the "redirection operator". They're actually distinct operators for "process substitution"[2] in bash terminology. They look similar to redirects, but they're not the same operator, so that stack overflow answer isn't really relevant.
[0]: http://man7.org/linux/man-pages/man7/pipe.7.html
[1]: http://man7.org/linux/man-pages/man2/pipe.2.html
[2]: https://tldp.org/LDP/abs/html/process-sub.html
On POSIX systems, the line between file and pipe doesn’t exist. If you create a named pipe w/ mknod or mkfifo, you would interact with the named pipe as you would a normal file. Or block/character device. Etc.
In case of unnamed pipe the fds just exist as long as the process is running, then it disappears. Point is it’s irrelevant what you refer to as pipe, at the end of the day some file-like object is either written to or read from. And really, that file-like object is actually just a buffer.
In case of unnamed pipe the fds just exist as long as the process is running, then it disappears. Point is it’s irrelevant what you refer to as pipe, at the end of the day some file-like object is either written to or read from. And really, that file-like object is actually just a buffer.
The line absolutely exists, it's just behind an abstraction barrier for read(), write(), and close() calls. The creator of a pipe calls pipe() or pipe2(), and holders of the fd can get access to pipe-specific features with fcntl() and ioctl().
On linux at least, these two constructions are effectively identical from the view of the "piped" process.
# cat <(ls -l /proc/self/fd/)
total 0
lrwx------ 1 root root 64 Jan 12 15:02 0 -> /dev/pts/0
l-wx------ 1 root root 64 Jan 12 15:02 1 -> pipe:[20519634]
lrwx------ 1 root root 64 Jan 12 15:02 2 -> /dev/pts/0
lr-x------ 1 root root 64 Jan 12 15:02 3 -> /proc/23611/fd/
# ls -l /proc/self/fd/ | cat -
total 0
lrwx------ 1 root root 64 Jan 12 15:02 0 -> /dev/pts/0
l-wx------ 1 root root 64 Jan 12 15:02 1 -> pipe:[20518265]
lrwx------ 1 root root 64 Jan 12 15:02 2 -> /dev/pts/0
lr-x------ 1 root root 64 Jan 12 15:02 3 -> /proc/23621/fd/Yes, because “cat” can use either STDIN or a file as input:
https://github.com/coreutils/coreutils/blob/master/src/cat.c...
It’s an intentional design decision.
https://github.com/coreutils/coreutils/blob/master/src/cat.c...
It’s an intentional design decision.
I think we're probably in agreement, but we're getting caught up on the definition of "piping".
In my case I'm using it to describe the use of a kernel pipe object. In your case, you are using it to describe the higher-level concept of connection from one processes stdout to another process's stdin (or something along those lines).
In my case I'm using it to describe the use of a kernel pipe object. In your case, you are using it to describe the higher-level concept of connection from one processes stdout to another process's stdin (or something along those lines).
Repeating shawnz's comment more explicitly: you said that the mechanism used by <() is "not actually a pipe". What is `pipe:[20519634]` doing in the output of `cat <(ls -l /proc/self/fd/)`?
What it looks like is ls having its stdout (fd 1) directed to a pipe.
What it looks like is ls having its stdout (fd 1) directed to a pipe.
I think the point is that the directory listing is the same, not that cat supports both syntaxes.
Does anybody know if this can be done in Windows (not WSL)? E.g. is it possible to open a named pipe (?) through its name, and work with it as a sequential file?
This isn’t a Linux thing, it’s a bash thing. The <() syntax isn’t using a named pipe — it is creating an anonymous pipe. But you can replicate the same functions with named pipes.
A quick search says that Windows has support for named pipes as an IPC mechanism [1], but I’ve never used them. But this is a Windows API. I am not aware for any way to do this with plain cmd.exe. You’re not the first to ask though... [2]
[1] https://docs.microsoft.com/en-us/windows/win32/ipc/named-pip...
[2] https://superuser.com/questions/430466/in-windows-can-i-redi...
A quick search says that Windows has support for named pipes as an IPC mechanism [1], but I’ve never used them. But this is a Windows API. I am not aware for any way to do this with plain cmd.exe. You’re not the first to ask though... [2]
[1] https://docs.microsoft.com/en-us/windows/win32/ipc/named-pip...
[2] https://superuser.com/questions/430466/in-windows-can-i-redi...
Are you sure? How does diff know to read from those two pipes?
Even the echo example hints, that the created pipe has a name, and that name is then simply passed as the argument to the command.
Even the echo example hints, that the created pipe has a name, and that name is then simply passed as the argument to the command.
> Even the echo example hints, that the created pipe has a name
Well, the echo example shows that the pipe is reified at /dev/fd/63 (or /dev/fd/62 for the other pipe).
That's not a named pipe. It's a file descriptor identified by the integer that defines it, accessed under the /dev/fd listing of all file descriptors.
So, to sort of answer your questions:
> How does diff know to read from those two pipes?
They're two different pipes. Diff takes two arguments. Each argument is one of the pipes.
> (implied) Is there such a thing as an anonymous pipe, in any context?
Not if you consider /dev/fd/id-of-pipe to be a name. The operating system lists them all there. But usually you would use a named pipe if you wanted to be able to know the name, so that you could find it again if you didn't already have a reference to it. If you jammed your finger into my chest, I would be there, and my physical form would block the passage of your finger analogously to how an anonymous pipe nevertheless exists as an entry (two entries?) in /dev/fd. But that wouldn't tell you my name in the conventional sense, even though it would be a valid and unambiguous way to refer to me.
Well, the echo example shows that the pipe is reified at /dev/fd/63 (or /dev/fd/62 for the other pipe).
That's not a named pipe. It's a file descriptor identified by the integer that defines it, accessed under the /dev/fd listing of all file descriptors.
So, to sort of answer your questions:
> How does diff know to read from those two pipes?
They're two different pipes. Diff takes two arguments. Each argument is one of the pipes.
> (implied) Is there such a thing as an anonymous pipe, in any context?
Not if you consider /dev/fd/id-of-pipe to be a name. The operating system lists them all there. But usually you would use a named pipe if you wanted to be able to know the name, so that you could find it again if you didn't already have a reference to it. If you jammed your finger into my chest, I would be there, and my physical form would block the passage of your finger analogously to how an anonymous pipe nevertheless exists as an entry (two entries?) in /dev/fd. But that wouldn't tell you my name in the conventional sense, even though it would be a valid and unambiguous way to refer to me.
The <(foo | bar) syntax evaluates to a file path, e.g.
EDIT: Heh, 5 different replies within the same 60 second window.
$ echo <(echo hi)
/dev/fd/63
$ cat <( (echo foo; echo bar) | grep bar )
bar
EDIT: Heh, 5 different replies within the same 60 second window.
Yes. You're directing the STDOUT of each command as a FILE into diff. Diff takes two files, thus the duplication.
I don't think sqlite uses direct IO -- at least there's no mention of O_DIRECT in the source code
SQLite doesn't use direct I/O.
Unfortunately, the sqlite3 python interface doesn't expose Sqlite's extended error info so that you could see what's happening.
https://bugs.python.org/issue16379
There an uncommitted patch to get the extended error in this other, related issue: https://bugs.python.org/issue24139
Or maybe just try cffi or ctypes in python to call sqlite3_errcode(), sqlite3_extended_errcode() and sqlite3_errmsg(). The incantation to find an already linked sqlite3 lib:
import ctypes
import ctypes.util
sqlite = ctypes.CDLL(ctypes.util.find_library('sqlite3'))
https://bugs.python.org/issue16379
There an uncommitted patch to get the extended error in this other, related issue: https://bugs.python.org/issue24139
Or maybe just try cffi or ctypes in python to call sqlite3_errcode(), sqlite3_extended_errcode() and sqlite3_errmsg(). The incantation to find an already linked sqlite3 lib:
import ctypes
import ctypes.util
sqlite = ctypes.CDLL(ctypes.util.find_library('sqlite3'))
As mentioned on the Github issue [1], I'd suspect their SQLite optimizations:
[1] https://github.com/nedbat/coveragepy/issues/915
self.execute("pragma journal_mode=off").close()
# This pragma makes writing faster.
self.execute("pragma synchronous=off").close()
[1] https://github.com/nedbat/coveragepy/issues/915
I have tried removing the two pragmas, and the problem still happens.
For some reason Docker is struggling to download the image right now (lots of interested parties? :), I just wanted to trap writes to the database file and look at what's actually getting written there.
Is there any possibility e.g. of an FD reuse issue in the code somewhere? It might even be possible to spot it with strace (although you need to start the container with the right caps to try that). An example where this can happen is programs closing 'stderr' fd 2, only for e.g. SQLite to open its database reusing fd 2, then some library code calls fprintf(stderr, ..) etc.
Would love a copy of the DB 'sometime before' and after to look at. There are tools around for inspecting SQLite databases on a page-by-page basis
Is there any possibility e.g. of an FD reuse issue in the code somewhere? It might even be possible to spot it with strace (although you need to start the container with the right caps to try that). An example where this can happen is programs closing 'stderr' fd 2, only for e.g. SQLite to open its database reusing fd 2, then some library code calls fprintf(stderr, ..) etc.
Would love a copy of the DB 'sometime before' and after to look at. There are tools around for inspecting SQLite databases on a page-by-page basis
Neither of those make much sense as the cause anyway. Both sacrifice database consistency if the hosting process crashes or os crashes or reboots due to power failure, but with something like a coverage database, neither is particularly worrying. Worst case the users nukes the db, and reruns the tests.
Even if removing those pragmas fixed things, it feels likely that would just be hiding the issue instead of fixing it.
Disk io errors are not supposed to happen in sqlite except in cases like: accessing a corrupted database, drives failing or running out of disk space, bad permissions on some folder sqlite needs to write to.
So unless the test case involves a python process crashing or getting killed, I question if this is actually a bug in your your code.
I feel like this being some bug in either sqlite, or in the file systems used by docker (specifically by behaving in some manner not anticipated by sqlite).
With those optimizations, the only remaining sqlite disk IO should be the database itself, and the temp store. Since I stronly suspect the error you are getting is coming from sqlite, it would be helpful to know which of those two disk io cases it is.
Perhaps you can try the pragma to force the temp store to be in memory? The results of that might help us know if the disk io error is in reading/writing the database file itself, or if it is in reading/writing temp files.
Knowing that could narrow things down a bit.
Even if removing those pragmas fixed things, it feels likely that would just be hiding the issue instead of fixing it.
Disk io errors are not supposed to happen in sqlite except in cases like: accessing a corrupted database, drives failing or running out of disk space, bad permissions on some folder sqlite needs to write to.
So unless the test case involves a python process crashing or getting killed, I question if this is actually a bug in your your code.
I feel like this being some bug in either sqlite, or in the file systems used by docker (specifically by behaving in some manner not anticipated by sqlite).
With those optimizations, the only remaining sqlite disk IO should be the database itself, and the temp store. Since I stronly suspect the error you are getting is coming from sqlite, it would be helpful to know which of those two disk io cases it is.
Perhaps you can try the pragma to force the temp store to be in memory? The results of that might help us know if the disk io error is in reading/writing the database file itself, or if it is in reading/writing temp files.
Knowing that could narrow things down a bit.
Thanks, this was a good idea. I tried adding "pragma temp_store = 2", and it did not fix the problem, though I'm not sure I wasn't already using memory temp storage. The interaction of compile-time macros and pragmas is hard to decipher from the Python bindings.
Can we please have an informative title?
Bug #915: Please help tells me nothing.
Bug #915: Please help tells me nothing.
The original title was something like "Ned Batchelder, the author of coverage.py, was asking for help on Bug #915", but it was against the guidelines, so the title was restored.
I knew from the domain name that it would be interesting; Ned Batchelder is no dummy.
Sure, but many of us have know idea who they are, and an informative title really helps, even if you're familiar with the source.
Whenever I come across a mailing list bug or a reddit post or similar with a "PLEASE HELP ME!" title I just auto-downvote/remove/delete it.
I refer to them as "help-me zombies". They have no brains and they want to eat yours. They don't know what they need help with, but they need help.
I refer to them as "help-me zombies". They have no brains and they want to eat yours. They don't know what they need help with, but they need help.
Can someone suggest what the admins can turn this title into so it's not mystery meat (e.g. no idea what we're clicking through to, no idea whether the domain in question is ad-laden, etc).
Help: Unresolved SQLite I/O Error in coverage.py
Had an I/O issue in SQLite yesterday: the cause was a full disk, due to years of SQLite backups!
The solution is clearly to restore a full system backup from a time that it had fewer SQLite backups :P
But first back it up in the present state in case anything goes wrong.
The fix was to undo a seemingly safe performance tweak...
From https://github.com/nedbat/coveragepy/commit/e4b8389b27ce4e76...
"Put the filename calc back in _connect. Fixes #916
It was moved to __init__ to avoid recalculating, but the directory could have changed, so we need to wait to do the work. Instead, only do the relpath on systems that need it (Windows Py 2)."
From https://github.com/nedbat/coveragepy/commit/e4b8389b27ce4e76...
"Put the filename calc back in _connect. Fixes #916
It was moved to __init__ to avoid recalculating, but the directory could have changed, so we need to wait to do the work. Instead, only do the relpath on systems that need it (Windows Py 2)."
This post is about issue #915 which seems unrelated/still open
This is an editorialized title and as far as I know it's not even accurate. The author maintains, but did not create coverage.py.
(Edit: Original title has been changed, so this comment is no longer necessary.)
(Edit: Original title has been changed, so this comment is no longer necessary.)
Looking at the commit history, he is the author of almost all the code written in the last ten years. Even if there was someone before him (who was it? I couldn't find any reference to them), how could we not consider him the author of the project by now?
It was Gareth Rees.
https://garethrees.org/2001/12/04/python-coverage/
https://garethrees.org/2001/12/04/python-coverage/
The original title made reference to Ned as the "creator of coveragepy", which specifically means the person who created it. The original author was Gareth Rees, according to Contributors.txt [1], and Ned Batchelder took over in 2004.
I don't mean it to suggest that Ned has a lessor role than creator, just that the post's title was changed to add inaccurate information. I figured that if the moderators saw this and decided that the context of the author's role in Coverage was important, they could update the title to reflect his actual role.
[1] https://github.com/nedbat/coveragepy/blob/master/CONTRIBUTOR...
I don't mean it to suggest that Ned has a lessor role than creator, just that the post's title was changed to add inaccurate information. I figured that if the moderators saw this and decided that the context of the author's role in Coverage was important, they could update the title to reflect his actual role.
[1] https://github.com/nedbat/coveragepy/blob/master/CONTRIBUTOR...
Sorry, I missed the obvious place to look into it.
I was unaware of that and it seems people have fixed the title for me. Thanks!
(Not helpful as such, but this reminds me of Mickens' "The Night Watch".
> HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS
https://www.usenix.org/system/files/1311_05-08_mickens.pdf )
> HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS
https://www.usenix.org/system/files/1311_05-08_mickens.pdf )
The fix is described at <a href="https://github.com/nedbat/coveragepy/commit/e4b8389b27ce4e76.... it was simple, but elusive because of unsafe assumptions made in the name of performance.
"Put the filename calc back in _connect. Fixes #916
It was moved to __init__ to avoid recalculating, but the directory could have changed, so we need to wait to do the work. Instead, only do the relpath on systems that need it (Windows Py 2)."
"Put the filename calc back in _connect. Fixes #916
It was moved to __init__ to avoid recalculating, but the directory could have changed, so we need to wait to do the work. Instead, only do the relpath on systems that need it (Windows Py 2)."
OT: After the lamentation of the "Github Monoculture", why was development of coveragepy moved from bitbucket to github?
https://bitbucket.org/ned/coveragepy/src/default/ original project https://nedbatchelder.com/blog/201405/github_monoculture.htm... original post https://news.ycombinator.com/item?id=7690897 HN comments
https://bitbucket.org/ned/coveragepy/src/default/ original project https://nedbatchelder.com/blog/201405/github_monoculture.htm... original post https://news.ycombinator.com/item?id=7690897 HN comments
strace log from the failure:
Notice the close(31) right after the lseek. This close(31) is not generated by SQLite; rather, it's being generated by the GC finalization of an unrelated file object:
Note that normal SQLite3 closes look like this:
EDIT 1: OK, I found the line of code that allocates this file, at line 398 in apprise_api/api/views.py:
If you disable this block of code everything runs fine (although the tests fail). It's kinda creepy that the NamedTemporaryFile is leaking - it really shouldn't...