From: Roland McGrath In the oddball situation where one thread is using ptrace on another thread sharing the same mm, and then someone sharing that mm causes a coredump, there is a deadlock possible if the traced thread is in TASK_TRACED state. It leaves all the threads sharing that mm wedged and permanently unkillable. This patch checks for that situation and brings a thread out of TASK_TRACED if its tracer is part of the same coredump (i.e. shares the same mm). It's not pretty, but it does the job. Signed-off-by: Roland McGrath Signed-off-by: Andrew Morton --- 25-akpm/fs/exec.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 56 insertions(+) diff -puN fs/exec.c~fix-coredump_wait-deadlock-with-ptracer-tracee-on-shared-mm fs/exec.c --- 25/fs/exec.c~fix-coredump_wait-deadlock-with-ptracer-tracee-on-shared-mm 2005-01-05 15:33:47.705697824 -0800 +++ 25-akpm/fs/exec.c 2005-01-05 15:33:47.710697064 -0800 @@ -1336,6 +1336,7 @@ static void zap_threads (struct mm_struc struct task_struct *g, *p; struct task_struct *tsk = current; struct completion *vfork_done = tsk->vfork_done; + int traced = 0; /* * Make sure nobody is waiting for us to release the VM, @@ -1351,10 +1352,65 @@ static void zap_threads (struct mm_struc if (mm == p->mm && p != tsk) { force_sig_specific(SIGKILL, p); mm->core_waiters++; + if (unlikely(p->ptrace) && + unlikely(p->parent->mm == mm)) + traced = 1; } while_each_thread(g,p); read_unlock(&tasklist_lock); + + while (unlikely(traced)) { + /* + * We are zapping a thread and the thread it ptraces. + * The tracee won't come out of TASK_TRACED state until + * its ptracer detaches. That happens when the ptracer + * dies, but it synchronizes with us and so won't get + * that far until we finish the core dump. If we're + * waiting for the tracee to synchronize but it stays + * blocked in TASK_TRACED, then we deadlock. So, for + * this weirdo case we have to do another round with + * tasklist_lock write-locked to __ptrace_unlink the + * children that might cause this deadlock. That will + * wake them up to process their pending SIGKILL. + * + * First, give everyone we just killed a chance to run + * so they can all get into the coredump synchronization. + * That should leave only the TASK_TRACED stragglers for + * us to wake up. If a ptracer is still running, we'll + * have to come around again after letting it finish. + */ + yield(); + traced = 0; + write_lock_irq(&tasklist_lock); + do_each_thread(g,p) { + if (mm != p->mm || p == tsk || + !p->ptrace || p->parent->mm != mm) + continue; + if ((p->parent->flags & (PF_SIGNALED|PF_EXITING)) || + (p->parent->state & (TASK_TRACED|TASK_STOPPED))) { + /* + * The parent is in the process of exiting + * itself, or else it's stopped right now. + * It cannot be in a ptrace call, and would + * have to read_lock tasklist_lock before + * it could start one, so we are safe here. + */ + __ptrace_unlink(p); + } else { + /* + * Blargh! The ptracer is not dying + * yet, so we cannot be sure that it + * isn't in the middle of a ptrace call. + * We'll have to let it run to get into + * coredump_wait and come around another + * time to detach its tracee. + */ + traced = 1; + } + } while_each_thread(g,p); + write_unlock_irq(&tasklist_lock); + } } static void coredump_wait(struct mm_struct *mm) _