Discussion:
BUG #7969: Postgres Recovery Fatal With: "incorrect local pin
(too old to reply)
Josh Berkus
2013-03-27 18:27:33 UTC
Permalink
Folks,

So I'm a bit surprised that this bug report hasn't gotten a follow-up.
Does this sound like the known 9.2.2 corruption issue, or is it
potentially something else?
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
--
Sent via pgsql-bugs mailing list (pgsql-***@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Heikki Linnakangas
2013-03-27 19:04:47 UTC
Permalink
Post by Josh Berkus
Folks,
So I'm a bit surprised that this bug report hasn't gotten a follow-up.
Does this sound like the known 9.2.2 corruption issue, or is it
potentially something else?
It seems like a new issue. At a quick glance, I think there's a bug in
heap_xlog_update, ie. the redo routine of a heap update. If the new
tuple is put on a different page, and at redo, the new page doesn't
exist (that's normal if it was later vacuumed away), heap_xlog_update
Post by Josh Berkus
{
nbuffer = XLogReadBuffer(xlrec->target.node,
ItemPointerGetBlockNumber(&(xlrec->newtid)),
false);
if (!BufferIsValid(nbuffer))
return;
page = (Page) BufferGetPage(nbuffer);
if (XLByteLE(lsn, PageGetLSN(page))) /* changes are applied */
{
UnlockReleaseBuffer(nbuffer);
if (BufferIsValid(obuffer))
UnlockReleaseBuffer(obuffer);
return;
}
}
Notice how in the first 'return' above, obuffer is not released.

I'll try to create a reproducible test case for this, and fix..

- Heikki
--
Sent via pgsql-bugs mailing list (pgsql-***@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Heikki Linnakangas
2013-03-27 19:48:33 UTC
Permalink
Post by Heikki Linnakangas
Post by Josh Berkus
Folks,
So I'm a bit surprised that this bug report hasn't gotten a follow-up.
Does this sound like the known 9.2.2 corruption issue, or is it
potentially something else?
It seems like a new issue. At a quick glance, I think there's a bug in
heap_xlog_update, ie. the redo routine of a heap update. If the new
tuple is put on a different page, and at redo, the new page doesn't
exist (that's normal if it was later vacuumed away), heap_xlog_update
Post by Josh Berkus
{
nbuffer = XLogReadBuffer(xlrec->target.node,
ItemPointerGetBlockNumber(&(xlrec->newtid)),
false);
if (!BufferIsValid(nbuffer))
return;
page = (Page) BufferGetPage(nbuffer);
if (XLByteLE(lsn, PageGetLSN(page))) /* changes are applied */
{
UnlockReleaseBuffer(nbuffer);
if (BufferIsValid(obuffer))
UnlockReleaseBuffer(obuffer);
return;
}
}
Notice how in the first 'return' above, obuffer is not released.
I'll try to create a reproducible test case for this, and fix..
Ok, here's how to reproduce it:

create table foo (i int4 primary key);
insert into foo select generate_series(1,1000);
checkpoint;
-- update a tuple from the first page, new tuple goes to last page
update foo set i = 10000 where i = 1;
-- delete everything on pages > 1
delete from foo where i > 10;
-- truncate the table, including the page the updated tuple went to
vacuum verbose foo;

pg_ctl stop -m immediate

This bug was introduced by commit
8805ff6580621d0daee350826de5211d6bb36ec3, in 9.2.2 (and 9.1.7 and
9.0.11), which fixed multiple WAL replay issues with Hot Standby. Before
that commit, replaying a heap update didn't try to keep both buffers
locked at the same time, which is necessary for the correctness of hot
standby. The patch fixed that, but missed releasing the old buffer in
this corner case. I was not able to come up with a scenario with
full_page_writes=on where this would fail, but I'm also not 100% sure it
can't happen.

I scanned through the commit, and couldn't see any other instances of
this kind of a bug. heap_xlog_update is more complicated than other redo
functions, with all the return statements inside it. It could use some
refactoring, but for now, I'll commit the attached small fix.

- Heikki
Tom Lane
2013-03-27 21:28:07 UTC
Permalink
Post by Heikki Linnakangas
This bug was introduced by commit
8805ff6580621d0daee350826de5211d6bb36ec3, in 9.2.2 (and 9.1.7 and
9.0.11), which fixed multiple WAL replay issues with Hot Standby.
Ooops. Thanks for finding that.

regards, tom lane
--
Sent via pgsql-bugs mailing list (pgsql-***@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Yunong Xiao
2013-04-11 16:52:07 UTC
Permalink
This bug was introduced by commit 8805ff6580621d0daee350826de5211d6bb36ec3, in 9.2.2 (and 9.1.7 and 9.0.11), which fixed multiple WAL replay issues with Hot Standby. Before that commit, replaying a heap update didn't try to keep both buffers locked at the same time, which is necessary for the correctness of hot standby. The patch fixed that, but missed releasing the old buffer in this corner case. I was not able to come up with a scenario with full_page_writes=on where this would fail, but I'm also not 100% sure it can't happen.
I scanned through the commit, and couldn't see any other instances of this kind of a bug. heap_xlog_update is more complicated than other redo functions, with all the return statements inside it. It could use some refactoring, but for now, I'll commit the attached small fix.
- Heikki
<fix-heap-update-redo-buffer-leak.patch>
Heikki:

Is it safe to apply the patch you attached? We'd really like to roll out this fix to production instead of waiting for this to be released as a postgres minor version update.

-Yunong
--
Sent via pgsql-bugs mailing list (pgsql-***@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Tom Lane
2013-04-11 17:45:13 UTC
Permalink
Post by Yunong Xiao
This bug was introduced by commit 8805ff6580621d0daee350826de5211d6bb36ec3, in 9.2.2 (and 9.1.7 and 9.0.11), which fixed multiple WAL replay issues with Hot Standby. Before that commit, replaying a heap update didn't try to keep both buffers locked at the same time, which is necessary for the correctness of hot standby. The patch fixed that, but missed releasing the old buffer in this corner case. I was not able to come up with a scenario with full_page_writes=on where this would fail, but I'm also not 100% sure it can't happen.
I scanned through the commit, and couldn't see any other instances of this kind of a bug. heap_xlog_update is more complicated than other redo functions, with all the return statements inside it. It could use some refactoring, but for now, I'll commit the attached small fix.
- Heikki
<fix-heap-update-redo-buffer-leak.patch>
Is it safe to apply the patch you attached? We'd really like to roll out this fix to production instead of waiting for this to be released as a postgres minor version update.
Eh? That patch *is* in the latest minor versions.

regards, tom lane
--
Sent via pgsql-bugs mailing list (pgsql-***@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Loading...