Skip to content

Do not abort connection after encoding error in postgresql - #537

Merged
G1gg1L3s merged 9 commits into
masterfrom
G1gg1L3s/T2561-postgresql-do-not-abort-connection
May 16, 2022
Merged

G1gg1L3s merged 9 commits into
masterfrom
G1gg1L3s/T2561-postgresql-do-not-abort-connection

Conversation

@G1gg1L3s

@G1gg1L3s G1gg1L3s commented May 13, 2022 •

Copy link
Copy Markdown
Contributor

This PR implement skipping postgres packets from database after encoding error, almost in the same way as the previous PR does for mysql. It also adds a bunch of tests for different cases:

  • Simple queries
  • Extended queries (and prepared statements, that are basically the same under the hood).
  • Batched extended queries (when database sends n packets and waits for the next request)

The only issue with postgres is that a state of transaction is stored at database side, and changes in case of some errors (even parsing ones). When the state is changed, commit is basically rollback:

test=# begin;
BEGIN
test=*# select 1/0;
ERROR:  division by zero
test=!# commit;
ROLLBACK

But our error will not trigger that:

test=# begin;
BEGIN
test=*# select value_bytes from testtype;
ERROR:  encoding error in column "value_bytes"
test=# commit;
COMMIT

So, for now I will mention it in the docs. Currently there is no straight and simple solution.

P.S. test.py crossed the milestone of 10 thousands lines, so my congratulations! It takes 5 second for my formatter to proceed the file. That's what I call success.

Checklist

G1gg1L3s added 5 commits May 11, 2022 16:31
Though they are failling right now.
This workflow is the most complicated, because database returns `n`
rows and then waits for another `Execute` packet. Test that
"encoding error" will not block, abort or desynchronize the connection.
Comment thread tests/test.py
self.assertEqual(row, None)

loop = asyncio.get_event_loop()
loop.run_until_complete(test())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fiound that python has high-level helper asyncio.run for that. Lets start to use it?) Works since python3.7.

@G1gg1L3s G1gg1L3s May 13, 2022 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to use that, because I use it in small scripts myself. But I've read from the docs:

This function cannot be called when another asyncio event loop is running in the same thread.

I don't know whether our test run in parallel, but if they do, this can cause a problem (or I just don't understand this statement correctly)

@Lagovas Lagovas May 13, 2022 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then lets create new event loop for that?) use new_event_loop instead of get_event_loop

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but I don't quite understand the reasons. Could you explain why?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will help run code in parallel if need. every thread or test case will have own loop and it will safe to stop loop on failure and don't touch other tests that can rely on common loop in current thread

Comment thread tests/test.py
self.assertEqual(ex.exception.message,
'encoding error in column "value_bytes"')
row = await conn.fetchrow(select_query, data['id'])
self.assertEqual(row, None)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, can you explain why we expect None here? As I understand:

  1. we inserted data and it is already in the database
  2. We tried to get this data in the transaction, took error from acra side and handled exception
  3. We try to get same data but get None as result, not exception due to same error and not error that select returns no data?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are a little wrong, here is the correct workflow:

  1. We insert corrupted data that will trigger decryption error and so it is accessible for the next operations.
  2. We begin transaction and insert another row of data.
  3. We trigger a decryption error and expect that db-driver does a rollback
  4. We check that indeed, because of the rollback, the data that we insert in the transaction, has not been saved.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, thanks. can you add some comment above this check with explanation why we expect here None?
Because I'm sure that someone will forget that most of ORM use ROLLBACK on any exception. Or mention that after that we expect that ORM will do ROLLBACK. Just for case if event this ORM will change his logic and stop do ROLLBACK on errors.

G1gg1L3s added 4 commits May 13, 2022 17:20
...instead of explicitly capturing the exception and then hacking
around to find out which of the properties contains it to test that
it's the same as we expected. It also allows not caring about '\n'.
@G1gg1L3s
G1gg1L3s merged commit 8cd2047 into master May 16, 2022
@G1gg1L3s
G1gg1L3s deleted the G1gg1L3s/T2561-postgresql-do-not-abort-connection branch May 16, 2022 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants