articles.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>Articles</title><link href="https://www.psycopg.org/articles/" rel="alternate"/><link href="https://www.psycopg.org/articles.xml" rel="self"/><id>urn:uuid:1f583092-c4bb-3a77-9f6f-c8ac41335aab</id><updated>2026-08-12T00:00:00Z</updated><author><name/></author><entry><title>Row-by-row streaming with server-side cursors</title><link href="https://www.psycopg.org/articles/2026/08/12/server-side-cursors/" rel="alternate"/><updated>2026-08-12T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:8f331f26-2ded-381a-8da3-9350e65eb0cd</id><content type="html">&lt;p&gt;One of the most common mistakes when working with large datasets in PostgreSQL
is fetching everything into memory at once. It is the path of least
resistance: you write your query, call &lt;tt class="docutils literal"&gt;fetchall()&lt;/tt&gt;, and iterate over the
results. For small datasets this is fine. For large ones, you will quickly
find your application consuming far more memory than it needs to.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Even after closing the cursor, Python’s memory allocator might hold on the
memory used longer than expected; a long-running service might end up with a
large memory footprint in a server. So this is usually a condition you will
want to avoid.&lt;/p&gt;
&lt;p&gt;Psycopg offers a cleaner solution: &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/cursors.html#server-side-cursors"&gt;server-side cursors&lt;/a&gt;, which allow you to
stream results from the database row by row, keeping your memory footprint
small and predictable regardless of how large the result set is. Server-side
cursors are available in both Psycopg 2 and Psycopg 3.&lt;/p&gt;
&lt;div class="section" id="how-it-works"&gt;
&lt;h2&gt;How it works&lt;/h2&gt;
&lt;p&gt;When you execute a query normally, PostgreSQL runs it, assembles the full
result set, and sends it to the client. With a &lt;a class="reference external" href="https://www.postgresql.org/docs/current/sql-declare.html"&gt;server-side cursor&lt;/a&gt;,
PostgreSQL instead keeps the result set on the server and sends rows to the
client in batches as they are requested. This means your Python process only
ever holds a small number of rows in memory at any one time. Giving the cursor
a name is what tells Psycopg to use a server-side cursor rather than a
client-side one.&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;dbname=mydb&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;my_cursor&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT * FROM very_large_table&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;process&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Everything else looks identical to a normal cursor: you can iterate over it,
call &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.fetchone"&gt;fetchone()&lt;/a&gt;, or call &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.fetchmany"&gt;fetchmany(n)&lt;/a&gt; to retrieve a specific number of
rows at a time.&lt;/p&gt;
&lt;p&gt;In Psycopg 3, server-side cursors are exposed through the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.ServerCursor"&gt;ServerCursor
class&lt;/a&gt;, which may also be instantiated directly. In Psycopg 2 only the name
parameter can be used.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="controlling-batch-size"&gt;
&lt;h2&gt;Controlling batch size&lt;/h2&gt;
&lt;p&gt;By default, Psycopg 3 fetches rows in batches of 100 when iterating over a
server-side cursor. You can control this with the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.ServerCursor.itersize"&gt;itersize attribute&lt;/a&gt;:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;my_cursor&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;itersize&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT * FROM very_large_table&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;process&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Larger batch sizes reduce the number of round-trips to the database but
increase memory usage per batch. The right value depends on the size of your
rows and your application's memory constraints. For most use cases the default
of 100 is a reasonable starting point.&lt;/p&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;itersize&lt;/tt&gt; only affects the iteration on the cursor using for. Other methods
such as &lt;tt class="docutils literal"&gt;fetchone()&lt;/tt&gt;, and &lt;tt class="docutils literal"&gt;fetchmany(n)&lt;/tt&gt; will fetch exactly the number of
rows requested.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="using-server-side-cursors-with-async"&gt;
&lt;h2&gt;Using server-side cursors with async&lt;/h2&gt;
&lt;p&gt;Psycopg 3's async support extends naturally to server-side cursors, making it
straightforward to stream large result sets in an async context without
blocking:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AsyncConnection&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;dbname=mydb&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;my_cursor&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT * FROM very_large_table&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;process&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The API is deliberately symmetric with the synchronous version. Just add the
await keyword where needed.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="a-practical-example-exporting-large-datasets"&gt;
&lt;h2&gt;A practical example: exporting large datasets&lt;/h2&gt;
&lt;p&gt;A common real-world use case is exporting data to a file. Without server-side
cursors, exporting a table with millions of rows means loading all of them
into memory before you can write a single line. With streaming, you can write
rows to disk as they arrive:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;csv&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;dbname=mydb&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;export_cursor&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT id, name, email FROM users ORDER BY id&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;users.csv&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;w&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;writer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;email&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;                &lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This will happily export a table of any size while using only a few kilobytes
of memory for the current batch of rows.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="things-to-be-aware-of"&gt;
&lt;h2&gt;Things to be aware of&lt;/h2&gt;
&lt;p&gt;Server-side cursors require an open transaction for their lifetime. Psycopg 3
handles this automatically, but it is worth knowing that this transaction
exists and must be kept open on the server as long as the cursor is used. For
long-running exports or processing jobs this is usually fine, but be mindful
of holding transactions open unnecessarily in high-concurrency environments.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="summary"&gt;
&lt;h2&gt;Summary&lt;/h2&gt;
&lt;p&gt;Server-side cursors are one of those features that you may not need every day,
but when you do need them: processing exports, running aggregations over large
tables, streaming data to downstream systems, they make a significant
difference. The API is clean, the behaviour is predictable, and the memory
savings for large datasets are substantial.&lt;/p&gt;
&lt;p&gt;If you find Psycopg useful in your work, please consider &lt;a class="reference external" href="https://github.com/sponsors/psycopg/"&gt;sponsoring the
project&lt;/a&gt; to help ensure its continued
development.&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>Integrating Psycopg with FastAPI and Pydantic</title><link href="https://www.psycopg.org/articles/2026/06/24/pydantic-fastapi/" rel="alternate"/><updated>2026-06-24T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:eaec8456-6be7-3dc7-bd65-e65620e74b08</id><content type="html">&lt;p&gt;&lt;a class="reference external" href="https://fastapi.tiangolo.com/"&gt;FastAPI&lt;/a&gt; is currently the established
framework to write a REST API in Python. Its clever use of dependency
injections and static typing makes developing web services a breeze, using a
minimal amount of code and a clean and consistent architecture.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;It's not difficult to see why. A minimal working server, serving a list and
the details of a set of people, is the following:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;fastapi&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;FastAPI&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;My People&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="nd"&gt;&amp;#64;app&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/person&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;list_people&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="nd"&gt;&amp;#64;app&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/person/&lt;/span&gt;&lt;span class="si"&gt;{id}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;get_person&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Just &lt;tt class="docutils literal"&gt;pip install 'fastapi[standard]'&lt;/tt&gt;, save the above script in a
&lt;tt class="docutils literal"&gt;server.py&lt;/tt&gt; file, and run it with &lt;tt class="docutils literal"&gt;fastapi dev server.py&lt;/tt&gt;: this will bring
up a development server serving your API on port 8000:&lt;/p&gt;
&lt;pre class="code sh literal-block"&gt;
$&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/person&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;[]&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/person/123&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;:&lt;span class="s2"&gt;&amp;quot;123&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Furthermore you can point your browser to &lt;a class="reference external" href="http://localhost:8000/docs"&gt;http://localhost:8000/docs&lt;/a&gt; and
you will be served a &lt;a class="reference external" href="https://swagger.io/"&gt;Swagger&lt;/a&gt; web interface allowing
you to interact with the API:&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img alt="swagger-1.png" src="swagger-1.png" /&gt;
&lt;/div&gt;
&lt;p&gt;That's already quite some infrastructure in place! However it doesn't specify
anything about our model. What does a &lt;em&gt;Person&lt;/em&gt; look like? This is where
Pydantic comes into play.&lt;/p&gt;
&lt;div class="section" id="enter-pydantic"&gt;
&lt;h2&gt;Enter Pydantic&lt;/h2&gt;
&lt;p&gt;&lt;a class="reference external" href="https://pydantic.dev/docs/validation/latest/get-started/"&gt;Pydantic&lt;/a&gt; lets you
define an object's attributes, validate them, and describe them, just using
&lt;a class="reference external" href="https://typing.python.org/en/latest/spec/annotations.html"&gt;Python type annotations&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Let's say that our person will have a name (mandatory) and a date of birth
(which we allow to be nullable because we might not know it). We will also
want an id on the record - let's say it will be a UUID. This definition can
be represented with:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;datetime&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;dt&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;uuid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;UUID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;uuid4&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;pydantic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UUID&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;uuid4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;date_of_birth&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This representation is enough to create an object with advanced validation,
arguments conversion and output. In a Python shell we can play with it:&lt;/p&gt;
&lt;pre class="code python-console literal-block"&gt;
&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;John Doe&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;date_of_birth&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1970-01-01&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;  &lt;span class="c1"&gt;# object with attributes converted to the right types&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;Person(id=UUID('94d69603-c3f7-4dd4-9d64-eb1a24fc6fed'), name='John Doe', date_of_birth=datetime.date(1970, 1, 1))

&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model_dump&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# convert the object to a Python dictionary&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;{'id': UUID('94d69603-c3f7-4dd4-9d64-eb1a24fc6fed'),
 'name': 'John Doe',
 'date_of_birth': datetime.date(1970, 1, 1)}

&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model_dump_json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# convert the object to a JSON string&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;'{&amp;quot;id&amp;quot;:&amp;quot;94d69603-c3f7-4dd4-9d64-eb1a24fc6fed&amp;quot;,&amp;quot;name&amp;quot;:&amp;quot;John Doe&amp;quot;,&amp;quot;date_of_birth&amp;quot;:&amp;quot;1970-01-01&amp;quot;}'&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Now that we have this model we can annotate the views to signify that they
will return a &lt;tt class="docutils literal"&gt;Person&lt;/tt&gt; object and a list of such objects:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="nd"&gt;&amp;#64;app&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/person&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;list_people&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="nd"&gt;&amp;#64;app&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/person/&lt;/span&gt;&lt;span class="si"&gt;{id}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;get_person&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UUID&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;John Doe&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This is enough to give a pretty representation, from formal definition to
result samples, in the Swagger interface:&lt;/p&gt;
&lt;div class="figure"&gt;
&lt;img alt="swagger-2.png" src="swagger-2.png" /&gt;
&lt;/div&gt;
&lt;p&gt;Of course the objects that we are returning are fake and incomplete. The
database knows the truth and it won't come as a surprise that we will want to
use &lt;a class="reference external" href="https://www.psycopg.org/"&gt;Psycopg&lt;/a&gt; to retrieve it...&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="connecting-views-and-the-database"&gt;
&lt;h2&gt;Connecting views and the database&lt;/h2&gt;
&lt;p&gt;Let's create a table for this example and seed it with a couple of test
records:&lt;/p&gt;
&lt;pre class="code sql literal-block"&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;TABLE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;person&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;PRIMARY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;DEFAULT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gen_random_uuid&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;NOT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;date_of_birth&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;person&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;date_of_birth&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
   &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Alice'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'1970-01-01'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;person&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;date_of_birth&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
   &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Bob'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;I'll assume that the table is created in the &amp;quot;default database&amp;quot;, the one you
connect to by typing &lt;tt class="docutils literal"&gt;psql&lt;/tt&gt; with no other argument. If you need to use a
different database you can specify &lt;a class="reference external" href="https://www.postgresql.org/docs/current/libpq-envars.html"&gt;&lt;tt class="docutils literal"&gt;PG*&lt;/tt&gt; env vars&lt;/a&gt; to select a different database to
connect to, or hardcode a connection string in the following
examples. Configuring FastAPI is outside the scope of this article (but psst...
check &lt;a class="reference external" href="https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/"&gt;Pydantic Settings&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;Creating a working database connection, configuring it correctly, managing its
life cycle, are matters separate from the code of the view functions, which
just &lt;em&gt;require&lt;/em&gt; a database connection. FastAPI offers an elegant method to
separate these two independent concerns: the &lt;a class="reference external" href="https://fastapi.tiangolo.com/tutorial/dependencies/"&gt;dependency injection&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The first part is creating a &lt;em&gt;dependency&lt;/em&gt;: this is simply a function that
creates, and optionally disposes of, an object. The one managing our
connection could be simply:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;get_connection&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# Use the PG* env vars to configure the database connection&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Using &lt;tt class="docutils literal"&gt;yield&lt;/tt&gt; in the context manager will make sure that the connection is
correctly closed when the view function terminates.&lt;/p&gt;
&lt;p&gt;Once we have this normal Python function we can define a &lt;em&gt;type alias&lt;/em&gt; to
enrich a type (the Psycopg connection) with the metadata defining how to
obtain one (there are other ways, but defining an alias minimises
repetitions):&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;typing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Annotated&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;Conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Annotated&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Depends&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;get_connection&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This completes the creation of the dependency. In order to use it, i.e. to
define that the &lt;tt class="docutils literal"&gt;get_person()&lt;/tt&gt; function requires a database connection to
work, it is sufficient to add an argument of type &lt;tt class="docutils literal"&gt;Conn&lt;/tt&gt; to the function
definition. In order to reuse code between the two views, we can write a
helper function, &lt;tt class="docutils literal"&gt;fetch_people()&lt;/tt&gt;, taking an optional &lt;tt class="docutils literal"&gt;id&lt;/tt&gt; argument
if we want only one, which would run the proper query:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;fastapi&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;HTTPException&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="nd"&gt;&amp;#64;app&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/person&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;list_people&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fetch_people&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="nd"&gt;&amp;#64;app&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/person/&lt;/span&gt;&lt;span class="si"&gt;{id}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;get_person&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UUID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;people&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;fetch_people&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;people&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="n"&gt;HTTPException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;404&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;fetch_people&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UUID&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# We will implement this later&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="o"&gt;...&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The views are pretty much just acting as a bridge between the environment (the
FastAPI runtime and how it's configured) and the &amp;quot;business logic&amp;quot; (our
database queries in this case, but there could be much more there). Note that
the views use the &lt;tt class="docutils literal"&gt;Conn&lt;/tt&gt; alias, so that FastAPI will inject the connection,
whereas &lt;tt class="docutils literal"&gt;fetch_people()&lt;/tt&gt; just asks for a plain &lt;tt class="docutils literal"&gt;psycopg.Connection&lt;/tt&gt;: it
doesn't know or care where the connection comes from. This separation also
makes the code testable: you could write unit tests
exercising &lt;tt class="docutils literal"&gt;fetch_people()&lt;/tt&gt; in isolation from FastAPI, for example using
&lt;a class="reference external" href="https://docs.pytest.org/en/stable/explanation/fixtures.html"&gt;PyTest fixtures&lt;/a&gt; to create a database connection (your test database is
likely reached differently from a running server's) and test your business
code independently from whether it is used in FastAPI or in a different
environment.&lt;/p&gt;
&lt;p&gt;Let's get down to the proper database querying: the way such code might have
been written in &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; could be something like:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;fetch_people&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UUID&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;SELECT id, name, date_of_birth FROM person&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot; WHERE id = &lt;/span&gt;&lt;span class="si"&gt;%(id)s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="n"&gt;people&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;rec_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;date_of_birth&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;people&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;rec_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;date_of_birth&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;date_of_birth&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;people&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This is a pretty verbose way to transform database records to instances of the
&lt;tt class="docutils literal"&gt;Person&lt;/tt&gt; class and classes like &lt;tt class="docutils literal"&gt;DictCursor&lt;/tt&gt; and similar only help
marginally. Psycopg 3 improved records conversion with the introduction of the
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/rows.html"&gt;row factories&lt;/a&gt;,
which allow creating a direct bridge from the query result to the final
objects.&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg.rows&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;class_row&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;fetch_people&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UUID&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;SELECT id, name, date_of_birth FROM person&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot; WHERE id = &lt;/span&gt;&lt;span class="si"&gt;%(id)s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;class_row&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Note that Psycopg 3 makes ample use of type annotations and &lt;a class="reference external" href="https://typing.python.org/en/latest/reference/generics.html"&gt;generic types&lt;/a&gt;:
if you use a static checker such as &lt;a class="reference external" href="https://mypy-lang.org/"&gt;Mypy&lt;/a&gt;, it will be able to infer that
&lt;tt class="docutils literal"&gt;cur.fetchall()&lt;/tt&gt; returns a list of &lt;tt class="docutils literal"&gt;Person&lt;/tt&gt; and not a list of tuples,
allowing you to detect a whole class of errors before even running a test.&lt;/p&gt;
&lt;p&gt;If you are using Python 3.14, the use of &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/tstrings.html"&gt;template strings&lt;/a&gt; makes writing
dynamic queries even simpler:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;fetch_people&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UUID&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;t&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT id, name, date_of_birth FROM person&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="sa"&gt;t&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot; WHERE id = &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;class_row&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Person&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Either implementation completes the code: the API can be tested using the
Swagger interface or &lt;tt class="docutils literal"&gt;curl&lt;/tt&gt; on the command line:&lt;/p&gt;
&lt;pre class="code sh literal-block"&gt;
$&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/person&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;a0aaa2d4-9fa4-4baf-83ed-a3fe35f051d3&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Alice&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;date_of_birth&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1970-01-01&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;,&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;706c17ea-2214-4482-810f-e6ed1cfab694&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Bob&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;date_of_birth&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;null&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/person/a0aaa2d4-9fa4-4baf-83ed-a3fe35f051d3&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;a0aaa2d4-9fa4-4baf-83ed-a3fe35f051d3&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Alice&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;date_of_birth&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1970-01-01&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# The status code here is 404, not 200
&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/person/00000000-0000-0000-0000-000000000000&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;detail&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Not Found&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="pools-and-applications"&gt;
&lt;h2&gt;Pools and applications&lt;/h2&gt;
&lt;p&gt;Creating a new connection at every request is a big performance hit:
connection creation is a relatively slow operation. For this reason Psycopg
makes a &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/pool.html#connection-pools"&gt;connection pool&lt;/a&gt; available, with a set of connections ready to be
used. Remember that the Psycopg pool is distributed as a separate package,
therefore you might want to have something like
&lt;tt class="docutils literal"&gt;psycopg[binary,pool]&lt;/tt&gt; in your dependencies.&lt;/p&gt;
&lt;p&gt;In your program you will want to create a single instance of the pool and have
the views using it. Creating a pool as a global variable works, but a much
more refined way to manage the lifetime of the pool is to tie it to the life
cycle of the application: FastAPI will create the pool at application start
and dispose of it cleanly on program exit. This avoids problems such as
partially initialized code or side effects at import time.&lt;/p&gt;
&lt;p&gt;What happens at startup and shutdown is controlled by a &lt;tt class="docutils literal"&gt;lifespan&lt;/tt&gt; function
passed as argument to the &lt;tt class="docutils literal"&gt;FastAPI&lt;/tt&gt; object. This too uses the single
&lt;tt class="docutils literal"&gt;yield&lt;/tt&gt; trick to allow suspending the execution and restoring it (to run the
cleanup part) when the end of the application lifetime is reached. We make use
of this function to create a pool adequately configured:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg_pool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ConnectionPool&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;lifespan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;ConnectionPool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;yield&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# Add the 'lifespan' argument to the existing app&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;My People&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lifespan&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;lifespan&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The pool creates and maintains a configured number of connections and makes
them available to the application code as soon as one is requested. We only
need to change the body of the dependency injection to obtain a connection
from the pool instead of creating a new one, no change to the views. The
FastAPI &lt;tt class="docutils literal"&gt;Request&lt;/tt&gt; object (which in turn is another dependency injection)
lets us get the object representing the running application so that we can
obtain the pool instance from it:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;fastapi&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Request&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;get_connection&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connection&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="conclusions"&gt;
&lt;h2&gt;Conclusions&lt;/h2&gt;
&lt;p&gt;We have taken a quick look at some FastAPI features (views and models
definition, dependency injections, application lifetime), some Psycopg
features (&lt;tt class="docutils literal"&gt;PG*&lt;/tt&gt; env vars, row factories, connection pool), and how they can
interact together, with the help of Pydantic as an expressive objects model.&lt;/p&gt;
&lt;p&gt;The complete and runnable code of the example can be downloaded &lt;a class="reference external" href="https://gist.github.com/dvarrazzo/0edf2b659e59a8a8e2053e0c633087a5"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 3.3 released</title><link href="https://www.psycopg.org/articles/2025/12/01/psycopg-33-released/" rel="alternate"/><updated>2025-12-01T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:2593b72d-e32e-37db-9fa0-57834bcf9188</id><content type="html">&lt;p&gt;We have released Psycopg 3.3 - and you should be excited about it!&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="template-string-queries"&gt;
&lt;h2&gt;Template string queries&lt;/h2&gt;
&lt;p&gt;This version lets you take advantage of one of the biggest innovation in
Python 3.14: the &lt;a class="reference external" href="https://docs.python.org/3.14/whatsnew/3.14.html#pep-750-template-string-literals"&gt;template strings&lt;/a&gt;, which allow you to write
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/tstrings.html"&gt;expressive and safe queries&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;How does it look? Something like:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;fetch_person&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="c1"&gt;# 'name' will be handled safely: as a server-side parameter or&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# correctly quoted and escaped if client-side binding is required&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;t&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT * FROM people WHERE name = &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The syntax is the same as that of &lt;a class="reference external" href="https://docs.python.org/3/tutorial/inputoutput.html#formatted-string-literals"&gt;f-strings&lt;/a&gt;, introduced back in the
venerable Python 3.6 (perhaps the feature that finally ended Python 2?),
but now paired with the safety and adaptation flexibility of Psycopg 3.
Template strings also help you generate dynamic SQL statements much more
succinctly than with the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/sql.html"&gt;psycopg.sql&lt;/a&gt; module:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;delete_something&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="c1"&gt;# Mixing client-side query composition with server-side parameters binding&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;t&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;DELETE FROM &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;table_name&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;i&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; WHERE name = &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="c1"&gt;# Composing non-parametric statements entirely client-side&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;t&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NOTIFY &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;table_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'.deleted'&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;i&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;l&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Check out the complete &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/tstrings.html"&gt;t-string support documentation&lt;/a&gt; for inspiration!&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="more-flexible-composite-adaptation"&gt;
&lt;h2&gt;More flexible composite adaptation&lt;/h2&gt;
&lt;p&gt;Previously, it was only possible to &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/pgtypes.html#adapt-composite"&gt;adapt PostgreSQL composites&lt;/a&gt; to Python
sequence types with a strict 1:1 mapping to the fields of the database type.&lt;/p&gt;
&lt;p&gt;We have now gained extra flexibility: we can customize both how to create
generic Python objects, for example ones only taking keyword arguments, and
how to extract a sequence of fields from the attributes of non-sequence
objects... Dataclasses anyone?&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;dataclasses&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg.types.composite&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;CompositeInfo&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;register_composite&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="nd"&gt;&amp;#64;dataclass&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;MiniPerson&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;age&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="nd"&gt;&amp;#64;classmethod&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;from_db&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;cls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;cls&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;seq&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;seq&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;to_db&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;age&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;CREATE TYPE mini_person AS (name text, age int)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;CompositeInfo&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;mini_person&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;register_composite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MiniPerson&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;make_object&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MiniPerson&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_db&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;make_sequence&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MiniPerson&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to_db&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT ('John', 33)::mini_person&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;# MiniPerson(age=33, name='John', height=None)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;SELECT (&lt;/span&gt;&lt;span class="si"&gt;%(person)s&lt;/span&gt;&lt;span class="s2"&gt;).name || ' next year will be ' || (&lt;/span&gt;&lt;span class="si"&gt;%(person)s&lt;/span&gt;&lt;span class="s2"&gt;).age + 1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;person&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;MiniPerson&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;John&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;33&lt;/span&gt;&lt;span class="p"&gt;)},&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;# 'John next year will be 34'&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="solving-the-fetchone-annoyance-with-type-checkers"&gt;
&lt;h2&gt;Solving the &lt;tt class="docutils literal"&gt;fetchone()&lt;/tt&gt; annoyance with type checkers&lt;/h2&gt;
&lt;p&gt;If you use Mypy or other type checkers with Psycopg, you've probably seen
false positives when calling &lt;tt class="docutils literal"&gt;fetchone()&lt;/tt&gt;. Even if you are 100% certain
your query will return a row, &lt;tt class="docutils literal"&gt;fetchone()&lt;/tt&gt; is annotated as possibly
returning &lt;tt class="docutils literal"&gt;None&lt;/tt&gt; - so type checkers complain about patterns like:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT count(*) FROM my_table&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# Always returns exactly one value&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;   &lt;span class="c1"&gt;# Error: value of type &amp;quot;tuple | None&amp;quot; is not indexable&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;In Psycopg 3.3, the cursor has become an &lt;a class="reference external" href="https://docs.python.org/3/glossary.html#term-iterator"&gt;iterator&lt;/a&gt;, whereas it was
previously only an &lt;a class="reference external" href="https://docs.python.org/3/glossary.html#term-iterable"&gt;iterable&lt;/a&gt;. The distinction is subtle but meaningful: an
iterator holds its own iteration state and does not need to create a new
object for each pass.&lt;/p&gt;
&lt;p&gt;More importantly, this change means you can use &lt;a class="reference external" href="https://docs.python.org/3/library/functions.html#next"&gt;next()&lt;/a&gt; or &lt;a class="reference external" href="https://docs.python.org/3/library/functions.html#anext"&gt;anext()&lt;/a&gt; to retrieve
a record - and these functions never return &lt;tt class="docutils literal"&gt;None&lt;/tt&gt;. This makes Mypy happy,
and probably you too:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT count(*) FROM my_table&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="improvements-to-the-connection-pools"&gt;
&lt;h2&gt;Improvements to the connection pools&lt;/h2&gt;
&lt;p&gt;A connection pool’s parameters can now be changed dynamically - useful for
example to support short-lived secret tokens as passwords, as requested
by some cloud database providers.&lt;/p&gt;
&lt;p&gt;A useful &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/pool.html#psycopg_pool.ConnectionPool.drain"&gt;drain()&lt;/a&gt; method is now available to re-create all connections in
a pool. This is helpful, for instance, when the database needs to be
introspected to find the OIDs of extension types to register: without draining
the pool the connections already in the pool would remain stale after the
adapters have been configured.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="and-more"&gt;
&lt;h2&gt;...And more!&lt;/h2&gt;
&lt;p&gt;Other improvements include greater flexibility when navigating results after
a &lt;tt class="docutils literal"&gt;fetchmany()&lt;/tt&gt; call or after statements returning multiple result sets,
the ability to reconfigure loaders after a query has run, and many other
assorted enhancements. You can find the full list in the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/news.html#psycopg-3-3-0"&gt;psycopg release
notes&lt;/a&gt; and the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/news_pool.html#psycopg-pool-3-3-0"&gt;pool release notes&lt;/a&gt;!&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="your-help-is-welcome"&gt;
&lt;h2&gt;Your help is welcome&lt;/h2&gt;
&lt;p&gt;Psycopg is the de-facto standard for communication between Python and
PostgreSQL - two major components powering countless businesses and
mission-critical infrastructure. Maintaining such an important library to the
highest standards of reliability, performance and security requires a lot of
care and ongoing work.&lt;/p&gt;
&lt;p&gt;If you use Python and PostgreSQL and want to help ensure that the interface
between them remains robust and continues to improve, supporting new language
and database features, please consider &lt;a class="reference external" href="https://github.com/sponsors/psycopg/"&gt;supporting the project&lt;/a&gt; 💛&lt;/p&gt;
&lt;p&gt;Thank you very much, and happy hacking!&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>Automatic async to sync code conversion</title><link href="https://www.psycopg.org/articles/2024/09/23/async-to-sync/" rel="alternate"/><updated>2024-09-23T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:765ce6dc-afc5-34ff-bf78-d5f78cc8aed5</id><content type="html">&lt;p&gt;Psycopg 3 provides both a sync and an async Python interface: for each object
used to perform I/O operations, such as &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/connections.html#psycopg.Connection"&gt;Connection&lt;/a&gt;, &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor"&gt;Cursor&lt;/a&gt;, there is an
async counterpart: &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/connections.html#psycopg.AsyncConnection"&gt;AsyncConnection&lt;/a&gt;, &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.AsyncCursor"&gt;AsyncCursor&lt;/a&gt;, with an intuitive
interface: just add the right &lt;tt class="docutils literal"&gt;async&lt;/tt&gt; or &lt;tt class="docutils literal"&gt;await&lt;/tt&gt; keyword where needed:&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# Familiar sync code&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select now()&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# Similar async code&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;aconn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AsyncConnection&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;acur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;aconn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select now()&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;acur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;())[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The decision to provide both sync and async code &lt;a class="reference external" href="https://www.varrazzo.com/blog/2020/03/26/psycopg3-first-report/"&gt;was made early in the
development of Psycopg 3&lt;/a&gt; and most of the internal code is written in a way
to be compatible with both sync and async code, in order to keep code
duplication to a minimum. This was achieved by making all the libpq
communication async, and writing the network code as generators,
&lt;tt class="docutils literal"&gt;yield&lt;/tt&gt;ing at the moment they need to wait, isolating the differences in
the sync/async wait policy all in &lt;tt class="docutils literal"&gt;wait()&lt;/tt&gt; functions.&lt;/p&gt;
&lt;p&gt;This helped to minimise the async/sync differences in the code related to the
communication between PostgreSQL and Psycopg. However, the interface between
Psycopg and the Python user is still a lot to maintain, and consists of a lot
of code that is very similar, almost duplicated, between the sync and async
sided. Apart from the obvious &lt;tt class="docutils literal"&gt;async&lt;/tt&gt;/&lt;tt class="docutils literal"&gt;await&lt;/tt&gt; keywords, there would be
subtle implementation differences, for example:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;using &lt;tt class="docutils literal"&gt;asyncio&lt;/tt&gt; functions instead of blocking counterparts, for instance
&lt;tt class="docutils literal"&gt;await asyncio.sleep()&lt;/tt&gt; instead of &lt;tt class="docutils literal"&gt;time.sleep()&lt;/tt&gt;;&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;asyncio.create_task(f(arg1, arg2))&lt;/tt&gt; is similar to &lt;tt class="docutils literal"&gt;thread.Thread(f,
(arg1, &lt;span class="pre"&gt;arg2)).start()&lt;/span&gt;&lt;/tt&gt;;&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;threading.Event&lt;/tt&gt; has a &lt;tt class="docutils literal"&gt;asyncio.Event&lt;/tt&gt; counterpart whose &lt;tt class="docutils literal"&gt;lock()&lt;/tt&gt;
method doesn't have a &lt;tt class="docutils literal"&gt;timeout&lt;/tt&gt;, parameter, so &lt;tt class="docutils literal"&gt;event.wait(timeout=10)&lt;/tt&gt;
needs to be rewritten as &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;asyncio.wait_for(event.wait(),&lt;/span&gt; timeout=10)&lt;/tt&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Up until Psycopg 3.1, the two variants of each object were kept in sync
manually. Every time changes were made on the sync side, they had to be ported
to the async side, with cumbersome and noisy diffs, with subtle differences
being introduced from time to time. Even the tests were pretty much duplicated
(with some sync tests being accidentally lost on the async side, or vice
versa). It seemed like a situation that could have been improved.&lt;/p&gt;
&lt;div class="section" id="this-is-so-boring-that"&gt;
&lt;h2&gt;This is so boring that...&lt;/h2&gt;
&lt;p&gt;...a computer should do it for me instead.&lt;/p&gt;
&lt;p&gt;Writing the async side starting from the sync side? Actually, the opposite. It
is obvious that the async side has more information than the sync side (every
method definition and call clearly indicates whether it will block or not) and
most of the differences are minimal and repetitive. What we want then is &lt;em&gt;a
script that takes asyncio-based source code as input and outputs
equivalent sync code&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;This article describes what we did to implement such a script and how we used
it for the initial transformation (replacing manually written sync code with
auto-generated code without loss of quality) and how we are currently using it
to maintain the Psycopg 3 codebase.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="abstract-syntax-tree"&gt;
&lt;h2&gt;Abstract Syntax Tree&lt;/h2&gt;
&lt;p&gt;You would be tempted to write a bunch of regular expressions to just scrub
away every &lt;tt class="docutils literal"&gt;async&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;await&lt;/tt&gt; keyword found, but the source code is
probably the wrong level to attack the problem: Python knows how to parse
Python itself well and can allow us to reason at a higher level.&lt;/p&gt;
&lt;p&gt;A better tool to work with is the &lt;a class="reference external" href="https://en.wikipedia.org/wiki/Abstract_syntax_tree"&gt;Abstract Syntax Tree&lt;/a&gt; (AST): an in-memory
representation of the code obtained after parsing. At this level we manipulate
objects that represent &amp;quot;the for loop&amp;quot;, or &amp;quot;the function call&amp;quot;, and we are not
fooled by unexpected spaces, extra brackets, comments, literal strings, and
other traps.&lt;/p&gt;
&lt;p&gt;The &lt;a class="reference external" href="https://docs.python.org/3/library/ast.html"&gt;Python 'ast' module&lt;/a&gt; is the obvious starting point: if you have a bit
of source code such as:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;asyncio&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;async_square&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# Squares are slow&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;you can pass it to the module to see the AST tree that represents it:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
$ python -m ast ast1.py
&lt;strong&gt;Module&lt;/strong&gt;(
   body=[
      &lt;strong&gt;Import&lt;/strong&gt;(
         names=[
            alias(name='asyncio')]),
      &lt;strong&gt;AsyncFunctionDef&lt;/strong&gt;(
         name='async_square',
         args=arguments(
            args=[
               arg(arg='n')],
            defaults=[]),
         body=[
            Expr(
               value=&lt;strong&gt;Await&lt;/strong&gt;(
                  value=&lt;strong&gt;Call&lt;/strong&gt;(
                     func=Attribute(
                        value=Name(id='asyncio'),
                        attr='sleep'),
                     args=[
                        Constant(value=1)]))),
            &lt;strong&gt;Return&lt;/strong&gt;(
               value=BinOp(
                  left=Name(id='n'),
                  op=Mult(),
                  right=Name(id='n')))],
         decorator_list=[],
         returns=Name(id='float'))])
&lt;/pre&gt;
&lt;p&gt;You can see, highlighted, the nodes in the tree representing the main
statements in the code: the tree represents a &lt;em&gt;module&lt;/em&gt;, whose body contains two
&lt;em&gt;statements&lt;/em&gt; - an &lt;tt class="docutils literal"&gt;import&lt;/tt&gt; and an &lt;tt class="docutils literal"&gt;async def&lt;/tt&gt; - with the function body
defining an &lt;tt class="docutils literal"&gt;await&lt;/tt&gt; call and a &lt;tt class="docutils literal"&gt;return&lt;/tt&gt; statement.&lt;/p&gt;
&lt;p&gt;The same &lt;tt class="docutils literal"&gt;ast&lt;/tt&gt; module can perform the reverse transformation, converting an
AST tree back to source:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
$ python -c &amp;quot;import ast; print(ast.unparse(ast.parse(open('ast1.py').read())))&amp;quot;
import asyncio

async def square(n: float) -&amp;gt; float:
    await asyncio.sleep(1)
    return n * n
&lt;/pre&gt;
&lt;p&gt;As you can see, the transformation back to code is unfortunately not a perfect
reconstruction of the original code, it is only &lt;em&gt;equivalent&lt;/em&gt;, with missing
comments and different spacing. This is because the syntax tree is &lt;em&gt;abstract&lt;/em&gt;
and whitespaces and comments don't affect it. If you wanted to take those
details into account you would need a &lt;em&gt;concrete&lt;/em&gt; syntax tree (&lt;a class="reference external" href="https://pypi.org/project/libcst/"&gt;something like
that exists&lt;/a&gt;, but I haven't played with it).&lt;/p&gt;
&lt;p&gt;Changing whitespaces is not a problem, but losing comments can be, especially
when they are used to control linters (such as Flake8's &lt;tt class="docutils literal"&gt;noqa&lt;/tt&gt; or Mypy's
&lt;tt class="docutils literal"&gt;type: ignore&lt;/tt&gt;), or simply when you happen to be a human being and want to
read the source code. Fortunately there is a simple wrapper module,
&lt;a class="reference external" href="https://pypi.org/project/ast-comments/"&gt;ast-comments&lt;/a&gt;, which does exactly what it says on the tin: it introduces
&lt;tt class="docutils literal"&gt;Comment&lt;/tt&gt; nodes as part of an AST. Playing around with it, it turned out to
be a good compromise between an abstract and a concrete syntax tree, after
some taming of the comments placement.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="du-ast-mich"&gt;
&lt;h2&gt;Du AST Mich&lt;/h2&gt;
&lt;p&gt;To perform the code transformation, we will walk over the abstract
syntax tree and we will perform some operation to generate a different tree of
our liking. Typically, this type of operation is performed using an
implementation of the &lt;a class="reference external" href="https://en.wikipedia.org/wiki/Visitor_pattern"&gt;visitor pattern&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;This pattern can be incredibly useful whenever you need to perform operations
on data structures composed of heterogeneous nodes (I've seen it in
applications ranging from converting UML representations to code, converting
markup language to HTML, converting Kubernetes manifests to Helm charts,
converting annotated lyrics files to Ukulele tab sheets...); unfortunately
many of the descriptions of the pattern you can find online fail to make its
brilliance immediately apparent (the Wikipedia page is pretty bad at it)
because they have historically focused on solving the &lt;em&gt;double-dispatch&lt;/em&gt;
problem in static languages like as C++ or Java (which is trivial in a dynamic
language like Python) rather than focusing on the &lt;strong&gt;awesome&lt;/strong&gt; things you
can do with it.&lt;/p&gt;
&lt;p&gt;In a nutshell, you will have an object that traverses an input data structure,
element by element, building an output structure in the process, allowing you
to run different code and to perform different manipulations depending on the
type of element being traversed.&lt;/p&gt;
&lt;p&gt;In our case, both the input and the output are AST trees, which will happen to
be very similar to each other (since we are just trying to translate some
subtle differences from one Python module to another): for many nodes, the
visitor will just output a copy of it (for example, the &lt;tt class="docutils literal"&gt;return&lt;/tt&gt; statement
in the above example is unchanged). But, if we see a pattern of interest, we
can tell our visitor to produce a different node.&lt;/p&gt;
&lt;p&gt;The &lt;tt class="docutils literal"&gt;ast&lt;/tt&gt; module provides a base class &lt;a class="reference external" href="https://docs.python.org/3/library/ast.html#ast.NodeTransformer"&gt;ast.NodeTransformer&lt;/a&gt; which
implements the node traversal and tree production parts. By itself it
doesn't perform any operations on the nodes, so it just produces a copy of the
input tree. However, by subclassing the class and adding visit methods, you
can implement node-specific transformations.&lt;/p&gt;
&lt;p&gt;With the AST node transformer, the method called is based on the name of the
node being visited; for example, if you add a method called &lt;tt class="docutils literal"&gt;visit_Import&lt;/tt&gt;
to your subclass, the visitor will call it whenever it traverses an &lt;tt class="docutils literal"&gt;Import&lt;/tt&gt;
node, giving you the chance to manipulate an &lt;tt class="docutils literal"&gt;import&lt;/tt&gt; statement. You can
then decide whether you want to change some of the details of the node (drop
some imports, change some names), or replace the node with something completely
different (such as replacing an async function definition with a sync one).&lt;/p&gt;
&lt;p&gt;Let's say that we want to produce a sync version of the above script: the
differences should be the following:&lt;/p&gt;
&lt;pre class="code diff literal-block"&gt;
&lt;span class="gu"&gt;&amp;#64;&amp;#64; -1,7 +1,7 &amp;#64;&amp;#64;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gd"&gt;-import asyncio&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+import time&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="gd"&gt;-async def async_square(n: float) -&amp;gt; float:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+def square(n: float) -&amp;gt; float:&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;    # Squares are slow&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gd"&gt;-    await asyncio.sleep(1)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+    time.sleep(1)&lt;/span&gt;&lt;span class="w"&gt;

 &lt;/span&gt;    return n * n
&lt;/pre&gt;
&lt;p&gt;In our toy example, we want to convert the &lt;tt class="docutils literal"&gt;asyncio&lt;/tt&gt; module into the &lt;tt class="docutils literal"&gt;time&lt;/tt&gt;
module (which is obviously not the right thing to do in the general case, but
let's keep the example simple). The following script implements the
transformation and prints the converted module:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;ast&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;MyTransformer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ast&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NodeTransformer&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;visit_Import&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;alias&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;names&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;alias&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;asyncio&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;                &lt;span class="n"&gt;alias&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;time&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ast1.py&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;tree&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ast&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;tree&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;MyTransformer&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;visit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tree&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ast&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;unparse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tree&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The script will print the new source, with an &lt;tt class="docutils literal"&gt;import time&lt;/tt&gt; replacing the
original &lt;tt class="docutils literal"&gt;import asyncio&lt;/tt&gt;.&lt;/p&gt;
&lt;p&gt;Changing the &lt;tt class="docutils literal"&gt;async&lt;/tt&gt; call is a bit trickier: we want to change the
highlighted parts of the original tree:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
&lt;strong&gt;value=Await&lt;/strong&gt;(   &amp;lt;&amp;lt; this node must be dropped, replaced by its &lt;strong&gt;value&lt;/strong&gt;
   value=Call(
      func=Attribute(
         value=Name(id='&lt;strong&gt;asyncio&lt;/strong&gt;'),  &amp;lt;&amp;lt; we want &lt;strong&gt;time&lt;/strong&gt; here
         attr='sleep'),
      args=[
         Constant(value=1)],
      keywords=[]))),
&lt;/pre&gt;
&lt;p&gt;Adding the following two methods to the above class will implement what has
been described.&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;visit_Await&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;new_node&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;  &lt;span class="c1"&gt;# drop the node, continue to operate on the value&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;visit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;new_node&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;new_node&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;visit_Call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;match&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;func&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;ast&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ast&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;asyncio&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sleep&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;func&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;time&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;To make sense of how these methods operate on their input nodes, and then to
implement your own transformations, you can always look at the output of
&lt;tt class="docutils literal"&gt;python &lt;span class="pre"&gt;-m&lt;/span&gt; ast&lt;/tt&gt; in order to see the attributes on each node and how they are
nested.&lt;/p&gt;
&lt;p&gt;The &lt;tt class="docutils literal"&gt;visit_Call&lt;/tt&gt; method shows how the &lt;a class="reference external" href="https://peps.python.org/pep-0636/"&gt;structural pattern matching&lt;/a&gt;
introduced in Python 3.10 comes in handy for this operation. The method is
called for each function call found in the input code; checking whether the
one just received requires any manipulation would have involved a cascade of
ifs (the value is a &lt;tt class="docutils literal"&gt;Name&lt;/tt&gt;, its id is &lt;tt class="docutils literal"&gt;asyncio&lt;/tt&gt;, the attr is &lt;tt class="docutils literal"&gt;sleep&lt;/tt&gt;...)
which becomes pretty ugly pretty quickly, whereas instead a &lt;tt class="docutils literal"&gt;match&lt;/tt&gt;
statement can describe a complex nested test very succinctly.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="problems-with-sleep"&gt;
&lt;h2&gt;Problems with &lt;tt class="docutils literal"&gt;sleep()&lt;/tt&gt;&lt;/h2&gt;
&lt;p&gt;Performing the transformation from &lt;tt class="docutils literal"&gt;asyncio.sleep&lt;/tt&gt; to &lt;tt class="docutils literal"&gt;time.sleep&lt;/tt&gt; for
real is much more complex than this. What if our source includes &lt;tt class="docutils literal"&gt;from asycio
import sleep, Event&lt;/tt&gt;? We would have to split the import into several parts:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;time&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sleep&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;threading&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Event&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;and the latter should be treated differently later because the two &lt;tt class="docutils literal"&gt;Event&lt;/tt&gt;
objects have a different &lt;tt class="docutils literal"&gt;wait()&lt;/tt&gt; signatures.&lt;/p&gt;
&lt;p&gt;To help with this operation, in Psycopg 3 we introduced &lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/d13137aacb82fed79459a9dd487846a2ec972571/psycopg/psycopg/_acompat.py"&gt;an internal
'_acompat' module&lt;/a&gt; (actually &lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/d13137aacb82fed79459a9dd487846a2ec972571/psycopg_pool/psycopg_pool/_acompat.py"&gt;two&lt;/a&gt;, because the pool is released
separately and uses different functions; actually &lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/d13137aacb82fed79459a9dd487846a2ec972571/tests/acompat.py"&gt;three&lt;/a&gt;, because the tests
also have their own...) to expose pairs of functions or objects that should be
used alternatively in sync or in async mode.&lt;/p&gt;
&lt;p&gt;For example we can solve the &lt;tt class="docutils literal"&gt;sleep()&lt;/tt&gt; problem with:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# module _acompat.py&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;time&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;asyncio&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;sleep&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sleep&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;asleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Coroutine&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Any&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Any&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;
    Equivalent to asyncio.sleep(), converted to time.sleep() by async_to_sync.
    &amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seconds&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Now it's easy to use &lt;tt class="docutils literal"&gt;from ._acompat import asleep; await asleep(1)&lt;/tt&gt; and do
some simple name substitutions in the AST: the resulting statement &lt;tt class="docutils literal"&gt;from
._acompat import sleep; sleep(1)&lt;/tt&gt; will work as expected.&lt;/p&gt;
&lt;p&gt;Other goodies we have implemented to help unify async and sync code are
&lt;tt class="docutils literal"&gt;aspawn&lt;/tt&gt;/&lt;tt class="docutils literal"&gt;spawn&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;agather&lt;/tt&gt;/&lt;tt class="docutils literal"&gt;gather&lt;/tt&gt; to unify threads and
asyncio tasks creation, &lt;tt class="docutils literal"&gt;alist()&lt;/tt&gt; to encapsulate &lt;tt class="docutils literal"&gt;[x for x in await
iterable]&lt;/tt&gt; in a way that can be easily converted to &lt;tt class="docutils literal"&gt;list(iterable)&lt;/tt&gt; and
many other helpers to smooth the transition.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="when-everything-else-fail"&gt;
&lt;h2&gt;When everything else fail&lt;/h2&gt;
&lt;p&gt;There may be parts of the codebase where the difference between sync and async
versions is too difficult to handle in a practical way, and is not worth to
put together a complex matching for a complex, one-off case. What we want is a
simple &amp;quot;if async, do this, else do that&amp;quot;.&lt;/p&gt;
&lt;p&gt;We have solved this problem by using a pattern like:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="c1"&gt;# ASYNC&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;foo&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;bar&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The AST with this code, including the comments, looks like:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
Module(
   body=[
      &lt;strong&gt;If(&lt;/strong&gt;
         &lt;strong&gt;test=Constant(value=True),&lt;/strong&gt;
         body=[
            &lt;strong&gt;Comment(value='# ASYNC', inline=True),&lt;/strong&gt;
            Expr(
               value=Call(
                  func=Name(id='foo'),
                  args=[],
                  keywords=[]))],
         orelse=[
            Expr(
               value=Call(
                  func=Name(id='bar'),
                  args=[],
                  keywords=[]))])],
   type_ignores=[])
&lt;/pre&gt;
&lt;p&gt;&lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/d13137aacb82fed79459a9dd487846a2ec972571/tools/async_to_sync.py#L253-L262"&gt;Our transformation&lt;/a&gt; will find the &lt;tt class="docutils literal"&gt;ASYNC&lt;/tt&gt; comment: in this case it will
simply discard the if side of the condition, as well as the &lt;tt class="docutils literal"&gt;if&lt;/tt&gt; itself, and
will leave only the &lt;tt class="docutils literal"&gt;else&lt;/tt&gt; branch in the sync code, allowing you to discard
unneeded imports or other code that would simply be invalid in the sync
context.&lt;/p&gt;
&lt;p&gt;This pattern is also efficient, because the Python compiler is able to
recognise that &lt;tt class="docutils literal"&gt;if True&lt;/tt&gt; will always take the first branch, so it will
discard the test and the code in the &lt;tt class="docutils literal"&gt;else&lt;/tt&gt; branch. The &lt;a class="reference external" href="https://docs.python.org/3/library/dis.html"&gt;dis&lt;/a&gt;(assembler)
module shows no jump and that no reference to the &lt;tt class="docutils literal"&gt;bar()&lt;/tt&gt; function call:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
$ python -m dis ast3.py
  1           0 NOP

  2           2 LOAD_NAME                0 (foo)
              4 CALL_FUNCTION            0
              6 POP_TOP
              8 LOAD_CONST               1 (None)
             10 RETURN_VALUE
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="conversion-methodology"&gt;
&lt;h2&gt;Conversion methodology&lt;/h2&gt;
&lt;p&gt;Once we have our conversion script, how do we use it to actually convert the
code base, making sure to not break it? The process, for us, was
iterative: going module by module and adding features to the script until
all the &amp;quot;duplicated&amp;quot; modules were complete.&lt;/p&gt;
&lt;p&gt;For each module to be converted, the procedure was roughly as follows.&lt;/p&gt;
&lt;p&gt;First step: refactoring the code with the intention of not changing any
behaviour, but of making the async module as similar as possible to the sync
module. This might have meant some code reorganisation, the renaming of some
variables, the swapping of some function definitions, the rediscovery of some
forgotten skeletons and a chance of giving them a proper burial.&lt;/p&gt;
&lt;p&gt;Often we would have implemented some non-I/O related helper function on the
sync side and imported it on the async side:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# connection.py&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="o"&gt;...&lt;/span&gt;  &lt;span class="c1"&gt;# hack hack&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;better_conninfo&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;better_conninfo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;connection_gen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bettern_conninfo&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# connection_async.py&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;.connection&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;connect_async&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;better_conninfo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;aconn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;async_wait&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;connection_gen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bettern_conninfo&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;aconn&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;In this case we would have moved the shared functionality in a separate
internal module and imported the function on both the sides:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# _connection.py&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="o"&gt;...&lt;/span&gt;  &lt;span class="c1"&gt;# hack hack&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;better_conninfo&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# connection.py&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;._connection&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;better_conninfo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;connection_gen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bettern_conninfo&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# connection_async.py&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;._connection&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;connect_async&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;better_conninfo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;clean_up_conninfo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conninfo&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;aconn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;async_wait&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;connection_gen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bettern_conninfo&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;aconn&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Now that the two modules are more similar we can run the test suite to verify
that the library still works and can commit the current state to git.&lt;/p&gt;
&lt;p&gt;Second step: run an async -&amp;gt; sync conversion with the current version of the
script. Even running a no-op script is useful: it produces changes that can be
easily seen with &lt;tt class="docutils literal"&gt;git diff&lt;/tt&gt;, suggesting which conversion feature is missing,
or what further cleanup we could have done in the code to make the sync and
async flavours more similar.&lt;/p&gt;
&lt;p&gt;For example, a no-op script that just copies the async side to the sync
side, would show up in &lt;tt class="docutils literal"&gt;git diff&lt;/tt&gt; as:&lt;/p&gt;
&lt;pre class="code diff literal-block"&gt;
&lt;span class="gu"&gt;&amp;#64;&amp;#64; -1,6 +1,6 &amp;#64;&amp;#64;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;from ._connection import clean_up_conninfo&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="gd"&gt;-def connect(conninfo):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+async def connect_async(conninfo):&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;    better_conninfo = clean_up_conninfo(conninfo)&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gd"&gt;-    conn = wait(connection_gen(bettern_conninfo))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gd"&gt;-    return conn&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+    aconn = await async_wait(connection_gen(bettern_conninfo))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+    return aconn&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The first feature to add to the conversion script is to remove the &lt;tt class="docutils literal"&gt;async&lt;/tt&gt;
and &lt;tt class="docutils literal"&gt;await&lt;/tt&gt; keywords. Run the conversion and diff again and you will see:&lt;/p&gt;
&lt;pre class="code diff literal-block"&gt;
&lt;span class="gu"&gt;&amp;#64;&amp;#64; -1,6 +1,6 &amp;#64;&amp;#64;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;from ._connection import clean_up_conninfo&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="gd"&gt;-def connect(conninfo):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+def connect_async(conninfo):&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;    better_conninfo = clean_up_conninfo(conninfo)&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gd"&gt;-    conn = wait(connection_gen(bettern_conninfo))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gd"&gt;-    return conn&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+    aconn = async_wait(connection_gen(bettern_conninfo))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+    return aconn&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The next step is some renaming. If &lt;tt class="docutils literal"&gt;connect()&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;connect_async()&lt;/tt&gt;
are public functions we don't want to change their names. The script should
have a name mapping function suggesting to convert:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;connect_async&lt;/tt&gt; -&amp;gt; &lt;tt class="docutils literal"&gt;connect&lt;/tt&gt;&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;wait_async&lt;/tt&gt; -&amp;gt; &lt;tt class="docutils literal"&gt;wait&lt;/tt&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Implementing this renaming in the AST we would bring us to the diff:&lt;/p&gt;
&lt;pre class="code diff literal-block"&gt;
&lt;span class="gu"&gt;&amp;#64;&amp;#64; -2,5 +2,5 &amp;#64;&amp;#64;&lt;/span&gt;&lt;span class="w"&gt;

 &lt;/span&gt;def connect(conninfo):&lt;span class="w"&gt;
 &lt;/span&gt;    better_conninfo = clean_up_conninfo(conninfo)&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gd"&gt;-    conn = wait(libpq.connect_async())&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gd"&gt;-    return conn&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+    aconn = async(libpq.connect_async())&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gi"&gt;+    return aconn&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;We are getting there. This remaining &lt;tt class="docutils literal"&gt;aconnn&lt;/tt&gt;/&lt;tt class="docutils literal"&gt;conn&lt;/tt&gt; is actually a
gratuitous difference: we can change the async side and call the local
variable &lt;tt class="docutils literal"&gt;conn&lt;/tt&gt; without losing readability and obviously without changing
any behaviour.&lt;/p&gt;
&lt;p&gt;Committing the change on the async side and re-running the conversion would
show no more difference on the sync side. At this point we can commit the
whole project (any remaining but acceptable change on the sync side, the new
features added to the conversion script, new entries in the renaming
mapping...), run the tests to verify that no regression has been introduced,
and move on to the next module.&lt;/p&gt;
&lt;p&gt;This operation, in Psycopg 3, started at commit &lt;a class="reference external" href="https://github.com/psycopg/psycopg/commit/765f663f171bf5d5e4862d5c4a5d572b7e3227d8"&gt;765f663f&lt;/a&gt; and can be seen in
the git history as a parallel branch that was eventually merged in &lt;a class="reference external" href="https://github.com/psycopg/psycopg/commit/8bb0f9bfef945861e8f671fba9073b3fae45c67f"&gt;8bb0f9bf&lt;/a&gt;.
The &lt;tt class="docutils literal"&gt;diff &lt;span class="pre"&gt;--stat&lt;/span&gt;&lt;/tt&gt; shows a whopping:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
99 files changed, 9697 insertions(+), 8486 deletions(-)
&lt;/pre&gt;
&lt;p&gt;which is obviously a monster changeset, but mostly consists of incremental
refactorings, conversions, finding new ways to minimise differences. It could
be an interesting ride if you have a project where you need to introduce
a similar automatic conversion.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="the-final-result"&gt;
&lt;h2&gt;The final result&lt;/h2&gt;
&lt;p&gt;Here is the &lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/3.2.0/tools/async_to_sync.py"&gt;Psycopg 3 async to sync conversion script&lt;/a&gt; (as of the &lt;a class="reference external" href="https://www.psycopg.org/articles/2024/06/30/psycopg-32-released/"&gt;Psycopg
3.2 release&lt;/a&gt;). At the time of writing, It processes 27 files and
automatically generates about the 25% of the codebase. Some of the features it
boasts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p class="first"&gt;the AST transformations described above, including tricks like recursion
into strings containing code to be transformed, such as Mypy annotations
expressed as strings, adjusting the output and the comments to make the
resulting unparsed code almost as good as the handwritten side;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;it inserts non-essential whitespace, and runs &lt;a class="reference external" href="https://black.readthedocs.io/"&gt;black&lt;/a&gt; on the output, in
order to make the resulting code as uniform as possible to the original and
as good for humans to work with (to read, debug, diff, etc);&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;since different Python versions may generate different ASTs and different
output code, it can run in a Docker container, whose image is created on the
fly using as base the Python image of the reference version;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;it adds a useful disclaimer to the top of the file:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# WARNING: this file is auto-generated by 'async_to_sync.py'&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;# from the original file 'connection_async.py'&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;# DO NOT CHANGE! Change the original file instead.&lt;/span&gt;
&lt;/pre&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;it has a &amp;quot;check&amp;quot; mode that runs in Github Action upon every commit, as part
of the lint step, and will fail if it finds any files to convert that haven't
been committed;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;the check mode has its own check: if any script containing the above disclaimer
is not included in the list of files to be converted, it will throw an
error (because a converted file has not been added to the automatic
conversion list);&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;the check of the check also has its own check! If no file with the
disclaimer is found then it means that something is wrong... Maybe the
disclaimer has been rewritten and the check doesn't work anymore;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;it can run in parallel and only on the files that have changed. Almost as
good as &lt;tt class="docutils literal"&gt;make&lt;/tt&gt; (but for certain tasks it is useful to have all the input
files at once, therefore, &amp;quot;better than &lt;tt class="docutils literal"&gt;make&lt;/tt&gt;&amp;quot;).&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The code is specific to the Psycopg 3 codebase and formatting style, so it's
probably not ready to be used as it is in other projects. But it is probably a
good starting point to to your own conversion: change the list of files to
process, the name mapping, and you should be good to start.&lt;/p&gt;
&lt;p&gt;Hope this helps. Happy hacking!&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 3.2 released</title><link href="https://www.psycopg.org/articles/2024/06/30/psycopg-32-released/" rel="alternate"/><updated>2024-06-30T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:31cb8a57-f930-3fb0-82c6-4067eb76bd76</id><content type="html">&lt;p&gt;It was quite the ride! But we made it!&lt;/p&gt;
&lt;p&gt;After almost two years, 846 commits, more than 700 new tests, more than 20000
changes in 310 files (I didn't even realise that there were 310 files in this
project...) Psycopg 3.2 has been released!&lt;/p&gt;
&lt;p&gt;This release brings a few new feature and hopefully no meaningful non-backward
compatible change. The whole list of changes is available &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/news.html#psycopg-3-2"&gt;in the changelog&lt;/a&gt;; these are
some of the major points explained.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="numpy-scalars-support"&gt;
&lt;h2&gt;Numpy scalars support&lt;/h2&gt;
&lt;p&gt;In many scientific applications, &lt;a class="reference external" href="https://numpy.org/doc/stable/reference/arrays.scalars.html#built-in-scalar-types"&gt;Numpy scalars&lt;/a&gt;
are widely used, either by themselves or in conjunction with regular Python
values. However there was no support for storing them to the database and a
conversion to normal Python values was necessary. Starting from Psycopg 3.2
storing Numpy scalars is automatic and the operation efficient.&lt;/p&gt;
&lt;p&gt;A natural extension would be to convert between Numpy and PostgreSQL arrays
too. However there hasn't been much demand for the feature, therefore it's
currently &lt;a class="reference external" href="https://github.com/psycopg/psycopg/issues/336"&gt;on the back burner&lt;/a&gt; but can be implemented if
there is demand.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="postgresql-parameters"&gt;
&lt;h2&gt;PostgreSQL parameters&lt;/h2&gt;
&lt;p&gt;Psycopg uses placeholders such as &lt;tt class="docutils literal"&gt;%s&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;%(name)s&lt;/tt&gt; to &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/params.html"&gt;pass values to
queries&lt;/a&gt;. These
formats are familiar to Python developers, but they are quite foreign in
PostgreSQL environment, because, natively, &lt;a class="reference external" href="https://www.postgresql.org/docs/current/libpq-exec.html#LIBPQ-PQEXECPARAMS"&gt;PostgreSQL uses a number-based
placeholder format&lt;/a&gt;
(such as &lt;tt class="docutils literal"&gt;$1&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;$2&lt;/tt&gt;...) Psycopg, internally, converts the first format
into the second.&lt;/p&gt;
&lt;p&gt;It is now possible to execute queries using the PostgreSQL format by using the
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/cursors.html#raw-query-cursors"&gt;raw query cursors&lt;/a&gt;,
which should feel more familiar to PostgreSQL developers and maybe lower the
barrier to convert programs using large bodies of native queries to Python
(the PostgreSQL test suite, maybe?)&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RawCursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT ($1 + $2) * $1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;,)&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="scalar-row-factory"&gt;
&lt;h2&gt;Scalar row factory&lt;/h2&gt;
&lt;p&gt;The example above shows a pretty common annoyance. How many times do you need
a single value from the database and you are returned a tuple?&lt;/p&gt;
&lt;p&gt;Psycopg normally emits records as Python tuples; the behaviour can be
customized to return named tuples, dictionaries, or entirely custom objects
with the use of &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/rows.html"&gt;row factories&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;In the frequent case of a query returning a single value, the new &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/rows.html#psycopg.rows.scalar_row"&gt;scalar_row&lt;/a&gt;
factory will return only that:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RawCursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;scalar_row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT ($1 + $2) * $1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This is not a feature of &lt;tt class="docutils literal"&gt;RawCursor&lt;/tt&gt; only, but it's independent from the
choice of the cursor class. We just needed to fix the example above!&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="libpq-17-features"&gt;
&lt;h2&gt;Libpq 17 features&lt;/h2&gt;
&lt;p&gt;In the upcoming PostgreSQL 17 release, the libpq (the PostgreSQL client
library used internally by Psycopg) has seen an unusually intense activity,
with the introduction of &lt;a class="reference external" href="https://www.postgresql.org/docs/17/release-17.html#RELEASE-17-LIBPQ"&gt;several new features&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Our friend Denis Laxalde has been quick to build features and improvements on
top of these new functionalities. So, when Psycopg is used with libpq 17, it
can benefit of features such as:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/connections.html#psycopg.Connection.cancel_safe"&gt;asynchronous, safe cancellation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.stream"&gt;chunked stream results&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/prepare.html#pgbouncer"&gt;better interaction with PgBouncer&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A new &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/objects.html#psycopg.Capabilities"&gt;capabilities object&lt;/a&gt;
can help to navigate the differences and to write programs either degrading
gracefully or crashing helpfully if the libpq used doesn't offer a requested
functionality.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="easier-interaction-with-notifications"&gt;
&lt;h2&gt;Easier interaction with notifications&lt;/h2&gt;
&lt;p&gt;Psycopg 3 introduced a &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/async.html#asynchronous-notifications"&gt;notifications generator&lt;/a&gt;
to receive asynchronous notification from the database. However the generator
turned out to be... difficult to stop! It could be stopped upon receiving a
specific notification as a message, but, because of Python quirks, not easily
from the rest of the program.&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;autocommit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;LISTEN mychan&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;gen&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notifies&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;notify&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;gen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;notify&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# ehm... please kill me!&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;New &lt;tt class="docutils literal"&gt;timeout&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;stop_after&lt;/tt&gt; parameters allow for better control of a
notification listening task (often a component of larger applications) and to
provide better ways to control its operations. Such as to kindly tell it that
its services are not requested anymore without having to kill the whole
program!&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="less-work-for-us"&gt;
&lt;h2&gt;Less work for us!&lt;/h2&gt;
&lt;p&gt;An interesting internal change has helped us to reduce the amount of code to
write and maintain.&lt;/p&gt;
&lt;p&gt;All the Psycopg objects interacting with the network come in two flavours: one
implementing &amp;quot;classic&amp;quot; blocking methods (with which concurrency in a process
can be implemented via multi-threading) and one implementing &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/async.html#asynchronous-operations"&gt;asynchronous
methods&lt;/a&gt;
to participate in &lt;a class="reference external" href="https://docs.python.org/3/library/asyncio.html"&gt;collaborative concurrency&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Thanks to an early design choice, all the libpq I/O interaction only happens
via asynchronous functions and is shared by both the sync and the async
objects; however the code implementing the outermost objects and highest level
behaviour had to be pretty much almost duplicated, with the same features
implemented almost identically with and without async/await keywords, bugs to
be tested and fixed on two sides...&lt;/p&gt;
&lt;p&gt;We have therefore developed an &lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/3.2.0/tools/async_to_sync.py"&gt;async_to_sync conversion tool&lt;/a&gt; to
generate the synchronous code starting from the AST of the asynchronous
counterpart. As a result, the 20-25% of the codebase is now automatically
generated and doesn't require specific maintenance. The process of converting
the sync side from hand-written to auto-generated has also highlighted subtle
differences between async and sync behaviours, which have been addressed, and
affects tests too.&lt;/p&gt;
&lt;p&gt;The technique could be useful for other projects maintaining both sync and
async code, and is interesting enough to require an article of its own to
be written...&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="we-need-your-help"&gt;
&lt;h2&gt;We need your help!&lt;/h2&gt;
&lt;p&gt;Psycopg, first v2, now v3, is the de-facto standard for the communication
between Python and PostgreSQL, two major components of innumerable businesses
and mission-critical infrastructures.&lt;/p&gt;
&lt;p&gt;Maintaining such a critical library to the highest standard of reliability,
completeness, performance requires a lot of care and work.&lt;/p&gt;
&lt;p&gt;If you are a Python and PostgreSQL user and would like to make sure that the
interface between the two is well maintained and continuously improved, please
consider &lt;a class="reference external" href="https://github.com/sponsors/psycopg/"&gt;sponsoring the project&lt;/a&gt;
and to be one of &lt;a class="reference external" href="https://www.psycopg.org/sponsors/"&gt;our sponsors&lt;/a&gt; 💛&lt;/p&gt;
&lt;p&gt;Thank you very much, happy hacking!&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>Pipeline mode in Psycopg</title><link href="https://www.psycopg.org/articles/2024/05/08/psycopg3-pipeline-mode/" rel="alternate"/><updated>2024-05-08T00:00:00Z</updated><author><name>Denis Laxalde</name></author><id>urn:uuid:cc006cca-31a0-34b1-a369-6a1f6b9d3fbd</id><content type="html">&lt;p&gt;&lt;a class="reference external" href="https://www.psycopg.org/articles/2022/08/30/psycopg-31-released/"&gt;Version 3.1&lt;/a&gt; of Psycopg added support for &lt;a class="reference external" href="https://www.postgresql.org/docs/current/libpq-pipeline-mode.html"&gt;libpq pipeline mode&lt;/a&gt;, bringing
significant performance boost, especially when network latency is important.
In this article, we’ll briefly describe how it works from users’ perspective
and &lt;em&gt;under the hood&lt;/em&gt; while also providing a few implementation details.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Supporting &lt;a class="reference external" href="https://www.postgresql.org/docs/current/libpq-pipeline-mode.html"&gt;libpq pipeline mode&lt;/a&gt; involved significant changes to the query
processing logic in the driver. Yet, the challenge was to make it compatible
with the “normal” query mode in order to keep the API almost unchanged and
thus bring performance benefits to users without exposing the complexity of
the batch query mode.&lt;/p&gt;
&lt;p&gt;For the impatient, head out to the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/pipeline.html"&gt;pipeline mode&lt;/a&gt; documentation of
Psycopg: it’s self-consistent, explains nicely the details for
client/server communication, as well as how things work from the user’s
perspective.&lt;/p&gt;
&lt;div class="section" id="using-the-pipeline-mode-in-psycopg"&gt;
&lt;h2&gt;Using the pipeline mode in Psycopg&lt;/h2&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;Connection&lt;/tt&gt; objects gained a &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/connections.html#psycopg.Connection.pipeline"&gt;pipeline()&lt;/a&gt; method to enable the
pipeline mode through a context manager (&lt;a class="reference external" href="https://docs.python.org/3/reference/datamodel.html#context-managers"&gt;&amp;quot;with&amp;quot; statement&lt;/a&gt;); so
using it is as simple as:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;   &lt;span class="c1"&gt;# do work&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="what-is-the-pipeline-mode-for"&gt;
&lt;h2&gt;What is the pipeline mode for?&lt;/h2&gt;
&lt;p&gt;&lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-pipeline-mode.html#LIBPQ-PIPELINE-TIPS"&gt;Postgres documentation&lt;/a&gt; contains advice on when the pipeline mode is
useful. One particular case is when the application is doing many write
operations (&lt;tt class="docutils literal"&gt;INSERT&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;UPDATE&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;DELETE&lt;/tt&gt;).&lt;/p&gt;
&lt;p&gt;For instance, let’s consider the following schema:&lt;/p&gt;
&lt;pre class="code sql literal-block"&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;TABLE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;numeric&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;timestamp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;and assume an application does a lot of queries like:&lt;/p&gt;
&lt;pre class="code sql literal-block"&gt;
&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;with distinct values. Maybe the application could make use of batch inserts
such as &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.executemany"&gt;executemany()&lt;/a&gt;, maybe not (e.g.&amp;nbsp;because it needs to do some other
operations between inserts, like querying another resource): this does not
matter much.&lt;/p&gt;
&lt;p&gt;Let’s put this together into a little &lt;tt class="docutils literal"&gt;demo.py&lt;/tt&gt; Python program:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;math&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;sys&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;datetime&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;create_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;DROP TABLE IF EXISTS t&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;CREATE UNLOGGED TABLE t (x numeric, d timestamp, p boolean)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;do_insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;INSERT INTO t (x, d, p) VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;factorial&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prepare&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;autocommit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;create_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;--pipeline&amp;quot;&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;do_insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;do_insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;row_count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select count(*) from t&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;→ &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;row_count&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; rows&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;we’ll run our script as &lt;tt class="docutils literal"&gt;python demo.py &lt;span class="pre"&gt;[--pipeline]&lt;/span&gt;&lt;/tt&gt;, the
&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;--pipeline&lt;/span&gt;&lt;/tt&gt; flag allowing to enable pipeline mode. Note that we
passed &lt;tt class="docutils literal"&gt;prepare=True&lt;/tt&gt; to &lt;tt class="docutils literal"&gt;Connection.execute()&lt;/tt&gt;, in order to issue a
&lt;tt class="docutils literal"&gt;PREPARE&lt;/tt&gt; statement as we’ll emit the same query many times.&lt;/p&gt;
&lt;p&gt;In general, each &lt;tt class="docutils literal"&gt;INSERT&lt;/tt&gt; query will be fast to execute server-side.
Without the pipeline mode enabled, the client will typically issue the
query and then wait for its result (though it is unused here): thus the
client/server round-trip time will probably be much larger than the
execution time (on server). With the pipeline mode, we basically save
these round-trips most of the times.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="interlude-tracing"&gt;
&lt;h2&gt;Interlude: tracing&lt;/h2&gt;
&lt;p&gt;When working on optimizing client/server communication, it’s essential
to be able to monitor this communication at a reasonably &lt;em&gt;low level&lt;/em&gt;.
From Psycopg’s perspective, the boundary is the libpq. Fortunately, the
library provides a tracing mechanism through the &lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-control.html#LIBPQ-PQTRACE"&gt;PQtrace&lt;/a&gt; function and
friends.&lt;/p&gt;
&lt;p&gt;The output of this function looks like (example taken from the
&lt;a class="reference external" href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/test/modules/libpq_pipeline/traces/prepared.trace;h=1a7de5c3e65e35da3f711e0eeea961cb0b77c5cd;hb=278273ccbad27a8834dfdf11895da9cd91de4114"&gt;PostgreSQL test suite&lt;/a&gt;):&lt;/p&gt;
&lt;pre class="literal-block"&gt;
F   68  Parse    &amp;quot;select_one&amp;quot; &amp;quot;SELECT $1, '42', $1::numeric, interval '1 sec'&amp;quot; 1 NNNN
F   16  Describe     S &amp;quot;select_one&amp;quot;
F   4   Sync
B   4   ParseComplete
B   10  ParameterDescription     1 NNNN
B   113 RowDescription   4 &amp;quot;?column?&amp;quot; NNNN 0 NNNN 4 -1 0 &amp;quot;?column?&amp;quot; NNNN 0 NNNN 65535 -1 0 &amp;quot;numeric&amp;quot; NNNN 0 NNNN 65535 -1 0 &amp;quot;interval&amp;quot; NNNN 0 NNNN 16 -1 0
B   5   ReadyForQuery    I
F   10  Query    &amp;quot;BEGIN&amp;quot;
B   10  CommandComplete  &amp;quot;BEGIN&amp;quot;
B   5   ReadyForQuery    T
F   43  Query    &amp;quot;DECLARE cursor_one CURSOR FOR SELECT 1&amp;quot;
B   19  CommandComplete  &amp;quot;DECLARE CURSOR&amp;quot;
B   5   ReadyForQuery    T
F   16  Describe     P &amp;quot;cursor_one&amp;quot;
F   4   Sync
B   33  RowDescription   1 &amp;quot;?column?&amp;quot; NNNN 0 NNNN 4 -1 0
B   5   ReadyForQuery    T
F   4   Terminate
&lt;/pre&gt;
&lt;p&gt;Each row contains the “direction indicator” (&lt;tt class="docutils literal"&gt;F&lt;/tt&gt; for messages from
client to server or &lt;tt class="docutils literal"&gt;B&lt;/tt&gt; for messages from server to client), the
message length, the &lt;a class="reference external" href="https://www.postgresql.org/docs/14/protocol-message-formats.html"&gt;message type&lt;/a&gt;, and its content. This example shows
messages from the &lt;a class="reference external" href="https://www.postgresql.org/docs/14/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY"&gt;Extended Query&lt;/a&gt; protocol.&lt;/p&gt;
&lt;p&gt;In Psycopg, we have access to the low-level &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/pq.html#psycopg.pq.PGconn"&gt;PGconn&lt;/a&gt; object,
representing the libpq connection, through &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/connections.html#psycopg.Connection.pgconn"&gt;Connection.pgconn&lt;/a&gt;
attribute.&lt;/p&gt;
&lt;p&gt;Here’s how to enable tracing to &lt;tt class="docutils literal"&gt;stderr&lt;/tt&gt;, for our &lt;tt class="docutils literal"&gt;demo.py&lt;/tt&gt; program
above:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;contextlib&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;contextmanager&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;typing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Iterator&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pq&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="nd"&gt;&amp;#64;contextmanager&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;trace_to_stderr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Iterator&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;Enable tracing of the client/server communication to STDERR.&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pgconn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fileno&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pgconn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_trace_flags&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pq&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SUPPRESS_TIMESTAMPS&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;pq&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;REGRESS_MODE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;yield&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pgconn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;untrace&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;do_insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# ...&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;trace_to_stderr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prepare&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="to-pipeline-or-not-to-pipeline"&gt;
&lt;h2&gt;To pipeline or not to pipeline&lt;/h2&gt;
&lt;p&gt;If we run our demo script (without pipeline mode), we’ll typically get
the following output:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
F   69  Parse    &amp;quot;_pg3_0&amp;quot; &amp;quot;INSERT INTO t (x, d, p) VALUES ($1, $2, $3)&amp;quot; 3 NNNN NNNN NNNN
F   4   Sync
B   4   ParseComplete
B   5   ReadyForQuery    I
F   49  Bind     &amp;quot;&amp;quot; &amp;quot;_pg3_0&amp;quot; 3 1 1 1 3 2 '\x00\x01' 8 '\x00\x02\xffffff8b\xffffff8fp~WN' 1 '\x00' 1 0
F   6   Describe     P &amp;quot;&amp;quot;
F   9   Execute  &amp;quot;&amp;quot; 0
F   4   Sync
B   4   BindComplete
B   4   NoData
B   15  CommandComplete  &amp;quot;INSERT 0 1&amp;quot;
B   5   ReadyForQuery    I
F   49  Bind     &amp;quot;&amp;quot; &amp;quot;_pg3_0&amp;quot; 3 1 1 1 3 2 '\x00\x01' 8 '\x00\x02\xffffff8b\xffffff8fp~^\xffffff80' 1 '\x00' 1 0
F   6   Describe     P &amp;quot;&amp;quot;
F   9   Execute  &amp;quot;&amp;quot; 0
F   4   Sync
B   4   BindComplete
B   4   NoData
B   15  CommandComplete  &amp;quot;INSERT 0 1&amp;quot;
B   5   ReadyForQuery    I
[ ... and so forth ~1000 more times ... ]
&lt;/pre&gt;
&lt;p&gt;we indeed see the client/server &lt;em&gt;round-trips&lt;/em&gt; in the form of sequences
of &lt;tt class="docutils literal"&gt;F&lt;/tt&gt; messages followed by sequences of &lt;tt class="docutils literal"&gt;B&lt;/tt&gt; messages for each
query.&lt;/p&gt;
&lt;p&gt;The first message sequence &lt;tt class="docutils literal"&gt;Parse&lt;/tt&gt;+&lt;tt class="docutils literal"&gt;ParseComplete&lt;/tt&gt; corresponds
to the &lt;tt class="docutils literal"&gt;PREPARE&lt;/tt&gt; statement. Next ones only have a
&lt;tt class="docutils literal"&gt;Bind&lt;/tt&gt;/&lt;tt class="docutils literal"&gt;Describe&lt;/tt&gt;/&lt;tt class="docutils literal"&gt;Execute&lt;/tt&gt; client messages followed by server
response.&lt;/p&gt;
&lt;p&gt;Now using the pipeline mode (run the script with &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;--pipeline&lt;/span&gt;&lt;/tt&gt;), we get
the following trace:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
F   69      Parse    &amp;quot;_pg3_0&amp;quot; &amp;quot;INSERT INTO t (x, d, p) VALUES ($1, $2, $3)&amp;quot; 3 NNNN NNNN NNNN
F   49      Bind     &amp;quot;&amp;quot; &amp;quot;_pg3_0&amp;quot; 3 1 1 1 3 2 '\x00\x00' 8 '\x00\x02\xffffff8e&amp;lt;k\x0c\x16\xffffffe6' 1 '\x01' 1 0
F   6       Describe         P &amp;quot;&amp;quot;
F   9       Execute  &amp;quot;&amp;quot; 0
F   49      Bind     &amp;quot;&amp;quot; &amp;quot;_pg3_0&amp;quot; 3 1 1 1 3 2 '\x00\x01' 8 '\x00\x02\xffffff8e&amp;lt;k\x0c\x18\xffffffcd' 1 '\x01' 1 0
F   6       Describe         P &amp;quot;&amp;quot;
F   9       Execute  &amp;quot;&amp;quot; 0
F   49      Bind     &amp;quot;&amp;quot; &amp;quot;_pg3_0&amp;quot; 3 1 1 1 3 2 '\x00\x02' 8 '\x00\x02\xffffff8e&amp;lt;k\x0c\x19\xffffff8a' 1 '\x01' 1 0
F   6       Describe         P &amp;quot;&amp;quot;
F   9       Execute  &amp;quot;&amp;quot; 0
[ ... ~300 more of those ... ]
B   4       ParseComplete
B   4       BindComplete
B   4       NoData
B   15      CommandComplete  &amp;quot;INSERT 0 1&amp;quot;
B   4       BindComplete
B   4       NoData
B   15      CommandComplete  &amp;quot;INSERT 0 1&amp;quot;
B   4       BindComplete
B   4       NoData
B   15      CommandComplete  &amp;quot;INSERT 0 1&amp;quot;
[ ... ~300 more of those ... ]
F   49      Bind     &amp;quot;&amp;quot; &amp;quot;_pg3_0&amp;quot; 3 1 1 1 3 2 '\x01&amp;lt;' 8 '\x00\x02\xffffff8e&amp;lt;k\x0c\xffffff96\xffffff8a' 1 '\x01' 1 0
F   6       Describe         P &amp;quot;&amp;quot;
F   9       Execute  &amp;quot;&amp;quot; 0
F   49      Bind     &amp;quot;&amp;quot; &amp;quot;_pg3_0&amp;quot; 3 1 1 1 3 2 '\x01=' 8 '\x00\x02\xffffff8e&amp;lt;k\x0c\xffffff9c'' 1 '\x01' 1 0
F   6       Describe         P &amp;quot;&amp;quot;
F   9       Execute  &amp;quot;&amp;quot; 0
F   49      Bind     &amp;quot;&amp;quot; &amp;quot;_pg3_0&amp;quot; 3 1 1 1 3 2 '\x01&amp;gt;' 8 '\x00\x02\xffffff8e&amp;lt;k\x0c\xffffff9c\xffffff85' 1 '\x01' 1 0
F   6       Describe         P &amp;quot;&amp;quot;
F   9       Execute  &amp;quot;&amp;quot; 0
[ ... ]
&lt;/pre&gt;
&lt;p&gt;We can see that the client sends more than 900 messages before the
server replies (with the same number of messages). Clearly, this can
have a huge impact on performance, especially when network latency
matters. And indeed, this runs twice faster even though the Postgres
server is on &lt;tt class="docutils literal"&gt;localhost&lt;/tt&gt;!&lt;/p&gt;
&lt;p&gt;What’s actually happening is that the client sends as many queries as
possible, until the server decides it cannot manage more (in general
because its output buffer is full, typically here because of the large
integers we’re inserting), at which point the server sends back the
results of all queries; rinse and repeat. Instead of producing small and
frequent client/server round-trips, the pipeline mode optimizes network
communication by producing large and scarce round-trips. The “downside”
(remember we got a 2x speed-up) is that the client program needs to
handle more data in memory in general.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="how-does-it-work"&gt;
&lt;h2&gt;How does it work?&lt;/h2&gt;
&lt;p&gt;As mentioned earlier, the entry point for the pipeline mode is the
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/connections.html#psycopg.Connection.pipeline"&gt;pipeline()&lt;/a&gt; method on &lt;tt class="docutils literal"&gt;Connection&lt;/tt&gt; object which enters and exists
pipeline mode. But what does this mean? Well, basically, this involves
calling underlying &lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-pipeline-mode.html#LIBPQ-PQENTERPIPELINEMODE"&gt;PQ{enter,exit}PipelineMode&lt;/a&gt; functions.&lt;/p&gt;
&lt;p&gt;But this does not tell much about how things work in Psycopg.&lt;/p&gt;
&lt;p&gt;To actually understand how things work, we need to step back and read
&lt;a class="reference external" href="https://www.postgresql.org/docs/current/libpq-pipeline-mode.html"&gt;libpq pipeline mode&lt;/a&gt; documentation, in which section “Interleaving
Result Processing and Query Dispatch” states:&lt;/p&gt;
&lt;blockquote&gt;
The client application should generally maintain a queue of work remaining
to be dispatched and a queue of work that has been dispatched but not yet
had its results processed. When the socket is writable it should dispatch
more work. When the socket is readable it should read results and process
them, matching them up to the next entry in its corresponding results
queue.&lt;/blockquote&gt;
&lt;p&gt;As often with PostgreSQL, everything is there although this paragraph is
somehow enigmatic. However, it, in fact, describes the heart of the
algorithm for the Psycopg driver (though it took us a while to grasp all
the details implied by these few sentences…).&lt;/p&gt;
&lt;div class="section" id="socket-communication"&gt;
&lt;h3&gt;Socket communication&lt;/h3&gt;
&lt;blockquote&gt;
When the socket is writable it should dispatch more work. When the
socket is readable it should read results […].&lt;/blockquote&gt;
&lt;p&gt;In Psycopg, socket communication for exchanging libpq messages is
implemented through &lt;em&gt;waiting functions&lt;/em&gt; and &lt;em&gt;generators&lt;/em&gt; that are tied
together by the I/O layer (either blocking or async): this is explained
in details in &lt;a class="reference external" href="https://www.varrazzo.com/blog/2020/03/26/psycopg3-first-report/"&gt;Daniele’s blog post&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;What’s important for the pipeline mode (mostly) is the generator part,
as it is responsible for &lt;em&gt;dispatching queries to&lt;/em&gt; or &lt;em&gt;reading results
from&lt;/em&gt; the socket. In contrast with normal query mode, where these steps
are handled sequentially by independent logic, the pipeline mode needs
&lt;em&gt;interleaving result processing and query dispatch&lt;/em&gt;: this is implemented
by the &lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/3.1/psycopg/psycopg/generators.py#L180"&gt;pipeline_communicate()&lt;/a&gt; generator. Without going too much into
the details, we can notice that: - the function takes a &lt;em&gt;queue&lt;/em&gt; of
“commands”, e.g. &lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-async.html#LIBPQ-PQSENDQUERYPARAMS"&gt;pgconn.send_query_params()&lt;/a&gt; or similar, - it
continuously waits for the socket to be either Read or Write ready (or
both) (&lt;tt class="docutils literal"&gt;ready = yield Wait.RW&lt;/tt&gt;), - when the socket is Read-ready
(&lt;tt class="docutils literal"&gt;if ready &amp;amp; Ready.R:&lt;/tt&gt;), results are fetched (calling
&lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-async.html#LIBPQ-PQGETRESULT"&gt;pgconn.get_result()&lt;/a&gt;), - when the socket is Write-ready
(&lt;tt class="docutils literal"&gt;if ready &amp;amp; Ready.W:&lt;/tt&gt;), commands are sent (calling &lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-async.html#LIBPQ-PQFLUSH"&gt;pgconn.flush()&lt;/a&gt;
to flush the queue of previously sent commands, and then calling any
pending one), - until the queue of commands gets empty.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="queueing-work-processing-results"&gt;
&lt;h3&gt;Queueing work, processing results&lt;/h3&gt;
&lt;p&gt;Around the &lt;tt class="docutils literal"&gt;pipeline_communicate()&lt;/tt&gt; generator described above, we need
to handle the commands queue as well as the queue of results pending
processing. The first part, filling the commands queue, is simply
managed by stacking commands instead of directly calling them along with
keeping a reference of the cursor used for &lt;tt class="docutils literal"&gt;execute()&lt;/tt&gt;. The second
part implies handling the output of &lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/3.1/psycopg/psycopg/generators.py#L180"&gt;pipeline_communicate()&lt;/a&gt; generator
described above, a list of &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/pq.html#psycopg.pq.PGresult"&gt;PGresult&lt;/a&gt;. Each fetched result item: - is
possibly bound back to its respective cursor (the one where respective
&lt;tt class="docutils literal"&gt;execute()&lt;/tt&gt; originates from), - might trigger an error if its status
is non-&lt;tt class="docutils literal"&gt;OK&lt;/tt&gt; (e.g.&amp;nbsp;&lt;tt class="docutils literal"&gt;FATAL_ERROR&lt;/tt&gt;).&lt;/p&gt;
&lt;p&gt;All this is handled in methods of the &lt;a class="reference external" href="https://github.com/psycopg/psycopg/blob/3.1/psycopg/psycopg/_pipeline.py#L37"&gt;BasePipeline&lt;/a&gt; class (see methods
prefixed with an &lt;tt class="docutils literal"&gt;_&lt;/tt&gt; at the end).&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="integration-with-high-level-features-transactions"&gt;
&lt;h2&gt;Integration with high-level features: transactions&lt;/h2&gt;
&lt;p&gt;Beside the low-level logic described above, implementing pipeline mode
in Psycopg implied handling some Psycopg-specific features such as:
transactions.&lt;/p&gt;
&lt;p&gt;Transactions need special attention because of how &lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-pipeline-mode.html#LIBPQ-PIPELINE-ERROS"&gt;error handling&lt;/a&gt;
works in the pipeline mode. There is a few distinct cases that need to
be handled properly, depending on whether the pipeline uses an &lt;em&gt;implicit
transaction&lt;/em&gt; or if it contains &lt;em&gt;explicit transactions&lt;/em&gt;. But the general
rule is that when an error occurs, the pipeline gets in &lt;em&gt;aborted&lt;/em&gt; state
meaning subsequent commands are skipped and prior statements might get
persisted or not (depending on the usage of explicit transactions or
not).&lt;/p&gt;
&lt;p&gt;Consider the following statements, executed within a pipeline:&lt;/p&gt;
&lt;pre class="code sql literal-block"&gt;
&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;transaction&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'abc'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;transaction&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;no_such_table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'x'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;ROLLBACK&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;transaction&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'xyz'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;abc&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The &lt;tt class="docutils literal"&gt;INSERT INTO no_such_table&lt;/tt&gt; statement would produce an error,
making the pipeline &lt;strong&gt;aborted&lt;/strong&gt;; accordingly, the following explicit
&lt;tt class="docutils literal"&gt;ROLLBACK&lt;/tt&gt; will not be executed. And the next statements (“transaction
3”) will also be skipped.&lt;/p&gt;
&lt;p&gt;Another example:&lt;/p&gt;
&lt;pre class="code sql literal-block"&gt;
&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;transaction&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'abc'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;transaction&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;no_such_table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'x'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;ROLLBACK&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;INTO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;VALUES&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'xyz'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Here, still due to the same error in &lt;tt class="docutils literal"&gt;INSERT INTO no_such_table&lt;/tt&gt;, the
final &lt;tt class="docutils literal"&gt;COMMIT&lt;/tt&gt; statement is not executed and the main (outer)
transaction is not committed (despite the inner sub-transaction is
explicitly rolled back).&lt;/p&gt;
&lt;p&gt;That’s typically something the user of a high level driver would not
want.&lt;/p&gt;
&lt;p&gt;In Psycopg, transactions are managed explicitly through the
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/transactions.html#transaction-contexts"&gt;transaction()&lt;/a&gt; context manager method on &lt;tt class="docutils literal"&gt;Connection&lt;/tt&gt; objects. So to
preserve a consistent behaviour, its logic needed to be adapted for the
pipeline mode. This got achieved by leveraging synchronization points
through &lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-pipeline-mode.html#LIBPQ-PIPELINESYNC"&gt;PQpipelineSync&lt;/a&gt; and nested pipelines.&lt;/p&gt;
&lt;div class="section" id="nested-pipelines"&gt;
&lt;h3&gt;Nested pipelines&lt;/h3&gt;
&lt;p&gt;In the libpq, there is no such thing as a nested pipeline as the
connection can only enter pipeline mode once. What’s referred to as a
“nested pipeline” in Psycopg is the operation to “isolate” a sequence of
commands in a pipeline session through synchronization points. By doing
so, we work around the surprising behaviour described above (where a
committed transaction got rolled back). Here’s what happens:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;  &lt;span class="c1"&gt;# emits PQenterPipelineMode&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;  &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;  &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;  &lt;span class="c1"&gt;# emits PQpipelineSync&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;  &lt;span class="c1"&gt;# exiting the inner 'with' block emits PQpipelineSync&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;# exiting the outermost 'with' block emits PQexitPipelineMode&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The &lt;a class="reference external" href="https://www.postgresql.org/docs/14/libpq-pipeline-mode.html#LIBPQ-PIPELINESYNC"&gt;PQpipelineSync&lt;/a&gt; operation &lt;em&gt;resets&lt;/em&gt; the pipeline state, thus
allowing subsequent commands to be run independently of whether previous
ones succeeded or not. (It also triggers results to be sent back from
the server, but that’s another matter.)&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="pipelined-transactions"&gt;
&lt;h3&gt;Pipelined transactions&lt;/h3&gt;
&lt;p&gt;By using nested pipelines for Psycopg transactions, we typically follow
the “logical unit of work” pattern that’s mentioned in &lt;a class="reference external" href="https://www.postgresql.org/docs/current/libpq-pipeline-mode.html"&gt;libpq pipeline
mode&lt;/a&gt; documentation:&lt;/p&gt;
&lt;blockquote&gt;
Pipelines should be scoped to logical units of work, usually (but not
necessarily) one transaction per pipeline.&lt;/blockquote&gt;
&lt;p&gt;(Except that we’re not strictly use one pipeline per transaction, rather
a nested one.)&lt;/p&gt;
&lt;p&gt;In practice, it means that it’s safe to use &lt;tt class="docutils literal"&gt;with &lt;span class="pre"&gt;transaction():&lt;/span&gt;&lt;/tt&gt;
block within a pipeline session as the semantics of both the transaction
and the pipeline are preserved: the transaction either succeeds or fails
overall, it only gets executed if previous commands in the pipeline
session succeeded:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;  &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;  &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;      &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;  &lt;span class="c1"&gt;# implicit nested pipeline (with conn.pipeline())&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;          &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;  &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;      &lt;span class="c1"&gt;# This will be executed independently of whether the previous&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;      &lt;span class="c1"&gt;# transaction succeeded or not.&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;      &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;So back to the (second) example above, if written using Psycopg:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;autocommit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;     &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;         &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;             &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;INSERT INTO s VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;abc&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;             &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;                 &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;                     &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;INSERT INTO no_such_table VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;x&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;             &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;UndefinedTable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;                 &lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;             &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;INSERT INTO s VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;xyz&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;     &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT * FROM s ).fetchall()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="s1"&gt;'abc'&lt;/span&gt;&lt;span class="p"&gt;,),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'xyz'&lt;/span&gt;&lt;span class="p"&gt;,)]&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;we indeed get the inner transaction rolled back, and the outer one
committed, just like without the pipeline mode.&lt;/p&gt;
&lt;p&gt;That’s an implementation detail, the user does not need to know about
this as the overall behaviour is hopefully natural.&lt;/p&gt;
&lt;hr class="docutils" /&gt;
&lt;p&gt;Supporting libpq pipeline mode in Psycopg was a large milestone. It
required months of work with a lot of thinking and testing. There is
probably more to say about it, like how it transparently manages
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/prepare.html"&gt;automatic prepared statement&lt;/a&gt; or how &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.executemany"&gt;executemany()&lt;/a&gt; got optimized to
use the pipeline mode implicitly (try adapting the demo script above to
use it - hint: no need for the &lt;tt class="docutils literal"&gt;with pipeline:&lt;/tt&gt; block). And be sure to
read the Psycopg’s &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/pipeline.html"&gt;pipeline mode&lt;/a&gt; documentation soon!&lt;/p&gt;
&lt;hr class="docutils" /&gt;
&lt;blockquote&gt;
This article, originally published at &lt;a class="reference external" href="https://blog.dalibo.com/2022/09/19/psycopg-pipeline-mode.html"&gt;Pipeline mode in Psycopg&lt;/a&gt;, is used
under &lt;a class="reference external" href="https://creativecommons.org/licenses/by-nc-sa/4.0/"&gt;CC BY-NC-SA&lt;/a&gt;
(introduction shortened).&lt;/blockquote&gt;
&lt;/div&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 3.1 released</title><link href="https://www.psycopg.org/articles/2022/08/30/psycopg-31-released/" rel="alternate"/><updated>2022-08-30T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:bca301bf-8fa7-306d-a4d6-022a4af604f5</id><content type="html">&lt;p&gt;Hello,&lt;/p&gt;
&lt;p&gt;After several months of development, we are proud to release Psycopg 3.1!&lt;/p&gt;
&lt;p&gt;Psycopg 3.1 is a gradual improvement on Psycopg 3.0, introducing new exciting
features, redefining what can be done on the boundary between Python and
PostgreSQL.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="pipeline-mode"&gt;
&lt;h2&gt;Pipeline mode&lt;/h2&gt;
&lt;p&gt;The &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/pipeline.html"&gt;pipeline mode&lt;/a&gt;
is by far the biggest feature introduced in Psycopg 3.1, largely the work of
Denis Laxalde and supported by &lt;a class="reference external" href="http://www.dalibo.com"&gt;Dalibo&lt;/a&gt;. In
pipeline mode, Psycopg will send batches of commands to the server without
waiting for a response for every operation, resulting in a massive speed
improvement.&lt;/p&gt;
&lt;p&gt;The pipeline mode is exposed to Python as a context block. Within the block,
Psycopg will manage the pipeline in a transparent way, even allowing the use of
features which break the pipeline flow, for instance to fetch results, or to
manage transactions:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;operations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;rec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;op2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;rec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="o"&gt;...&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Quantifying the speedup is difficult, as it depends on the network conditions
and on the pattern of statements executed. In particularly bad conditions (250
ms of ping time between client and server), we measured a loop of 100 inserts
to take 25 s in normal mode and just 0.5 s in pipeline mode, for a 50x
speedup. Testing the same operation on a localhost connection, we have
measured a &amp;gt;20x speedup (a 5000 inserts batch taking 0.3 s instead of 6.6 s).
If the program requires results from the server before sending further
statement (for instance to insert in a table and then use the new record's
primary key to insert related records in different tables) you can expect
less dramatic speedups.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="client-binding-cursors"&gt;
&lt;h2&gt;Client-binding cursors&lt;/h2&gt;
&lt;p&gt;Psycopg 3 uses &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html#server-side-binding"&gt;server-side binding&lt;/a&gt;,
passing the query and adapted arguments separately. This allows to use several
features otherwise unavailable, such as prepared statements. However, many
types of statements, especially data-definition, don't support server-side
parameters. The &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/sql.html"&gt;'sql' module&lt;/a&gt; allows to compose
statements on the client, but using it might require many changes to programs
making heavy use of data-definition statements.&lt;/p&gt;
&lt;p&gt;Psycopg 3.1 introduces a &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/cursors.html#client-side-binding-cursors"&gt;ClientCursor&lt;/a&gt;
object, which makes psycopg2 programs easier to port. This cursor reproduces
psycopg2's way of composing queries, which is not the most efficient, but for
certain programs it is exactly what is needed.&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cursor_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ClientCursor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# this is a client-side binding cursor&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# This statement doesn't support server-side parameters&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ALTER TABLE x ALTER y SET DEFAULT &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;hello&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;As a bonus, the client cursor reintroduces the handy and often requested
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.ClientCursor.mogrify"&gt;mogrify()&lt;/a&gt;
method, which returns the query merged with the parameters the way it is
passed to the server:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mogrify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ALTER TABLE x ALTER y SET DEFAULT &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;hell'o&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ALTER TABLE x ALTER y SET DEFAULT 'hell''o'&amp;quot;&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="enum-adaptation"&gt;
&lt;h2&gt;Enum adaptation&lt;/h2&gt;
&lt;p&gt;Python has &lt;a class="reference external" href="https://docs.python.org/3/library/enum.html"&gt;enums&lt;/a&gt;, PostgreSQL
has &lt;a class="reference external" href="https://www.postgresql.org/docs/current/static/datatype-enum.html"&gt;enums&lt;/a&gt;...
Why not map them into each other? Well for a start because they are
actually pretty different from each other (Python enums have a type and value,
Postgres ones are just identities) and because often, in programs,
differences between the enums in the db and the code creep in.&lt;/p&gt;
&lt;p&gt;Psycopg 3.1 introduces &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/adapt.html#enum-adaptation"&gt;a flexible adapter between Python and Postgres enums&lt;/a&gt;.
It can be used in a simple way when there is a one-to-one mapping between the
enums:&lt;/p&gt;
&lt;pre class="code literal-block"&gt;
=# CREATE TYPE numbers AS ENUM ('ONE', 'TWO', 'THREE');
&lt;/pre&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Numbers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Enum&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;ONE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;TWO&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;THREE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;EnumInfo&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;numbers&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;register_enum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Numbers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT 'TWO'::numbers&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;Numbers&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TWO&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT pg_typeof(&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Numbers&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ONE&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="s1"&gt;'numbers'&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The facility can also be customized in order to adapt enums when the mapping
is not one-to-one:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;NumbersPlus&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Enum&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;ONE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;TWO&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;THREE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;THREE_PLUS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# has some meaning in the program, but it is not stored&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;register_enum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;NumbersPlus&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# - Items not mentioned map naturally&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="c1"&gt;# - This order gives THREE priority over THREE_PLUS when loading from db.&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;mapping&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;NumbersPlus&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;THREE_PLUS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;THREE&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;NumbersPlus&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;THREE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;THREE&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;::text&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="n"&gt;NumbersPlus&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ONE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;NumbersPlus&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;THREE_PLUS&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="s1"&gt;'{ONE,THREE}'&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select '{TWO,THREE}'::numbers[]&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;NumbersPlus&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TWO&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;NumbersPlus&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;THREE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="in-partnership-with-cockroachdb"&gt;
&lt;h2&gt;In partnership with CockroachDB&lt;/h2&gt;
&lt;p&gt;&lt;a class="reference external" href="https://www.cockroachlabs.com/"&gt;CockroachDB&lt;/a&gt; is a distributed database
presenting an SQL interface on top of a distributed key-value store. Although
it is a completely independent implementation, it uses the same PostgreSQL
client-server protocol.&lt;/p&gt;
&lt;p&gt;In the past few months, we have collaborated to create an even smoother
integration, so that every PostgreSQL feature, also supported by CockroachDB,
can be used in a transparent way: &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/cursors.html#server-side-cursors"&gt;server-side cursors&lt;/a&gt;,
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/pipeline.html"&gt;pipeline mode&lt;/a&gt;, CockroachDB
data types are all supported out-of-the-box.&lt;/p&gt;
&lt;p&gt;CockroachDB also implements &lt;a class="reference external" href="https://www.cockroachlabs.com/docs/stable/changefeed-for.html"&gt;CHANGEFEED&lt;/a&gt;, a
streaming query, which Psycopg can consume using its Cursor.stream() feature.
This immediately receives every change happening in a database table, enabling
interesting new ways to write distributed applications.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="executemany-improvements"&gt;
&lt;h2&gt;&lt;tt class="docutils literal"&gt;executemany()&lt;/tt&gt; improvements&lt;/h2&gt;
&lt;p&gt;The &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.executemany"&gt;executemany()&lt;/a&gt;
method now supports returning the executed statements' output. You can now for
instance execute an &lt;tt class="docutils literal"&gt;INSERT ... RETURNING&lt;/tt&gt; on a batch of records and
retrieve the ids associated to the newly inserted records.&lt;/p&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;executemany()&lt;/tt&gt; automatically uses &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/pipeline.html"&gt;the pipeline mode&lt;/a&gt; already
described, making use of pipeline mode speedups without changing any code in
the programs using this method.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="and-many-more-improvements"&gt;
&lt;h2&gt;And many more improvements&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;The &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/transactions.html#two-phase-commit"&gt;Two-phase commit protocol&lt;/a&gt;
is now available as per &lt;a class="reference external" href="https://peps.python.org/pep-0249/#optional-two-phase-commit-extensions"&gt;DBAPI specification&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Asynchronous connections &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/async.html#asynchronous-operations"&gt;don't block on DNS names resolution on connect&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;Cursor.copy()&lt;/tt&gt; now &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.copy"&gt;takes parameters&lt;/a&gt;,
like a normal query.&lt;/li&gt;
&lt;li&gt;It is also possible to &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/copy.html#copy-writers"&gt;replace the writer of a 'Copy' object&lt;/a&gt;, in
order to use psycopg just to format data in COPY format and to do something
else with the data produced, for instance save it to a file for later
processing.&lt;/li&gt;
&lt;li&gt;...And many more improvements you can find in our &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/news.html#psycopg-3-1"&gt;release notes&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div class="section" id="thank-you-very-much"&gt;
&lt;h2&gt;Thank you very much!&lt;/h2&gt;
&lt;p&gt;We hope you will enjoy to use Psycopg 3.1 and will benefit from its new
features. Psycopg 3 is developed and maintained thanks to the support
of &lt;a class="reference external" href="https://www.psycopg.org/sponsors/"&gt;our sponsors&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;If you are a Python and PostgreSQL user and would like to make sure that the
interface between the two is well maintained and continuously improved, please
consider &lt;a class="reference external" href="https://github.com/sponsors/psycopg/"&gt;sponsoring the project 💛&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Thank you very much, happy hacking!&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 3.0 released</title><link href="https://www.psycopg.org/articles/2021/10/13/psycopg-30-released/" rel="alternate"/><updated>2021-10-13T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:060c020e-db6c-391f-b8bc-af629192c059</id><content type="html">&lt;p&gt;Hello,&lt;/p&gt;
&lt;p&gt;I am extremely excited to announce the first stable release of Psycopg 3!&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Psycopg 3 is a complete rewrite based on the experience accumulated with the
development and maintenance of psycopg2. Psycopg 3 targets all the current
versions of Python (3.6-3.10) and PostgreSQL (10-14) and allows the use of
modern Python development techniques, such as async and statically typed code.
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/"&gt;A list of the new features&lt;/a&gt; is
available in the documentation.&lt;/p&gt;
&lt;p&gt;This was a long journey: I would like to thank the people who have helped to
make this project amazing with their ideas and their code: Denis Laxalde (row
factories), Daniel Fortunov (transaction blocks), Jacopo Farina (PostGIS
support) and many who have tested, helped, discussed, cheered for us.&lt;/p&gt;
&lt;p&gt;And an immense thank you to the sponsors who have made this project possible:
&lt;a class="reference external" href="https://postgrespro.com/"&gt;Postgres Professional&lt;/a&gt; and &lt;a class="reference external" href="https://www.commandprompt.com/"&gt;Command Prompt&lt;/a&gt; have been our biggest supporters so far,
but &lt;a class="reference external" href="https://www.psycopg.org/sponsors/"&gt;many companies and individuals&lt;/a&gt;
have given their generous contribution. Surely there will be more work to come
in the future: if you want you can help &lt;a class="reference external" href="https://github.com/sponsors/psycopg/"&gt;sponsoring the project&lt;/a&gt; too.&lt;/p&gt;
&lt;p&gt;If you would like to try out the project please check out the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/install.html"&gt;installation
and usage instructions&lt;/a&gt;. We are eager to
hear your feedback and to share this journey with you.&lt;/p&gt;
&lt;p&gt;--- Daniele, on behalf of Psycopg&lt;/p&gt;
</content></entry><entry><title>Psycopg 3.0 beta 1 released!</title><link href="https://www.psycopg.org/articles/2021/08/30/psycopg-30-beta1-released/" rel="alternate"/><updated>2021-08-30T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:e099f1fe-54e1-3c67-849f-2c3d0d1ecb27</id><content type="html">&lt;p&gt;We are immensely proud to &lt;a class="reference external" href="https://pypi.org/project/psycopg/3.0b1/"&gt;release on PyPI&lt;/a&gt; the first beta package of Psycopg
3!&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Psycopg 3 is a complete rewrite of Psycopg 2, maintaining the same fundamental
libpq wrapper architecture and DB-API interface design, but exposing new
features to better work with the newer versions of Python and PostgreSQL.&lt;/p&gt;
&lt;p&gt;On the Python side, Psycopg 3 allows the use of asyncio-based concurrency and
static typing. Many improvement to the Python interface make the library much
simpler and more idiomatic to use,&lt;/p&gt;
&lt;p&gt;On the PostgreSQL side, Psycopg 3 makes use of server-side parameters,
prepared statements, binary parameters, and great support for COPY operations.&lt;/p&gt;
&lt;p&gt;But the most outstanding feature of the project is not a technical one:
Psycopg 3 was made possible by the great generosity of many &lt;a class="reference external" href="https://www.psycopg.org/sponsors/"&gt;sponsors&lt;/a&gt;, who have funded the development of the
project. Among the many backers, we are especially grateful to &lt;a class="reference external" href="https://postgrespro.com/"&gt;Postgres
Professional&lt;/a&gt; and &lt;a class="reference external" href="https://www.commandprompt.com/"&gt;Command Prompt, Inc&lt;/a&gt;, which have given the most outstanding
support. But many other companies and individuals, each one in their capacity,
have shown concrete support for free software development and progress. We
sincerely hope that you will find this work useful and that you will feel
proud for having contributed to it.&lt;/p&gt;
&lt;p&gt;&lt;a class="reference external" href="https://www.psycopg.org/sponsors/"&gt;https://www.psycopg.org/sponsors/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Where do we go from here? The hope is that the interface of the adapter will
not change excessively before a definitive 3.0 release: the project has
already been used in a few production environments, in the past months, and a
lot of real world feedback has already helped to improve the interface and
functionalities. We invite you to test the project and give us your feedback.
So...&lt;/p&gt;
&lt;pre class="literal-block"&gt;
pip install -U pip
pip install --pre psycopg[binary]
&lt;/pre&gt;
&lt;p&gt;Please try it, test it, and let us know how it goes!&lt;/p&gt;
&lt;p&gt;For more info you can dive into the docs: start from the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/install.html"&gt;install&lt;/a&gt; and &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/usage.html"&gt;usage&lt;/a&gt;
pages, I'm sure you will find your way.&lt;/p&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
</content></entry><entry><title>Building a Django driver for Psycopg 3</title><link href="https://www.psycopg.org/articles/2021/08/02/psycopg3-django-driver/" rel="alternate"/><updated>2021-08-02T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:415d9dfd-ed5e-3485-b48f-15ccd11bee38</id><content type="html">&lt;p&gt;One of the goals of the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/"&gt;Psycopg 3 project&lt;/a&gt; is to make easy to port code
developed from Psycopg 2. For this reason the creation of a Django backend
(the module you specify in the settings as your &lt;a class="reference external" href="https://docs.djangoproject.com/en/3.2/ref/settings/#databases"&gt;database ENGINE&lt;/a&gt;) was
a project with a double goal:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;A Django driver is a way to make Psycopg 3 useful from the start, with the
possibility of dropping it in a project transparently and have available,
when needed the new features offered (for instance the superior &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/copy.html"&gt;COPY
support&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;The difficulty of introducing Psycopg 3 in the Django codebase and the type
of changes required are indicative of the type of problems that could be
found porting other projects.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;...and it's done! A few days ago, the new &lt;a class="reference external" href="https://github.com/dvarrazzo/django-psycopg3-backend"&gt;Psycopg 3 Django backend&lt;/a&gt; could
pass the entire Django test suite!&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="figure"&gt;
&lt;img alt="test-logs.png" src="test-logs.png" style="width: 700px;" /&gt;
&lt;/div&gt;
&lt;p&gt;The implementation of the Django backend actually started several months ago,
but it can be seen, from the test progression above, that its development had
been suspended for several months. The problem, in the first attempts, was
that too much of the Django code was in need of being adapted: this was a sign
that the changes needed to use the new adapter were too invasive and that the
same type of difficulties would have been met by everyone trying to replace
Psycopg 2 with Psycopg 3. Back to the design board then, but hopefully the
resulting adapter will behave mostly as you might expect and will not force
users to change every query in their program (which would have been a deal
breaker for most non-trivial projects).&lt;/p&gt;
&lt;p&gt;The backend cannot be used with the current Django version: &lt;a class="reference external" href="https://github.com/django/django/compare/stable/3.2.x...dvarrazzo:psycopg3-support"&gt;a few
modifications&lt;/a&gt; to the Django codebase are needed in order to use it. These
changes will be proposed to the Django project: if the Django maintainer
will accept them, the driver should be usable starting from one of the next
Django releases.&lt;/p&gt;
&lt;p&gt;The aim of this article is to take a look at some of these modifications, to
understand where the behaviour of Psycopg 3 diverges from its well known
predecessor and how to work around the differences.&lt;/p&gt;
&lt;div class="section" id="server-side-parameters-binding"&gt;
&lt;h2&gt;Server-side parameters binding&lt;/h2&gt;
&lt;p&gt;Many of these changes are the consequence of using server-side binding for the
query parameters (using the libpq &lt;a class="reference external" href="https://www.postgresql.org/docs/13/libpq-exec.html#LIBPQ-PQEXECPARAMS"&gt;PQexecParams()&lt;/a&gt; function), instead of
merging the arguments to the query on the client-side and using the simpler
&lt;a class="reference external" href="https://www.postgresql.org/docs/13/libpq-exec.html#LIBPQ-PQEXEC"&gt;PQexec()&lt;/a&gt; function.&lt;/p&gt;
&lt;p&gt;In the &lt;tt class="docutils literal"&gt;PQexec()&lt;/tt&gt; case, the Postgres query parser has access to the literal
values in the context where they are used and it looks like it is able to use
this information in ways we don't appreciate...until we lose them. Do you
think that &lt;tt class="docutils literal"&gt;text&lt;/tt&gt; is the best PostgreSQL data type to convert Python strings
to? I wish it was so simple. Below is an experiment with the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/pq.html"&gt;psycopg.pq&lt;/a&gt; objects,
the low level libpq wrapper that Psycopg 3 exposes:&lt;/p&gt;
&lt;pre class="code pycon literal-block"&gt;
&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg.postgres&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;create table testjson (id serial primary key, data jsonb)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# &amp;lt;psycopg.Cursor [COMMAND_OK] [INTRANS] (database=piro) at 0x7f92d43d7b80&amp;gt;

# Note: $1, $2... are the low level Postgres placeholders.
# In a normal Psycopg query you would use classic '%s'.
&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pgconn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exec_params&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;insert into testjson (data) values ($1)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;text&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# &amp;lt;psycopg_c.pq.PGresult [FATAL_ERROR] at 0x7f92cec70a90&amp;gt;

&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error_message&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;utf8&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# ERROR:  column &amp;quot;data&amp;quot; is of type jsonb but expression is of type text
# LINE 1: insert into testjson (data) values ($1)
#                                             ^
# HINT:  You will need to rewrite or cast the expression.&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Specifying the &lt;tt class="docutils literal"&gt;text&lt;/tt&gt; Postgres type is an excessively strict type
indication: in most cases, Postgres will not be able to automatically convert
the value to the required type.&lt;/p&gt;
&lt;p&gt;When we use a literal &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;'{}'&lt;/span&gt;&lt;/tt&gt; in the query, we are specifying an &lt;em&gt;untyped
literal&lt;/em&gt;. &lt;a class="reference external" href="https://www.postgresql.org/docs/13/libpq-exec.html#LIBPQ-PQEXECPARAMS"&gt;Postgres docs&lt;/a&gt; say that we can do the same using 0 as type OID for
the parameter (see the &lt;tt class="docutils literal"&gt;paramTypes[]&lt;/tt&gt; description). But it seems this isn't
always the case. For instance:&lt;/p&gt;
&lt;pre class="code pycon literal-block"&gt;
&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select concat(&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# ...becomes...
&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pgconn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exec_params&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select concat($1, $2)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# &amp;lt;psycopg_c.pq.PGresult [FATAL_ERROR] at 0x7f92d43db4d0&amp;gt;

&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error_message&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;utf8&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# ERROR:  could not determine data type of parameter $1&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This problem doesn't happen with every function: it seems to be only a problem
with &amp;quot;variadic&amp;quot; functions, such as &lt;tt class="docutils literal"&gt;concat()&lt;/tt&gt; or &lt;tt class="docutils literal"&gt;json_build_object()&lt;/tt&gt;. As
sporadic as it is, it doesn't seem like there is a universally correct way of
mapping Python types to PostgreSQL type OIDs: we can try to get it right &lt;em&gt;most
of the time&lt;/em&gt; (so, by default, Psycopg 3 dumps Python strings using the OID 0),
but places where this isn't right do exist...and they exist in Django, of
course.&lt;/p&gt;
&lt;p&gt;There are two different ways to work around the problem, both have their
merit and one might be easier to use than the other in different contexts.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p class="first"&gt;Add a cast to the placeholder: specifying &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;%s::text&lt;/span&gt;&lt;/tt&gt; (or other types) in
your query, it is possible to disambiguate the type where &amp;quot;unknown&amp;quot; doesn't
work:&lt;/p&gt;
&lt;pre class="code pycon literal-block"&gt;
&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select concat(&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;::text, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;::text)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# ...becomes...
&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pgconn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exec_params&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select concat($1::text, $2::text)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# &amp;lt;psycopg_c.pq.PGresult [TUPLES_OK] at 0x7f92cebfb630&amp;gt;

&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get_value&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# b'foobar'&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;One place in Django where this was needed is &lt;a class="reference external" href="https://github.com/django/django/compare/stable/3.2.x...dvarrazzo:psycopg3-support#diff-b6a6632418c964155865691be58c9f76717ef512ced704d461f41cc3612a1db3"&gt;in array comparisons&lt;/a&gt;,
because they follow stricter rules than the base type comparisons and may
require an explicit cast to work.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;The other option is to use a different type than the Python &lt;tt class="docutils literal"&gt;str&lt;/tt&gt; and
handle it using a different dumper.&lt;/p&gt;
&lt;pre class="code pycon literal-block"&gt;
&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select concat(&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# ...becomes...
&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pgconn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exec_params&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select concat($1, $2)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;text&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;text&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# &amp;lt;psycopg_c.pq.PGresult [TUPLES_OK] at 0x7f92cebfbbd0&amp;gt;

&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get_value&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# b'foobar'&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This is &lt;a class="reference external" href="https://github.com/django/django/compare/stable/3.2.x...dvarrazzo:psycopg3-support#diff-76917634c6c088f56f8dec9493294c657953e61b01e38e33b02d876d5e96dd3a"&gt;proposed in Django&lt;/a&gt; and used, for instance, &lt;a class="reference external" href="https://github.com/django/django/compare/stable/3.2.x...dvarrazzo:psycopg3-support#diff-444d9c5689a90ceeb46dc42d947ff042d67d5fa229e633734afbe1235604ec28"&gt;in the concat()
case&lt;/a&gt;.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Note that both the solutions make the queries compatible with Psycopg 2
and 3: the &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;%s::text&lt;/span&gt;&lt;/tt&gt; casts are no problem in &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; queries and
&lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; is smart enough to notice that &lt;tt class="docutils literal"&gt;Text&lt;/tt&gt; is a &lt;tt class="docutils literal"&gt;str&lt;/tt&gt; subclass and
to apply the same conversion rules used for the vanilla strings.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="server-side-binding-doesn-t-work-for-all-the-queries"&gt;
&lt;h2&gt;Server-side binding doesn't work for all the queries&lt;/h2&gt;
&lt;p&gt;Argument binding on the server only works for queries that select and modify
data, but it doesn't work in the Data Definition Language. For example:&lt;/p&gt;
&lt;pre class="code pycon literal-block"&gt;
&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;create table test (id int default &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;# Traceback (most recent call last):
#     ...
# psycopg.errors.UndefinedParameter: there is no parameter $1
# LINE 1: create table test (id int default $1)
#                                           ^&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The solution for this type of problem is to use the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/sql.html"&gt;psycopg.sql&lt;/a&gt; module to
explicitly generate a query client-side and send it to the server without
parameters:&lt;/p&gt;
&lt;pre class="code pycon literal-block"&gt;
&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SQL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;create table test (id int default &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;... &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;A similar module is &lt;a class="reference external" href="https://www.psycopg.org/docs/sql.html"&gt;available in psycopg2 too&lt;/a&gt; so it is easy to write code
that works for both versions: it's just the &lt;tt class="docutils literal"&gt;import&lt;/tt&gt; statement that needs
to be changed.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="group-by-order-by-with-parameters"&gt;
&lt;h2&gt;GROUP BY/ORDER BY with parameters&lt;/h2&gt;
&lt;p&gt;Another unexpected problem manifested with tests failing with a message like
&amp;quot;column &lt;em&gt;name&lt;/em&gt; must appear in the GROUP BY clause or be used in an aggregate
function&amp;quot;. This type of error appeared in tests leveraging &lt;a class="reference external" href="https://docs.djangoproject.com/en/3.2/topics/db/aggregation/#topics-db-aggregation"&gt;Django ORM
aggregation features&lt;/a&gt;, when the aggregation keys contain parameters.&lt;/p&gt;
&lt;p&gt;For example, if you want to count your people by grouping them by the first
two letters of their name, you may use a query such as:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
SELECT left(name, 2) AS prefix, count(*) AS number
FROM people
GROUP BY left(name, 2)
ORDER BY left(name, 2)
&lt;/pre&gt;
&lt;p&gt;If the &amp;quot;2&amp;quot; is actually a parameter, Django ends up composing and executing a
query like:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&amp;quot;
    SELECT left(name, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;) AS prefix, count(*) AS number
    FROM people
    GROUP BY left(name, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)
    ORDER BY left(name, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)
    &amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;If composed by the client, this query presents no problem, because the server
query parser can clearly see the same expression in the output column and in
the group/order predicates. However, moving to use server-side parameters, the
query would be transformed to:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
SELECT left(name, $1) AS prefix, count(*) AS number
FROM people
GROUP BY left(name, $2)
ORDER BY left(name, $3)
&lt;/pre&gt;
&lt;p&gt;This query could be executed only if the three parameters are the same, but at
parsing time the server cannot make sure this will be the case, failing with
the error above.&lt;/p&gt;
&lt;p&gt;If this query was under someone's control, it could be easily rewritten using
named parameters instead of positional ones. I would personally write:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&amp;quot;
    SELECT left(name, &lt;/span&gt;&lt;span class="si"&gt;%(len)s&lt;/span&gt;&lt;span class="s2"&gt;) AS prefix, count(*) AS number
    FROM people
    GROUP BY left(name, &lt;/span&gt;&lt;span class="si"&gt;%(len)s&lt;/span&gt;&lt;span class="s2"&gt;)
    ORDER BY left(name, &lt;/span&gt;&lt;span class="si"&gt;%(len)s&lt;/span&gt;&lt;span class="s2"&gt;)
    &amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;len&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;which is transformed in a query with a single &lt;tt class="docutils literal"&gt;$1&lt;/tt&gt; placeholder used three
times, and parsed successfully. Unfortunately Django only uses positional
placeholders throughout its entire codebase and switching to a parameters
mapping would be a very invasive change. A more localised change is to use
column aliases: the same query can be rewritten as:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
SELECT left(name, %s) AS prefix, count(*) AS number
FROM people
GROUP BY 1
ORDER BY 1
&lt;/pre&gt;
&lt;p&gt;where &amp;quot;1&amp;quot; refers to the first output column. It is not a particularly loved
syntax but it turns out useful here.&lt;/p&gt;
&lt;p&gt;Probably not every database supports this syntax, so, in the &lt;a class="reference external" href="https://github.com/dvarrazzo/django/commit/597b2fcd9b3154bcdcb6946c73d38e62b5ca48e7"&gt;proposed Django
changeset&lt;/a&gt;, new feature parameters can be used to signal that the feature is
accepted by specific databases, and currently enabled only for PostgreSQL.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="a-different-package-organisation"&gt;
&lt;h2&gt;A different package organisation&lt;/h2&gt;
&lt;p&gt;This is less mysterious to understand. The &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; package has a bit of a
chaotic organisation, with a couple of kitchen-sink modules (&lt;tt class="docutils literal"&gt;extensions&lt;/tt&gt;
and &lt;tt class="docutils literal"&gt;extras&lt;/tt&gt;) containing a bit of everything: cursor subclasses, extra data
types, utility functions, symbolic constants...starting from Psycopg 3, the
package is organised in a more rational way and &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/"&gt;separated in different
modules and sub-packages&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The bulk of the changes required to use Psycopg 3 from Django is just not to
assume that talking to PostgreSQL will be done using &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt;, but to
import and use the objects according to the version of the driver in use.&lt;/p&gt;
&lt;p&gt;Note that, with these modifications, Django is able to use both Psycopg 2 and
3, possibly both in the same project. While this is possible, it might not be
the average use case in your project: more often than not you will be just
interested in upgrading from Psycopg 2 to 3. In the simpler case of an update,
you will just have to change your import statements, assuming unconditionally
that the &lt;tt class="docutils literal"&gt;psycopg&lt;/tt&gt; package is installed (there is no &lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt; package:
the same module name will do for the following major versions too) and just
say goodbye to the glorious workhorse that &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; has been so far. 👋&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="takeaway-points"&gt;
&lt;h2&gt;Takeaway points&lt;/h2&gt;
&lt;p&gt;In the design of Psycopg 3, a great effort has been made to allow a smooth
adoption for users who have experience, and codebases, in &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt;. The
experience of the Django backend porting shows that most of the adjustments
required fall in these categories:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Different package organisation&lt;/li&gt;
&lt;li&gt;Over/under-specified types, which can be tweaked via casts and data
wrappers&lt;/li&gt;
&lt;li&gt;Inability to use server-side parameters, which can be worked around with
client-side query composition&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Hopefully now you know how to address these problems in case you are thinking
of using Psycopg 3 in your next or current project!&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 2.9 released</title><link href="https://www.psycopg.org/articles/2021/06/16/psycopg-29-released/" rel="alternate"/><updated>2021-06-16T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:5ffbbc8a-c1c5-32ed-bdf5-6eac4b92e794</id><content type="html">&lt;p&gt;Psycopg 2.9 has been released!&lt;/p&gt;
&lt;p&gt;This is a relatively small release compared to previous major releases.
However the creation of the packages took a lot of effort. The previously used
CI system now has reduced support for free software projects - it was decided
that package building should be moved to GitHub Actions.&lt;/p&gt;
&lt;p&gt;Packaging has also become more complex because of the evolution of the Python
packaging standards and the need to support multiple architectures (Intel,
ARM, PPC...).&lt;/p&gt;
&lt;p&gt;Maintaining a project such as Psycopg requires a lot of effort. For this
reason, we are extremely grateful to all our &lt;a class="reference external" href="https://www.psycopg.org/sponsors/"&gt;sponsors&lt;/a&gt; who are enabling the
maintenance and development of Psycopg. Thank you very much! &lt;span class="raw-html"&gt;&lt;i
class="fa fa-heart"&gt;&lt;/i&gt;&lt;/span&gt;&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;What’s new in psycopg 2.9&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;with connection&lt;/tt&gt; starts a transaction on autocommit transactions too (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/941"&gt;ticket #941&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Timezones with fractional minutes are supported on Python 3.7 and following (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1272"&gt;ticket #1272&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Escape table and column names in &lt;tt class="docutils literal"&gt;copy_from()&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;copy_to()&lt;/tt&gt;.&lt;/li&gt;
&lt;li&gt;Connection exceptions with sqlstate 08XXX reclassified as &lt;tt class="docutils literal"&gt;OperationalError&lt;/tt&gt; (a subclass of the previously used &lt;tt class="docutils literal"&gt;DatabaseError&lt;/tt&gt;) (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1148"&gt;ticket #1148&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Include library dirs required from libpq to work around MacOS build problems (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1200"&gt;ticket #1200&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Other changes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Dropped support for Python 2.7, 3.4, 3.5 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1198"&gt;ticket #1198&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1000"&gt;ticket #1000&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1197"&gt;ticket #1197&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Dropped support for &lt;tt class="docutils literal"&gt;mx.DateTime&lt;/tt&gt;.&lt;/li&gt;
&lt;li&gt;Use &lt;tt class="docutils literal"&gt;datetime.timezone&lt;/tt&gt; objects by default in &lt;tt class="docutils literal"&gt;datetime&lt;/tt&gt; objects instead of &lt;tt class="docutils literal"&gt;FixedOffsetTimezone&lt;/tt&gt;.&lt;/li&gt;
&lt;li&gt;The &lt;tt class="docutils literal"&gt;psycopg2.tz&lt;/tt&gt; module is deprecated and scheduled to be dropped in the next major release.&lt;/li&gt;
&lt;li&gt;Provide &lt;a class="reference external" href="https://peps.python.org/pep-0599"&gt;PEP 599&lt;/a&gt; wheels packages (&lt;tt class="docutils literal"&gt;manylinux2014&lt;/tt&gt; tag) for &lt;tt class="docutils literal"&gt;i686&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;x86_64&lt;/tt&gt; platforms.&lt;/li&gt;
&lt;li&gt;Provide &lt;a class="reference external" href="https://peps.python.org/pep-0600"&gt;PEP 600&lt;/a&gt; wheels packages (&lt;tt class="docutils literal"&gt;manylinux_2_24&lt;/tt&gt; tag) for &lt;tt class="docutils literal"&gt;aarch64&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;ppc64le&lt;/tt&gt; platforms.&lt;/li&gt;
&lt;li&gt;Wheel package compiled against OpenSSL 1.1.1k and PostgreSQL 13.3.&lt;/li&gt;
&lt;li&gt;Build system for Linux/MacOS binary packages moved to GitHub Actions.&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Designing a connection pool for psycopg3</title><link href="https://www.psycopg.org/articles/2021/01/17/pool-design/" rel="alternate"/><updated>2021-01-17T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:8fc8d66b-6b9d-36c2-8233-36bdf56a7b08</id><content type="html">&lt;p&gt;The &lt;a class="reference external" href="https://www.psycopg.org/docs/pool.html"&gt;psycopg2 pool&lt;/a&gt; is a pretty simple object, little more than... a pool of
open connections, and I think it falls short in several ways:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;the top usability problem is the fact that it cannot be used as context
manager;&lt;/li&gt;
&lt;li&gt;if a connection is broken it is not noticed it until it is used by a client;&lt;/li&gt;
&lt;li&gt;if &lt;tt class="docutils literal"&gt;minconn&lt;/tt&gt; connections are already taken, a new one is created and
disposed of as soon as finished using, regardless of whether other clients
may need it;&lt;/li&gt;
&lt;li&gt;if more than &lt;tt class="docutils literal"&gt;maxconn&lt;/tt&gt; connections are requested the client will receive
an error.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For &lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt; I would like something better. I have read around, looking
into other pool implementations to figure out what a well designed connection
pool ought to do (a very well thought one seems the Java &lt;a class="reference external" href="https://github.com/brettwooldridge/HikariCP"&gt;HikariCP&lt;/a&gt;) and
these are a few ideas I'd like to work on: they are here for a feedback,
before I jump into enthusiastically implementing the wrong thing...&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="client-interface"&gt;
&lt;h2&gt;Client interface&lt;/h2&gt;
&lt;div class="section" id="context-manager"&gt;
&lt;h3&gt;Context manager&lt;/h3&gt;
&lt;p&gt;In modern Python it is expected that resources are handled by a context
manager, so the canonical way to use a pooled connection should be:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connection&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;...&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;consume_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Note that, because there is a &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/connection.html#psycopg3.Connection.execute"&gt;Connection.execute()&lt;/a&gt; method, the minimal use
would be:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connection&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;DML_QUERY&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="blocking-behaviour"&gt;
&lt;h3&gt;Blocking behaviour&lt;/h3&gt;
&lt;p&gt;Another important usability point is the behaviour in the face of an exhausted
pool: the sane behaviour here is to block: not forever, but until either a
connection is available or until a timeout expires, and only then throw an
exception, so that in the face of a spike there is a buffer before requests
start to get killed. I would expect no crash here:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;worker&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;work_for_sec&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select pg_sleep(&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;work_for_sec&lt;/span&gt;&lt;span class="p"&gt;,))&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AsyncPool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;maxconn&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;connection_timeout_sec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connection&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;create_task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;worker&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;work_for_sec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Which also illustrates that, as many other objects in &lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt;, there
would be an &lt;a class="reference external" href="https://docs.python.org/3/library/asyncio.html"&gt;asyncio&lt;/a&gt; version of the object and a sync one (which would work
with normal threads and with Eventlet/gevent green threads).&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="connection-configuration"&gt;
&lt;h3&gt;Connection configuration&lt;/h3&gt;
&lt;p&gt;There should be a way to configure a Python connection, for instance to
register adapters, before it is available to the application. Is subclassing
the best way to do it? Someone might find more useful to choose dynamically
what configuration to use.&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# Subclassing:&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;MyPool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;psycopg3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;configure&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;MyAdapter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;register&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;MyPool&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# Configuring:&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;my_configure&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;MyAdapter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;register&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;configure_func&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;my_configure&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="exclusive-connections"&gt;
&lt;h3&gt;Exclusive connections?&lt;/h3&gt;
&lt;p&gt;Should there be a way to create an exclusive connection? Sometimes I need
one, in an otherwise pooled application, e.g. to receive notifications:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;listen_notifications&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exclusive&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;autocommit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;notify&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notifies&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;handle_notification&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;notify&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;But I don't think there is such need if the connection parameters are made
available on the pool:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;pool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;maxconn&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pool_maxconn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db_dsn&lt;/span&gt;&lt;span class="p"&gt;,))&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;listen_notifications&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;autocommit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;notify&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;notifies&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;handle_notification&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;notify&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;which leaves to the user the choice between what connection class to use,
how to configure it, etc. So that's probably not so useful.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="internal-behaviour"&gt;
&lt;h2&gt;Internal behaviour&lt;/h2&gt;
&lt;div class="section" id="pool-worker"&gt;
&lt;h3&gt;Pool worker&lt;/h3&gt;
&lt;p&gt;If we desire the pool to be a fast provider of connection, not slowing down
the application operations, certain things should happen behind the scene:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;do we have a spike, so we need more connections?&lt;/li&gt;
&lt;li&gt;do we have a moment of calm, so we can do with less connections?&lt;/li&gt;
&lt;li&gt;are the connections in the pool still sane?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A trivial behaviour would be something like &amp;quot;if there are no connections
available create a new one&amp;quot;:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;getconn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_get_a_connection_from_the_pool&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_put_it_in_the_pool_when_done&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This &lt;a class="reference external" href="https://github.com/brettwooldridge/HikariCP/blob/dev/documents/Welcome-To-The-Jungle.md"&gt;interesting article&lt;/a&gt; shows that this might not be the best way to do
it, especially if the connection time is particularly long compared to the
processing time. Growing the pool could be demanded to a background worker,
following a strategy like:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;getconn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_pool&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;worker&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_a_new_connection_and_put_it_in_the_pool&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;wait_for_a_connection_from_the_pool&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;If there is a workers infrastructure in place there can me more jobs for them:
periodically checking for the state of the unused connections in the pool, for
instance, or closing them if they have been alive more than a configured
amount of time (30 minutes?) and replacing them with a fresh one, (or not
replacing them in case we have more than &lt;tt class="docutils literal"&gt;minconn&lt;/tt&gt; connections and a
previous spike has now passed).&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="what-to-do-in-case-of-connection-failure"&gt;
&lt;h3&gt;What to do in case of connection failure?&lt;/h3&gt;
&lt;p&gt;The pool acts as a shield helping the application to reconnect in case the
connection with the database is lost. However, when does this behaviour stop
being sane? If the database is unreachable - maybe misconfigured, maybe moved
- there is a risk that the application doesn't die, because the pool keeps
insisting to reconnect, but it doesn't work either, because there is no
working database connection.&lt;/p&gt;
&lt;p&gt;In my experience with live systems this can be a difficult condition to
diagnose: several times I have wished that an application, screaming loud on a
dashboard, instead of trying to stay up complaining quietly in some logging
stream. On the other hand maybe the state the application holds is precious
enough that it can be worth to wait a few minutes before crashing and burning.&lt;/p&gt;
&lt;p&gt;A first step towards a sane behaviour could be to die early, on startup, if
the connection is not working when the first pool population is done: the
program would fail hard and fast if the database - assumed to be a
prerequisite - is nowhere to be found.&lt;/p&gt;
&lt;p&gt;What if the database is missing in action during the program lifetime?
Following connection attempts may be repeated, with an exponential backoff,
until dying after a few minutes of fruitless attempts (with a &lt;tt class="docutils literal"&gt;sys.exit(1)&lt;/tt&gt;
or some other termination function which might be subclassed).&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="connections-usage-pattern"&gt;
&lt;h3&gt;Connections usage pattern&lt;/h3&gt;
&lt;p&gt;In which order should the connections in the pool be used? As a stack (put
back a connection on top of the stack, it will be the next one to use)? As a
queue (put back the connection at the bottom of the queue, try to use them
uniformly)? Randomly (whatever comes out from whatever hash map used to keep
the connection)?&lt;/p&gt;
&lt;p&gt;Is there a reason to prefer one way or the other? ISTM that a stack behaviour
allows a better reuse of prepared statements and an easy implementation of
the &lt;tt class="docutils literal"&gt;max_idle_sec&lt;/tt&gt; parameter (get connections from the top of the stack,
evict idle ones from the bottom). I am sure I haven't thought about this well
enough though.&lt;/p&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="proposed-api"&gt;
&lt;h2&gt;Proposed API&lt;/h2&gt;
&lt;p&gt;Given the behaviour described, the methods available on the pool object might
look like the ones described here.&lt;/p&gt;
&lt;p&gt;The interface would be implemented in two version: a &lt;tt class="docutils literal"&gt;Pool&lt;/tt&gt; class
using normal blocking functions and threads for concurrency (which greenlet
libraries such as Eventlet or gevent would be able to monkeypatch) and an
&lt;tt class="docutils literal"&gt;AsyncPool&lt;/tt&gt; implementation using the usual &lt;tt class="docutils literal"&gt;await&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;async with&lt;/tt&gt; on
blocking methods and using &lt;tt class="docutils literal"&gt;asyncio.Task&lt;/tt&gt; for concurrency.&lt;/p&gt;
&lt;dl class="docutils"&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;connection(timeout_sec=None)&lt;/tt&gt; method:&lt;/dt&gt;
&lt;dd&gt;&lt;p class="first"&gt;Open a context block and return a connection from the pool. The connection
is returned to the pool at the end of the block.&lt;/p&gt;
&lt;p class="last"&gt;On block exit, if a transaction is open, &lt;strong&gt;commit or roll back&lt;/strong&gt; an open
transaction, according to whether an exception has been raised in the
block, consistently with what the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/transactions.html#transaction-blocks"&gt;connection block&lt;/a&gt; does.&lt;/p&gt;
&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;getconn(timeout_sec)&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;putconn(conn)&lt;/tt&gt; methods:&lt;/dt&gt;
&lt;dd&gt;&lt;p class="first"&gt;Obtain a connection from the pool and return it. To use if, for some
reason, the context manager cannot be used.&lt;/p&gt;
&lt;p class="last"&gt;On &lt;tt class="docutils literal"&gt;putconn()&lt;/tt&gt; check the state of the connection: if it is broken
dispose of it and create a new one; if in state of transaction of error
&lt;strong&gt;roll back the transaction&lt;/strong&gt;, consistently with what
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/connection.html#psycopg3.Connection.close"&gt;Connection.close()&lt;/a&gt; does.&lt;/p&gt;
&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;close()&lt;/tt&gt;&lt;/dt&gt;
&lt;dd&gt;Close all the connections in the pool. Further attempts to create a new
connection will fail immediately with a helpful message (like, &amp;quot;dude,
what do you want to do, seriously?&amp;quot;)&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;configure(conn)&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Configure a connection before making it available to the pool. The default
implementation is no-op: subclasses may override it to configure the
connections; alternatively a &lt;tt class="docutils literal"&gt;configure_func&lt;/tt&gt; might be passed to the
&lt;tt class="docutils literal"&gt;Pool&lt;/tt&gt; constructor.&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;get_info()&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Return stats about the behaviour of the pool so far (connections open,
reused, returned...) for monitoring.&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;terminate()&lt;/tt&gt;&lt;/dt&gt;
&lt;dd&gt;Terminate the program, invoked by a pool worker after connection attempts
have been repeatedly fruitless. It is exposed to allow overriding the
default behaviour (&lt;tt class="docutils literal"&gt;sys.exit(1)&lt;/tt&gt;) in a subclass.&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;get_maintenance_task()&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;&lt;p class="first"&gt;Get the next maintenance task to perform on the pool. Having them
available externally would make possible to have full control of when such
tasks are executed (useful for testing or to provide some sort of
sync behaviour, using no background worker).&lt;/p&gt;
&lt;p class="last"&gt;Experimental idea: will drop it if it would make the pool implementation
more difficult than it should be.&lt;/p&gt;
&lt;/dd&gt;
&lt;/dl&gt;
&lt;div class="section" id="key-configuration-parameters"&gt;
&lt;h3&gt;Key configuration parameters&lt;/h3&gt;
&lt;p&gt;These parameter would be passed to the class constructor.&lt;/p&gt;
&lt;dl class="docutils"&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;minconn&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Minimum number of connections to keep open in the pool. If some are closed
the pool should try to create new ones as quickly as possible to replace
them. Proposed default: 4 (very defensive: to enable the pooling behaviour
but to avoid to saturate a server unless configured up).&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;maxconn&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Maximum number of connections to open at any given time. If &lt;tt class="docutils literal"&gt;maxconn&lt;/tt&gt; ==
&lt;tt class="docutils literal"&gt;minconn&lt;/tt&gt; no extra connection is created in case more client requests
arrive: they will just wait until a connection is made available again. If
&lt;tt class="docutils literal"&gt;maxconn&lt;/tt&gt; &amp;gt; &lt;tt class="docutils literal"&gt;minconn&lt;/tt&gt; the pool can create new connections if the demand
increases; later the extra connections created may be closed if deemed no
more required (default: &lt;tt class="docutils literal"&gt;minconn&lt;/tt&gt;).&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;args&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;kwargs&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Arguments to pass to the &lt;tt class="docutils literal"&gt;connection_factory&lt;/tt&gt; to create a new connection
(default: empty, connect as per &lt;a class="reference external" href="https://www.postgresql.org/docs/current/libpq-envars.html"&gt;PG* env vars&lt;/a&gt;, like
&lt;tt class="docutils literal"&gt;psycopg3.connect()&lt;/tt&gt;).&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;connection_factory&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;The connection class to create (default: &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/connection.html#psycopg3.Connection"&gt;Connection&lt;/a&gt; for &lt;tt class="docutils literal"&gt;Pool&lt;/tt&gt;,
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/connection.html#psycopg3.AsyncConnection"&gt;AsyncConnection&lt;/a&gt; for &lt;tt class="docutils literal"&gt;AsyncPool&lt;/tt&gt;).&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;configure_function&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;A function to configure the connection after its creation and before making
it available to the pool. Alternative to subclassing the &lt;tt class="docutils literal"&gt;Pool&lt;/tt&gt; class to
configure &lt;tt class="docutils literal"&gt;Pool.configure()&lt;/tt&gt;. It can be a callable taking a connection as
parameter, or the dotted name of such callable, so that e.g. the function
name might be passed to the application as an env var.&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;timeout_sec&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Default timeout before raising an exception if a connection
cannot be served. It may be overridden by
&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;Pool.connection(timeout_sec=...)&lt;/span&gt;&lt;/tt&gt; (default: 30 sec).&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;max_lifetime_sec&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Maximum time a connection should be kept in the pool
and used. After such time, when the connection is not in use, it can be
closed and replaced by a new one (default: 30 minutes - 10% random factor to
avoid mass evictions).&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;max_idle_sec&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Time a connection can sit idle in the pool before being removed. Only
connections above &lt;tt class="docutils literal"&gt;minconn&lt;/tt&gt; are removed, if &lt;tt class="docutils literal"&gt;maxconn&lt;/tt&gt; allows to create
them (default: 10 min + 10% random).&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;terminate_after_sec&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Number of seconds to wait for a reconnection attempt before giving up and
terminate the program, calling &lt;tt class="docutils literal"&gt;Pool.terminate()&lt;/tt&gt; (default: 5 min,
starting from 1sec ± 10%, doubled up until the precise threshold, then
kaputt).&lt;/dd&gt;
&lt;dt&gt;&lt;tt class="docutils literal"&gt;num_workers&lt;/tt&gt;:&lt;/dt&gt;
&lt;dd&gt;Number of background workers to perform maintenance tasks. If set to 0 the
dynamic characteristics of the pool might be downgraded (e.g. creating a new
connection will happen in the requesting thread, blocking it for the time it
takes). Background jobs might be executed by the application calling
&lt;tt class="docutils literal"&gt;get_maintenance_task()&lt;/tt&gt; (default: 3).&lt;/dd&gt;
&lt;/dl&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;div class="section" id="thoughts"&gt;
&lt;h2&gt;Thoughts?&lt;/h2&gt;
&lt;p&gt;Please let me know what you think your Best Ever Connection Pool for psycopg3
should do. Thank you very much!&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>The psycopg3 adaptation system</title><link href="https://www.psycopg.org/articles/2020/11/24/psycopg3-adaptation/" rel="alternate"/><updated>2020-11-24T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:eef60882-aee1-318d-9e8b-771bfe1d5026</id><content type="html">&lt;p&gt;The adaptation system between Python objects and PostgreSQL types is at the
core of &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt;. The flexibility of the &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt;
adaptation system provides &lt;a class="reference external" href="https://www.psycopg.org/docs/usage.html#adaptation-of-python-values-to-sql-types"&gt;good out-of-the-box object mapping&lt;/a&gt; and allows
users to &lt;a class="reference external" href="https://www.psycopg.org/docs/advanced.html#adapting-new-python-types-to-sql-syntax"&gt;customise it to suit any need&lt;/a&gt;. Do you want your &lt;tt class="docutils literal"&gt;decimal&lt;/tt&gt;
numbers returned as &lt;tt class="docutils literal"&gt;float&lt;/tt&gt; because you need speed over pennies? Do you want
to map PostgreSQL Infinity dates to the 25th of December 3099? That's
certainly doable.&lt;/p&gt;
&lt;blockquote&gt;
&lt;/blockquote&gt;
&lt;p&gt;The &lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt; adaptation system needs some modification compared to
&lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt;, because &lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt; uses the &amp;quot;extended query protocol&amp;quot; to send
query parameters separately from the query. Together, with the differences to
accommodate, there is also a chance to improve a system that has been in use
for several years and has shown its shortcomings together with its strengths.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="server-side-binding"&gt;
&lt;h2&gt;Server-side binding&lt;/h2&gt;
&lt;p&gt;Server-side parameter binding has been a long-time desired feature. So far
&lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; has adapted arguments on the client-side:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# psycopg2&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;INSERT INTO tbl (s, n, d) VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Hel'lo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2020&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# has arguments adapted and quoted:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;42&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;'Hel''lo'&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;'2020-12-31'::date&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# merged to the query:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# and passed as a single string:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;libpq&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PQexec&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;INSERT INTO tbl (s, n, d) VALUES (42, 'Hel''lo', '2020-12-31'::date)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The &lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt; end user interface is unchanged, but behind the scenes will only
perform adaptation, not quoting, and will send the arguments separately:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# psycopg3&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;INSERT INTO tbl (s, n, d) VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Hel'lo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2020&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# has arguments adapted:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;42&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Hel'lo&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;2020-12-31&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# and passed as separate information:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;libpq&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PQexecParams&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;INSERT INTO tbl (s, n, d) VALUES ($1, $2, $3)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# Postgres placeholders&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;42&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Hel'lo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;2020-12-31&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;                   &lt;span class="c1"&gt;# Postgres formats, no quoting&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;oids&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;INT8_OID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DATE_OID&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;                     &lt;span class="c1"&gt;# Type indications if available&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Server-side binding brings better performance, the possibility to use prepared
statements and binary data, as well as better integration with server-side
logging and monitoring. Theoretically it also brings better safety against SQL
injections, but &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; already does a good job at &lt;a class="reference external" href="https://www.psycopg.org/docs/usage.html#the-problem-with-the-query-parameters"&gt;providing a safe path
for parameter binding&lt;/a&gt;: in &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt;, creating an unsafe query is already
harder than doing things the right way.&lt;/p&gt;
&lt;p&gt;However server-side binding brings a few incompatibilities, such as:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p class="first"&gt;cannot send more than one query at once if parameters are used:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# Must use separate queries or psycopg3.sql client-side adaptation&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;INSERT INTO tbl1 VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;); INSERT INTO tbl2 VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/pre&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;cannot use certain commands such as &lt;tt class="docutils literal"&gt;SET&lt;/tt&gt; or &lt;tt class="docutils literal"&gt;NOTIFY&lt;/tt&gt; with parameters:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# Must use &amp;quot;SELECT set_config('timezone', %s)&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SET timezone to &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# Must use &amp;quot;SELECT pg_notify('channel', %s)&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NOTIFY channel, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/pre&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;cannot use the &lt;tt class="docutils literal"&gt;IN &lt;span class="pre"&gt;(...)&lt;/span&gt;&lt;/tt&gt; construct:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# Must use &amp;quot;AND nation = any (%s)&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;... AND nation in &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;IT&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;FR&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;DE&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;
&lt;/pre&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All in all, most queries will just work, however the few incompatibilities
require a non-backward-compatible change in version.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="a-new-adaptation-system"&gt;
&lt;h2&gt;A new adaptation system&lt;/h2&gt;
&lt;p&gt;Embracing the new type of communication requires to change the way Python
parameters and PostgreSQL data types are adapted. The new system:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p class="first"&gt;cannot use the whole SQL syntax, but must limit to literals. For instance
the Python list &lt;tt class="docutils literal"&gt;[&amp;quot;foo, bar&amp;quot;, &amp;quot;Hel'lo&amp;quot;]&lt;/tt&gt; cannot be expressed with
&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;ARRAY['foo,&lt;/span&gt; bar', &lt;span class="pre"&gt;'Hel''lo']&lt;/span&gt;&lt;/tt&gt; but must use array literal rules and become
&lt;tt class="docutils literal"&gt;{&amp;quot;foo, &lt;span class="pre"&gt;bar&amp;quot;,Hel'lo}&lt;/span&gt;&lt;/tt&gt;.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;Makes possible to specify type OIDs, in case that's useful (&lt;a class="reference external" href="https://www.varrazzo.com/blog/2020/11/07/psycopg3-adaptation/"&gt;not always an
easy choice&lt;/a&gt;).&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p class="first"&gt;Allows for the use of binary types, which is especially useful for a large
binary blob, otherwise things become bloated by &lt;a class="reference external" href="https://www.postgresql.org/docs/current/datatype-binary.html"&gt;binary escaping&lt;/a&gt; and have
to traverse the many layers of lexing/parsing in the server, each with its
own memory copy. This can be done by simply using the &lt;tt class="docutils literal"&gt;%b&lt;/tt&gt; placeholder
over &lt;tt class="docutils literal"&gt;%s&lt;/tt&gt;:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;image_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;rb&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="s2"&gt;&amp;quot;INSERT INTO images (name, data) VALUES (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, %b)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;image_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;()])&lt;/span&gt;
&lt;/pre&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;It's also a good chance to review the work that must be done by the client to
adapt values. &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; creates several instances of &amp;quot;adapter&amp;quot; wrappers,
one for each value adapted. In &lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt;, the adaptation objects have a
different life cycle: choices based on the environment, such as the connection
encoding, can be made once for each Python type, rather than once per value,
doing radically less work for each converted object. You can check out the
&lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/adapt.html"&gt;psycopg3 adaptation documentation&lt;/a&gt; for all the details.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="customising-psycopg3-adaptation"&gt;
&lt;h2&gt;Customising psycopg3 adaptation&lt;/h2&gt;
&lt;p&gt;Customising types adaptation can now be done using &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/adapt.html#psycopg3.adapt.Dumper"&gt;Dumper&lt;/a&gt; classes, either
creating new ones or mapping existing ones on different Python classes to save
to the database. For instance, the builtin &lt;tt class="docutils literal"&gt;DateDumper&lt;/tt&gt; converts Python
dates to PostgreSQL ones. PostgreSQL can handle an &amp;quot;infinity&amp;quot; date, which
Python cannot. If we wanted to store Python's &lt;tt class="docutils literal"&gt;date.max&lt;/tt&gt; to PostgreSQL
infinity, we could create a subclass for the dumper and register it in the
scope we want to use it, globally or just on a connection or cursor:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;InfDateDumper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;DateDumper&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;infinity&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;::text, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;::text&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2020&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;# ('2020-12-31', '9999-12-31')&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;InfDateDumper&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;register&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SELECT &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;::text, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;::text&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2020&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;# ('2020-12-31', 'infinity')&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The system is pretty symmetric and employs similar &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/advanced/adapt.html#psycopg3.adapt.Loader"&gt;Loader&lt;/a&gt; objects to map OIDs
to the code responsible for its decoding. For instance, if we wanted to
reverse the above customisation and map PostgreSQL infinity date to
&lt;tt class="docutils literal"&gt;date.max&lt;/tt&gt; (instead of raising an exception), it could be done using a
subclass of the builtin loader (or using an entirely new object if required):&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;InfDateLoader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;DateLoader&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;infinity&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select '2020-12-31'::date, 'infinity'::date&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;# Raises DataError: Python date doesn't support years after 9999: got infinity&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg3.oids&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;builtins&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;InfDateLoader&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;register&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;builtins&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;date&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select '2020-12-31'::date, 'infinity'::date&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2020&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9999&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The customisation automatically applies to recursive types, such as arrays or
composite types: if the date loader is customised then the date array works as
expected:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select '{2020-12-31,infinity}'::date[]&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2020&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9999&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;31&lt;/span&gt;&lt;span class="p"&gt;)],)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;All in all the new adaptation system provides better performance and easier
customisation &lt;a class="reference external" href="https://www.psycopg.org/docs/advanced.html#adapting-new-python-types-to-sql-syntax"&gt;compared to psycopg2&lt;/a&gt;. The new adapters are easier to
compose, such as using them in &lt;a class="reference external" href="/articles/2020/11/15/psycopg3-copy/"&gt;COPY operations&lt;/a&gt;. And if client-side adaptation is still needed (to
generate dynamically data definition statements, to prepare offline update
scripts...) &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/sql.html"&gt;psycopg3.sql&lt;/a&gt; still allows for the flexibility of client-side
query composition.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="project-sponsorship"&gt;
&lt;h2&gt;Project sponsorship&lt;/h2&gt;
&lt;div class="sponsor docutils container"&gt;
&lt;p&gt;The new adaptation system is one of several new features that are being
designed and implemented in &lt;a class="reference external" href="/psycopg3/"&gt;psycopg3&lt;/a&gt;. The project is currently &lt;a class="reference external" href="https://github.com/psycopg/psycopg3"&gt;under
active development&lt;/a&gt;: if there is enough support it will be possible to
work at the project full-time and bring it to release swiftly.&lt;/p&gt;
&lt;p&gt;If you use Python and PostgreSQL, and you would like to support the
creation of the most advanced adapter between the two systems, &lt;a class="reference external" href="https://github.com/sponsors/psycopg/"&gt;please
consider becoming a sponsor&lt;/a&gt;. Thank you!&lt;/p&gt;
&lt;iframe src="https://github.com/sponsors/psycopg/button"
    title="Sponsor the project" style="border: 0"
    width="116" height="35"&gt;
&lt;/iframe&gt;&lt;/div&gt;
&lt;/div&gt;
</content></entry><entry><title>The new COPY support in psycopg3</title><link href="https://www.psycopg.org/articles/2020/11/15/psycopg3-copy/" rel="alternate"/><updated>2020-11-15T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:67b93afe-21e3-30fb-8a32-4dbaf5daaa4e</id><content type="html">&lt;p&gt;&lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; allows &lt;a class="reference external" href="https://www.psycopg.org/docs/usage.html#copy"&gt;interaction with PostgreSQL COPY commands&lt;/a&gt;. However
what is possible to do with them is relatively limited: the only possible
interaction is with file-like objects:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;there is no adaptation from Python objects to PostgreSQL, as there is for
normal queries: data must be formatted &amp;quot;manually&amp;quot; by the user;&lt;/li&gt;
&lt;li&gt;psycopg2 &amp;quot;pulls&amp;quot; data from the file: writing a system that produces data and
pushes it into PostgreSQL is a very contrived operation, requiring to write
a blocking file-like object;&lt;/li&gt;
&lt;li&gt;there is no support for &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/428"&gt;asynchronous copy&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;a class="reference external" href="/psycopg3/"&gt;psycopg3&lt;/a&gt; &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/basic/copy.html#copy"&gt;addresses these shortcomings&lt;/a&gt; and makes it easy to write Python
programs producing data and pushing it efficiently to the database using the
&lt;tt class="docutils literal"&gt;COPY&lt;/tt&gt; protocol.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; adaptation system is designed to compose queries client-side, so
it is concerned with the right use of the quotes: the python string
&lt;tt class="docutils literal"&gt;O'Reilly&lt;/tt&gt; is converted to &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;'O''Reilly'&lt;/span&gt;&lt;/tt&gt;, and the &lt;tt class="docutils literal"&gt;date(2020, 11, 15)&lt;/tt&gt;
to &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;'2020-11-15'::date&lt;/span&gt;&lt;/tt&gt;. These extra quotes get in the way of COPY, and
there isn't an intermediate level where a conversion to string is performed,
but no quote or other SQL construct are added.&lt;/p&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;psycopg3&lt;/tt&gt; uses the PostgreSQL extended query protocol and sends query and
parameters separately. Parameters require adaptation to the PostgreSQL
formats, but quoting, and quotes escaping, are no more its concern: the string
&lt;tt class="docutils literal"&gt;O'Reilly&lt;/tt&gt; doesn't need further manipulation and the date is converted only
to the string &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;2020-11-15&lt;/span&gt;&lt;/tt&gt;; types information are passed as additional
separate information according to &lt;a class="reference external" href="https://www.postgresql.org/docs/current/libpq-exec.html#LIBPQ-PQEXECPARAMS"&gt;the libpq API&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The server-side format of these values are exactly what the &lt;tt class="docutils literal"&gt;COPY FROM&lt;/tt&gt;
command expects, so it's now easy to compose a row by adapting Python objects
and to pass&lt;/p&gt;
&lt;pre class="literal-block"&gt;
O'Reilly\t2020-11-15\n
&lt;/pre&gt;
&lt;p&gt;to the server. The mechanism to do so is exposed to Python by a new context
manager, returned by the &lt;a class="reference external" href="https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.copy"&gt;Cursor.copy()&lt;/a&gt; method, which enables to write:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;COPY mytable FROM STDIN&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write_row&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;O'Reilly&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2020&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Any list of tuples of values, or generator of sequences of values, can be used
to push data into Postgres:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;COPY mytable FROM STDIN&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;my_generator&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write_row&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The copy operation is concluded as soon as the &lt;tt class="docutils literal"&gt;with&lt;/tt&gt; block is exited and,
in case a Python exception is raised, the error is pushed to the server, which
will cancel the COPY operation in progress.&lt;/p&gt;
&lt;div class="section" id="binary-format"&gt;
&lt;h2&gt;Binary format&lt;/h2&gt;
&lt;p&gt;The &lt;tt class="docutils literal"&gt;Copy&lt;/tt&gt; object is also able to write data in &lt;a class="reference external" href="https://www.postgresql.org/docs/13/sql-copy.html#id-1.9.3.55.9.4"&gt;binary format&lt;/a&gt;: at Python
level this is entirely transparent:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;COPY mytable FROM STDIN (FORMAT BINARY)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;generator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write_row&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;which might be more efficient than the textual format, but requires more care
with the data types, as the server will not even perform an innocent
&lt;tt class="docutils literal"&gt;int4&lt;/tt&gt; -&amp;gt; &lt;tt class="docutils literal"&gt;int8&lt;/tt&gt; cast for you.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="block-level-copy"&gt;
&lt;h2&gt;Block-level COPY&lt;/h2&gt;
&lt;p&gt;&lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; allows (only) to operate on a COPY stream using a Python
file-like objects: behind the scenes it reads one block of data from the
source and writes it to the destination:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="c1"&gt;# From a file to the database&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;input.data&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cursor2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy_expert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;COPY mytable FROM STDIN&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# From the database to a file&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;output.data&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;w&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cursor2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy_expert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;COPY mytable TO STDOUT&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;This way of operating is not lost, but now the responsibility of moving data
from one stream to the other is left to the user's code:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;input.data&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;cursor3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;COPY mytable FROM STDIN&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SIZE&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;cursor3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;COPY mytable TO STDOUT&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;output.data&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;wb&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;While the new pattern is more verbose, it allows to produce and consume data
with interfaces different than the file one, whereas previously it would have
required to write some form of file-like adapter, blocking the copy in case no
data was ready. This inversion of control allows, finally, the use of...&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="asynchronous-copy"&gt;
&lt;h2&gt;Asynchronous COPY&lt;/h2&gt;
&lt;p&gt;If your data producer, either at rows level or at blocks level, is capable of
asynchronous operations, it is now possible to combine it asynchronously with
COPY using exactly the same pattern as the sync code, only sprinkling the
magic words here and there:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;COPY mytable FROM STDIN (FORMAT BINARY)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;my_async_generator&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write_row&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;async_producer&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;COPY mytable FROM STDIN&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;which covers an important use case pretty much impossible to introduce in
&lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="project-sponsorship"&gt;
&lt;h2&gt;Project sponsorship&lt;/h2&gt;
&lt;div class="sponsor docutils container"&gt;
&lt;p&gt;The new COPY support is one of several new features that are being
designed and implemented in &lt;a class="reference external" href="/psycopg3/"&gt;psycopg3&lt;/a&gt;. The project is currently &lt;a class="reference external" href="https://github.com/psycopg/psycopg3"&gt;under
active development&lt;/a&gt;: if there is enough support it will be possible to
work at the project full-time and bring it to release swiftly.&lt;/p&gt;
&lt;p&gt;If you use Python and PostgreSQL, and you would like to support the
creation of the most advanced adapter between the two systems, &lt;a class="reference external" href="https://github.com/sponsors/psycopg/"&gt;please
consider becoming a sponsor&lt;/a&gt;. Thank you!&lt;/p&gt;
&lt;iframe src="https://github.com/sponsors/psycopg/button"
    title="Sponsor the project" style="border: 0"
    width="116" height="35"&gt;
&lt;/iframe&gt;&lt;/div&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 2.8.6 released</title><link href="https://www.psycopg.org/articles/2020/09/06/psycopg-286-released/" rel="alternate"/><updated>2020-09-06T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:95d68852-ec64-3745-82b4-defa6188cd5e</id><content type="html">&lt;p&gt;Psycopg 2.8.6 has been released.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;You can download the files from the new release &lt;a class="reference external" href="https://pypi.org/project/psycopg2/2.8.6/#files"&gt;from PyPI&lt;/a&gt; or install it
with:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
pip install --upgrade psycopg2
&lt;/pre&gt;
&lt;p&gt;The changes included in the release are:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed memory leak changing connection encoding to the current one
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1101"&gt;ticket #1101&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed search of mxDateTime headers in virtualenvs (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/996"&gt;ticket #996&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added missing values from errorcodes (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1133"&gt;ticket #1133&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;cursor.query&lt;/tt&gt; reports the query of the last &lt;tt class="docutils literal"&gt;COPY&lt;/tt&gt; operation too
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1141"&gt;ticket #1141&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;errorcodes&lt;/tt&gt; map and &lt;tt class="docutils literal"&gt;errors&lt;/tt&gt; classes updated to PostgreSQL 13.&lt;/li&gt;
&lt;li&gt;Wheel package compiled against OpenSSL 1.1.1g.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Thank you very much to everyone who gave their contribution!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.8.5 released</title><link href="https://www.psycopg.org/articles/2020/04/06/psycopg-285-released/" rel="alternate"/><updated>2020-04-06T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:0d6cc67c-2767-3298-b069-34bc493f8e6b</id><content type="html">&lt;p&gt;Psycopg 2.8.5 has been released.&lt;/p&gt;
&lt;p&gt;This release adds support for AIX and brings a few bug fixes.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;You can download the files from the new release &lt;a class="reference external" href="https://pypi.org/project/psycopg2/2.8.5/#files"&gt;from PyPI&lt;/a&gt; or install it
with:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
pip install --upgrade psycopg2
&lt;/pre&gt;
&lt;p&gt;The changes included in the release are:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed use of &lt;tt class="docutils literal"&gt;connection_factory&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;cursor_factory&lt;/tt&gt; together
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1019"&gt;ticket #1019&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added support for &lt;tt class="docutils literal"&gt;logging.LoggerAdapter&lt;/tt&gt; in
&lt;tt class="docutils literal"&gt;LoggingConnection&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1026"&gt;ticket #1026&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;Column&lt;/tt&gt; objects in &lt;tt class="docutils literal"&gt;cursor.description&lt;/tt&gt; can be sliced (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1034"&gt;ticket #1034&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added AIX support (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1061"&gt;ticket #1061&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;copy()&lt;/tt&gt; of &lt;tt class="docutils literal"&gt;DictCursor&lt;/tt&gt; rows (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/1073"&gt;ticket #1073&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.8.4 released</title><link href="https://www.psycopg.org/articles/2019/10/20/psycopg-284-released/" rel="alternate"/><updated>2019-10-20T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:f1c59860-afc2-3a75-bd1f-55eceed64247</id><content type="html">&lt;p&gt;Psycopg 2.8.4 has been released.&lt;/p&gt;
&lt;p&gt;The release brings a few assorted bugfixes and adds support for Python 3.8 and PostgreSQL 12.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;A more detailed changes list is&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed building with Python 3.8 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/854"&gt;ticket #854&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Don't swallow keyboard interrupts on connect when a password is specified
in the connection string (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/898"&gt;ticket #898&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Don't advance replication cursor when the message wasn't confirmed
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/940"&gt;ticket #940&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed inclusion of &lt;tt class="docutils literal"&gt;time.h&lt;/tt&gt; on linux (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/951"&gt;ticket #951&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed int overflow for large values in &lt;tt class="docutils literal"&gt;Column.table_oid&lt;/tt&gt;
and &lt;tt class="docutils literal"&gt;Column.type_code&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/961"&gt;ticket #961&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;errorcodes&lt;/tt&gt; map and &lt;tt class="docutils literal"&gt;errors&lt;/tt&gt; classes updated to
PostgreSQL 12.&lt;/li&gt;
&lt;li&gt;Wheel package compiled against OpenSSL 1.1.1d and PostgreSQL at least 11.4.&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Psycopg 2.8.3 released</title><link href="https://www.psycopg.org/articles/2019/06/14/psycopg-283-released/" rel="alternate"/><updated>2019-06-14T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:eaa9eab2-886c-35c0-8ce5-5f2d6cc2f665</id><content type="html">&lt;p&gt;We have released Psycopg 2.8.3, which includes a slight change to the logical replication.&lt;/p&gt;
&lt;p&gt;Choosing the right frequency to send replication feedback messages from the client to the server was previously the developer's responsibility, with too many feedback messages being a waste of bandwidth and server resources, too few slowing down WAL cleanup and possibly preventing a server graceful shutdown.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Psycopg will now make sure that feedback is only sent after a certain period of time from the previous one, so that the client can simply call &lt;tt class="docutils literal"&gt;send_feedback()&lt;/tt&gt; at each message without the fear of overwhelming the server.&lt;/p&gt;
&lt;p&gt;Further details are available in &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/pull/913"&gt;the MR&lt;/a&gt; by &lt;a class="reference external" href="https://github.com/CyberDem0n"&gt;Alexander Kukushkin&lt;/a&gt; and &lt;a class="reference external" href="https://github.com/a1exsh"&gt;Oleksandr Shulgin&lt;/a&gt;. Thank you very much!&lt;/p&gt;
&lt;p&gt;For completeness, the changes included in the release are:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added &lt;em&gt;interval_status&lt;/em&gt; parameter to &lt;tt class="docutils literal"&gt;start_replication()&lt;/tt&gt; method and other facilities to send automatic replication keepalives at periodic intervals (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/913"&gt;ticket #913&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed namedtuples caching introduced in 2.8 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/928"&gt;ticket #928&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Psycopg 2.8.1, 2.8.2 released</title><link href="https://www.psycopg.org/articles/2019/04/14/psycopg-281-282-released/" rel="alternate"/><updated>2019-04-14T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:35398f6f-d9ff-3e0c-bbfd-1c0172c42437</id><content type="html">&lt;p&gt;Hello,&lt;/p&gt;
&lt;p&gt;We have just released Psycopg 2.8.2; a few days ago Psycopg 2.8.1 was released.&lt;/p&gt;
&lt;p&gt;Some of the bugs addressed are ordinary teething problem with the 2.8
release, but an important change landed with 2.8.2: binary packages
now ship with OpenSSL 1.1 instead of 1.0. This should fix concurrency
problems on connection experienced both on Windows and Linux. Many
thanks to Matthew Brett and Jason Erickson for this improvement!&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;The combined list of changes is:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed &amp;quot;there's no async cursor&amp;quot; error polling a connection with no cursor
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/887"&gt;ticket #887&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;RealDictCursor&lt;/tt&gt; when there are repeated columns (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/884"&gt;ticket #884&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;RealDictRow&lt;/tt&gt; modifiability (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/886"&gt;ticket #886&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Binary packages built with OpenSSL 1.1.1b. Should fix concurrency problems
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/543"&gt;ticket #543&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/836"&gt;ticket #836&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.8 released</title><link href="https://www.psycopg.org/articles/2019/04/04/psycopg-28-released/" rel="alternate"/><updated>2019-04-04T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:51a61d12-fff7-38d4-9d87-3210fac55ec9</id><content type="html">&lt;p&gt;After about two years from the previous major release, psycopg 2.8 is finally here!&lt;/p&gt;
&lt;p&gt;Among the highlights, &lt;a class="reference external" href="/docs/errors.html"&gt;PostgreSQL errors are now mapped to Python exceptions&lt;/a&gt; for a more idiomatic way to handle them. Several additions allow a better insight of the &lt;a class="reference external" href="/docs/extensions.html#psycopg2.extensions.ConnectionInfo"&gt;connection status&lt;/a&gt; and &lt;a class="reference external" href="/docs/extensions.html#psycopg2.extensions.Column"&gt;query results&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Behind the scene, asynchronous communication and concurrency received several improvements, and dropping support for older versions of Python gave the chance to refactor and modernise the codebase (with the especial help from Jon Dufresne who ruthlessly butchered our code into a streamlined pulp).&lt;/p&gt;
&lt;p&gt;Thank you very much to everyone contributing so far. Happy hacking!&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;You can download as usual from the canonical urls:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.8.tar.gz"&gt;source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.8.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Or just use &lt;tt class="docutils literal"&gt;pip&lt;/tt&gt;:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
pip install psycopg2
&lt;/pre&gt;
&lt;p&gt;New features:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;errors&lt;/tt&gt; module. Every PostgreSQL error is converted into a specific exception class (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/682"&gt;ticket #682&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;encrypt_password()&lt;/tt&gt; function (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/576"&gt;ticket #576&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;BYTES&lt;/tt&gt; adapter to manage databases with mixed encodings on Python 3 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/835"&gt;ticket #835&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;table_oid&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;table_column&lt;/tt&gt; attributes on &lt;tt class="docutils literal"&gt;cursor.description&lt;/tt&gt; items (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/661"&gt;ticket #661&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;connection.info&lt;/tt&gt; object to retrieve various PostgreSQL connection information (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/726"&gt;ticket #726&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;get_native_connection()&lt;/tt&gt; to expose the raw &lt;tt class="docutils literal"&gt;PGconn&lt;/tt&gt; structure to C extensions via Capsule (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/782"&gt;ticket #782&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;pgconn_ptr&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;pgresult_ptr&lt;/tt&gt; to expose raw C structures to Python and interact with libpq via ctypes (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/782"&gt;ticket #782&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;sql.Identifier&lt;/tt&gt; can represent qualified names in SQL composition (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/732"&gt;ticket #732&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;ReplicationCursor.wal_end&lt;/tt&gt; attribute (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/800"&gt;ticket #800&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;em&gt;fetch&lt;/em&gt; parameter to &lt;tt class="docutils literal"&gt;execute_values()&lt;/tt&gt; function (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/813"&gt;ticket #813&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;str()&lt;/tt&gt; on &lt;tt class="docutils literal"&gt;Range&lt;/tt&gt; produces a human-readable representation (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/773"&gt;ticket #773&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;DictCursor&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;RealDictCursor&lt;/tt&gt; rows maintain columns order (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/177"&gt;ticket #177&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;severity_nonlocalized&lt;/tt&gt; attribute on the &lt;tt class="docutils literal"&gt;Diagnostics&lt;/tt&gt; object (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/783"&gt;ticket #783&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;More efficient &lt;tt class="docutils literal"&gt;NamedTupleCursor&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/838"&gt;ticket #838&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Bug fixes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed connections occasionally broken by the unrelated use of the multiprocessing module (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/829"&gt;ticket #829&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed async communication blocking if results are returned in different chunks, e.g. with notices interspersed to the results (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/856"&gt;ticket #856&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed adaptation of numeric subclasses such as &lt;tt class="docutils literal"&gt;IntEnum&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/591"&gt;ticket #591&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Other changes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Dropped support for Python 2.6, 3.2, 3.3.&lt;/li&gt;
&lt;li&gt;Dropped &lt;tt class="docutils literal"&gt;psycopg1&lt;/tt&gt; module.&lt;/li&gt;
&lt;li&gt;Dropped deprecated &lt;tt class="docutils literal"&gt;register_tstz_w_secs()&lt;/tt&gt; (was previously a no-op).&lt;/li&gt;
&lt;li&gt;Dropped deprecated &lt;tt class="docutils literal"&gt;PersistentConnectionPool&lt;/tt&gt;. This pool class was mostly
designed to interact with Zope. Use &lt;tt class="docutils literal"&gt;ZPsycopgDA.pool&lt;/tt&gt; instead.&lt;/li&gt;
&lt;li&gt;Binary packages no longer installed by default. The &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;psycopg2-binary&lt;/span&gt;&lt;/tt&gt; package must be used explicitly.&lt;/li&gt;
&lt;li&gt;Dropped &lt;tt class="docutils literal"&gt;PSYCOPG_DISPLAY_SIZE&lt;/tt&gt; build parameter.&lt;/li&gt;
&lt;li&gt;Dropped support for mxDateTime as the default date and time adapter. mxDatetime support continues to be available as an alternative to Python's builtin datetime.&lt;/li&gt;
&lt;li&gt;No longer use 2to3 during installation for Python 2 &amp;amp; 3 compatibility. All source files are now compatible with Python 2 &amp;amp; 3 as is.&lt;/li&gt;
&lt;li&gt;The &lt;tt class="docutils literal"&gt;psycopg2.test&lt;/tt&gt; package is no longer installed by &lt;tt class="docutils literal"&gt;python setup.py install&lt;/tt&gt;.&lt;/li&gt;
&lt;li&gt;Wheel package compiled against OpenSSL 1.0.2r and PostgreSQL 11.2 libpq.&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Psycopg 2.7.5 released</title><link href="https://www.psycopg.org/articles/2018/06/17/psycopg-275-released/" rel="alternate"/><updated>2018-06-17T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:1f5d57e4-8e09-3f5d-86b0-bbb69911d846</id><content type="html">&lt;p&gt;psycopg2 version 2.7.5 has been released, fixing a few bugs found in the last months:&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Summary of changes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Allow non-ascii chars in namedtuple fields (regression introduced fixing &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/211"&gt;ticket #211&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed adaptation of arrays of arrays of nulls (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/325"&gt;ticket #325&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed building on Solaris 11 and derivatives such as SmartOS and illumos (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/677"&gt;ticket #677&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Maybe fixed building on MSYS2 (as reported in &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/658"&gt;ticket #658&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Allow string subclasses in connection and other places (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/679"&gt;ticket #679&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Don't raise an exception closing an unused named cursor (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/716"&gt;ticket #716&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Wheel package compiled against PostgreSQL 10.4 libpq and OpenSSL 1.0.2o.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;You can install psycopg2 &lt;a class="reference external" href="https://pypi.python.org/pypi/psycopg2"&gt;from PyPI&lt;/a&gt; or grab the new code from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.5.tar.gz"&gt;source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.5.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.7.4 released</title><link href="https://www.psycopg.org/articles/2018/02/08/psycopg-274-released/" rel="alternate"/><updated>2018-02-08T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:60fc514c-be7a-3b25-9fda-2eeabb8d38e9</id><content type="html">&lt;p&gt;We have released Psycopg version 2.7.4, bringing a few bug fixes... and working out the problem with Wheel packages.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="what-s-the-problem-with-wheels"&gt;
&lt;h2&gt;What's the problem with Wheels?&lt;/h2&gt;
&lt;p&gt;&lt;a class="reference external" href="https://pythonwheels.com/"&gt;Wheel packages&lt;/a&gt; are a Python standard to distribute self-contained binary package. They are especially convenient for packages containing C extensions, such as Psycopg, and for packages depending on external C libraries... such as Psycopg, because the package will contain a binary, pre-compiled, version of the C extension and all the depending libraries (excluding a list of libraries expected to be found on any system and with a versioned ABI, such as libc). Since the release of the wheel packages with &lt;a class="reference external" href="/articles/2017/03/01/psycopg-27-released/"&gt;psycopg 2.7&lt;/a&gt; it has been possible to install Psycopg without the &lt;a class="reference external" href="/docs/install.html#build-prerequisites"&gt;prerequisites&lt;/a&gt; of a C compiler and of Python/Postgres dev packages at build time, and the need of a system libpq at runtime.&lt;/p&gt;
&lt;p&gt;Unfortunately, after the packages were released, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/543"&gt;it was reported&lt;/a&gt; of occasional segfaults of Python processes using Psycopg from the Wheel package. This was traced to the use of the Python &lt;tt class="docutils literal"&gt;ssl&lt;/tt&gt; library concurrently with Psycopg opening connections, for instance in a multithread program opening concurrently &lt;tt class="docutils literal"&gt;https&lt;/tt&gt; resources and database connections. The problem seems caused by a non-reentrant OpenSSL initialization function (unfortunately invoked by libpq at every connection) and the fact that the Python process ends up binding two different versions of the &lt;tt class="docutils literal"&gt;libssl&lt;/tt&gt; library: the system one via the Python &lt;tt class="docutils literal"&gt;ssl&lt;/tt&gt; library (e.g. imported by &lt;tt class="docutils literal"&gt;urllib&lt;/tt&gt; or &lt;tt class="docutils literal"&gt;requests&lt;/tt&gt;) and the Wheel one imported by the &lt;tt class="docutils literal"&gt;libpq&lt;/tt&gt;, in turn imported by &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt;.&lt;/p&gt;
&lt;p&gt;While the problem doesn't affect many users, a library behaving unreliably in combination with part of the stdlib is a situation less than optimal. The workaround is to force installing Psycopg from source, but this must be specified explicitly in the project dependencies (e.g. using the &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;--no-binary&lt;/span&gt;&lt;/tt&gt; flag in the &lt;tt class="docutils literal"&gt;pip&lt;/tt&gt; command line or in the &lt;tt class="docutils literal"&gt;requirements.txt&lt;/tt&gt; file); the Python packaging system doesn't really have a way to declare something like &amp;quot;install a package preferably from source&amp;quot;... so we had to create one ourselves.&lt;/p&gt;
&lt;p&gt;Starting with Psycopg 2.7.4, we are releasing two different packages on PyPI: &lt;a class="reference external" href="https://pypi.python.org/pypi/psycopg2/"&gt;psycopg2&lt;/a&gt; and &lt;a class="reference external" href="https://pypi.python.org/pypi/psycopg2-binary/"&gt;psycopg2-binary&lt;/a&gt;. The latter is a Wheels-only package, with a behaviour identical to the classic one – the different name is used only in installation (it is installed by &lt;tt class="docutils literal"&gt;pip install &lt;span class="pre"&gt;psycopg2-binary&lt;/span&gt;&lt;/tt&gt;, but still imported with &lt;tt class="docutils literal"&gt;import psycopg2&lt;/tt&gt; in Python).&lt;/p&gt;
&lt;p&gt;For the lifespan of the Psycopg 2.7 cycle, the &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; PyPI package will still contain wheel packages, but starting from Psycopg 2.8 it will become again a source-only package. Starting from Psycopg 2.7.4, if the package is installed as binary from the &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; PyPI entry, a warning will be reported on import:&lt;/p&gt;
&lt;blockquote&gt;
The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use &amp;quot;pip install psycopg2-binary&amp;quot; instead. For details see: &amp;lt;/docs/install.html#binary-install-from-pypi&amp;gt;.&lt;/blockquote&gt;
&lt;p&gt;The choices for the users are then two:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;if the program works fine with the Wheel packages, and the convenience of the binary distribution is preferred, it is possible to &lt;a class="reference external" href="/docs/install.html#binary-install-from-pypi"&gt;specify the dependency on the binary package&lt;/a&gt; using the &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;psycopg2-binary&lt;/span&gt;&lt;/tt&gt; instead of the &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; PyPI package. No change to the program code is needed;&lt;/li&gt;
&lt;li&gt;if there are concerns about the unreliability of the Wheels package, it is advised to &lt;a class="reference external" href="/docs/install.html#disabling-wheel-packages-for-psycopg-2-7"&gt;force installation from source&lt;/a&gt;. This requires the presence of build tools and runtime libraries on the client, but again it requires no change to the code.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;We hope this solution will suggest the default use of a reliable version of the library, while still allowing the convenience of a dependencies-free package. Feedback is welcome on the &lt;a class="reference external" href="https://www.postgresql.org/list/psycopg/"&gt;mailing list&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="other-changes-in-this-version"&gt;
&lt;h2&gt;Other changes in this version&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Convert fields names into valid Python identifiers in
&lt;tt class="docutils literal"&gt;NamedTupleCursor&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/211"&gt;ticket #211&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed Solaris 10 support (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/532"&gt;ticket #532&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;cursor.mogrify()&lt;/tt&gt; can be called on closed cursors (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/579"&gt;ticket #579&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed setting session characteristics in corner cases on autocommit
connections (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/580"&gt;ticket #580&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;MinTimeLoggingCursor&lt;/tt&gt; on Python 3 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/609"&gt;ticket #609&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed parsing of array of points as floats (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/613"&gt;ticket #613&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;__libpq_version__&lt;/tt&gt; building with libpq &amp;gt;= 10.1
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/632"&gt;ticket 632&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;rowcount&lt;/tt&gt; after &lt;tt class="docutils literal"&gt;executemany()&lt;/tt&gt; with &lt;tt class="docutils literal"&gt;RETURNING&lt;/tt&gt;
statements (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/633"&gt;ticket 633&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed compatibility problem with pypy3 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/649"&gt;ticket #649&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Wheel packages compiled against PostgreSQL 10.1 libpq and OpenSSL 1.0.2n.&lt;/li&gt;
&lt;li&gt;Wheel packages for Python 2.6 no more available (support dropped from
wheel building infrastructure).&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 2.7.3.2 released</title><link href="https://www.psycopg.org/articles/2017/10/24/psycopg-2732-released/" rel="alternate"/><updated>2017-10-24T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:d04b31ed-bb14-30e7-8adb-2658369c7f3a</id><content type="html">&lt;p&gt;Following the release of &lt;a class="reference external" href="https://www.postgresql.org/about/news/1786/"&gt;PostgreSQL 10&lt;/a&gt; we have released new binary packages as Psycopg 2.7.3.2. There are no code changes from Psycopg 2.7.3, but the new binary packages ship with the PostgreSQL 10 client library, enabling the use of new features such as &lt;a class="reference external" href="http://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/"&gt;multiple hosts in connection string&lt;/a&gt;, &lt;a class="reference external" href="http://paquier.xyz/postgresql-2/postgres-10-libpq-read-write/"&gt;read-only mode&lt;/a&gt;, &lt;a class="reference external" href="https://www.postgresql.org/docs/10/static/sasl-authentication.html#sasl-scram-sha256"&gt;SCRAM-SHA-256 authentication&lt;/a&gt;.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;The packages are already &lt;a class="reference external" href="https://pypi.python.org/pypi/psycopg2/2.7.3.2"&gt;available on PyPI&lt;/a&gt; and automatically installed by &lt;tt class="docutils literal"&gt;pip install psycopg2&lt;/tt&gt; (if your pip is up-to-date). You can grab the source code from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.3.2.tar.gz"&gt;source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.3.2.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.7.3.1 released</title><link href="https://www.psycopg.org/articles/2017/08/26/psycopg-2731-released/" rel="alternate"/><updated>2017-08-26T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:5101b867-469b-3bd0-81ff-58baeeddc374</id><content type="html">&lt;p&gt;We have released psycopg2 release 2.7.3.1 as a new build of psycopg 2.7.3. The new build only affects the wheel packages and contains no change in the code.&lt;/p&gt;
&lt;p&gt;The release 2.7.3.1 fixes &lt;a class="reference external" href="https://github.com/psycopg/psycopg2-wheels/issues/2"&gt;psycopg2-wheels bug #2&lt;/a&gt; which was in turn caused by &lt;a class="reference external" href="https://github.com/pypa/auditwheel/issues/80"&gt;auditwheel bug #80&lt;/a&gt;, resulting in incompatibility with glibc 2.26. The problem only affects Linux wheels users, it doesn't affect Windows, OSX, or user installing psycopg2 from source.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;You can install psycopg2 &lt;a class="reference external" href="https://pypi.python.org/pypi/psycopg2"&gt;from PyPI&lt;/a&gt; or grab the new code from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.3.1.tar.gz"&gt;source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.3.1.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Psycopg 2.7.3 released</title><link href="https://www.psycopg.org/articles/2017/07/24/psycopg-273-released/" rel="alternate"/><updated>2017-07-24T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:f522218f-040d-337a-bdb8-df17ba00dc99</id><content type="html">&lt;p&gt;A quick release to fix a regression found in &lt;a class="reference external" href="/psycopg/articles/2017/07/22/psycopg-272-released/"&gt;psycopg 2.7.2&lt;/a&gt;:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Restored default &lt;tt class="docutils literal"&gt;timestamptz[]&lt;/tt&gt; typecasting to Python &lt;tt class="docutils literal"&gt;datetime&lt;/tt&gt;.
Regression introduced in Psycopg 2.7.2 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/578"&gt;ticket #578&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;You can install psycopg2 &lt;a class="reference external" href="https://pypi.python.org/pypi/psycopg2"&gt;from PyPI&lt;/a&gt; or grab the new code from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.3.tar.gz"&gt;source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.3.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.7.2 released</title><link href="https://www.psycopg.org/articles/2017/07/22/psycopg-272-released/" rel="alternate"/><updated>2017-07-22T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:5ad9993a-c322-3804-8675-286812d7f5c9</id><content type="html">&lt;p&gt;Releasing psycopg2 version 2.7.2: a release fixing a host of bugs found in the last 3-4 months.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Summary of changes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed inconsistent state in externally closed connections (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/263"&gt;ticket #263&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/311"&gt;ticket #311&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/443"&gt;ticket #443&lt;/a&gt;). Was fixed in 2.6.2 but not included in 2.7 by mistake.&lt;/li&gt;
&lt;li&gt;Fixed Python exceptions propagation in green callback (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/410"&gt;ticket #410&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Don't display the password in &lt;tt class="docutils literal"&gt;connection.dsn&lt;/tt&gt; when the connection string is specified as an URI (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/528"&gt;ticket #528&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Return objects with timezone parsing &amp;quot;infinity&amp;quot; &lt;tt class="docutils literal"&gt;timestamptz&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/536"&gt;ticket #536&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Dropped dependency on VC9 runtime on Windows binary packages (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/541"&gt;ticket #541&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed segfault in &lt;tt class="docutils literal"&gt;lobject()&lt;/tt&gt; when &lt;em&gt;mode&lt;/em&gt;=&lt;tt class="docutils literal"&gt;None&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/544"&gt;ticket #544&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;lobject()&lt;/tt&gt; keyword argument &lt;em&gt;lobject_factory&lt;/em&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/545"&gt;ticket #545&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;ReplicationCursor.consume_stream()&lt;/tt&gt; &lt;em&gt;keepalive_interval&lt;/em&gt; argument (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/547"&gt;ticket #547&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Maybe fixed random import error on Python 3.6 in multiprocess environment (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/550"&gt;ticket #550&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed random &lt;tt class="docutils literal"&gt;SystemError&lt;/tt&gt; upon receiving abort signal (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/551"&gt;ticket #551&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Accept &lt;tt class="docutils literal"&gt;sql.Composable&lt;/tt&gt; objects in &lt;tt class="docutils literal"&gt;ReplicationCursor.start_replication_expert()&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/554"&gt;ticket 554&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Parse intervals returned as microseconds from Redshift (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/558"&gt;ticket #558&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;Json.prepare()&lt;/tt&gt; method to consider connection params when adapting (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/562"&gt;ticket #562&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;errorcodes&lt;/tt&gt; map updated to PostgreSQL 10 beta 1.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;You can install psycopg2 &lt;a class="reference external" href="https://pypi.python.org/pypi/psycopg2"&gt;from PyPI&lt;/a&gt; or grab the new code from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.2.tar.gz"&gt;source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.2.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.7.1 released</title><link href="https://www.psycopg.org/articles/2017/03/13/psycopg-271-released/" rel="alternate"/><updated>2017-03-13T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:385e4a6f-e45c-37ac-9180-c30115a6a77b</id><content type="html">&lt;p&gt;A quick bugfix release to solve some teething problems with some of the changes introduced in 2.7:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Ignore None arguments passed to &lt;tt class="docutils literal"&gt;connect()&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;make_dsn()&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/517"&gt;ticket #517&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;OpenSSL upgraded from major version 0.9.8 to 1.0.2 in the Linux wheel packages (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/518"&gt;ticket #518&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed build with libpq versions &amp;lt; 9.3 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/520"&gt;ticket #520&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Code fresh to grab at:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.1.tar.gz"&gt;source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.1.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Enjoy the release!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.7 released</title><link href="https://www.psycopg.org/articles/2017/03/01/psycopg-27-released/" rel="alternate"/><updated>2017-03-01T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:851acb9c-c295-3fbd-81d5-8bbb4765e75a</id><content type="html">&lt;p&gt;Finally here! Thank you very much for waiting so long: we have finally released Psycopg 2.7!&lt;/p&gt;
&lt;p&gt;Buzzwords:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;em&gt;Faster!&lt;/em&gt; Helps generating SQL for repeatedly executed statements and faster Unicode decoding.&lt;/li&gt;
&lt;li&gt;&lt;em&gt;Safer!&lt;/em&gt; Helps generating dynamic SQL statements with variable table and field names.&lt;/li&gt;
&lt;li&gt;&lt;em&gt;Easier!&lt;/em&gt; Use the binary package to avoid the need of C compiler, pg_config, libpq required on the clients.&lt;/li&gt;
&lt;li&gt;&lt;em&gt;Replication!&lt;/em&gt; Support for PostgreSQL physical and logical replication.&lt;/li&gt;
&lt;li&gt;&lt;em&gt;Plays-better-with-pgbouncer-at-transaction-pooling-level!&lt;/em&gt; This.&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;You can download as usual from the canonical urls:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.tar.gz"&gt;source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.7.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Or just use &lt;tt class="docutils literal"&gt;pip&lt;/tt&gt;:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
pip install psycopg2
&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;What's new in psycopg 2.7&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;New features:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added &lt;a class="reference external" href="/docs/sql.html#module-psycopg2.sql"&gt;psycopg2.sql&lt;/a&gt; module to generate SQL dynamically (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/308"&gt;ticket #308&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;a class="reference external" href="/docs/advanced.html#replication-support"&gt;replication protocol support&lt;/a&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/322"&gt;ticket #322&lt;/a&gt;). Main authors are
Oleksandr Shulgin and Craig Ringer, who deserve a huge thank you.&lt;/li&gt;
&lt;li&gt;Added &lt;a class="reference external" href="/docs/extensions.html#psycopg2.extensions.parse_dsn"&gt;parse_dsn()&lt;/a&gt; and
&lt;a class="reference external" href="/docs/extensions.html#psycopg2.extensions.make_dsn"&gt;make_dsn()&lt;/a&gt; functions (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/321"&gt;ticket #321&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/363"&gt;ticket #363&lt;/a&gt;).
&lt;a class="reference external" href="/docs/module.html#psycopg2.connect"&gt;connect()&lt;/a&gt; now can take both &lt;em&gt;dsn&lt;/em&gt; and keyword arguments, merging
them together.&lt;/li&gt;
&lt;li&gt;Added &lt;a class="reference external" href="/docs/module.html#psycopg2.__libpq_version__"&gt;__libpq_version__&lt;/a&gt; and &lt;a class="reference external" href="/docs/extensions.html#psycopg2.extensions.libpq_version"&gt;libpq_version()&lt;/a&gt; to inspect the version of the &lt;tt class="docutils literal"&gt;libpq&lt;/tt&gt; library the module was compiled/loaded with
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/35"&gt;ticket #35&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/323"&gt;ticket #323&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;The attributes &lt;a class="reference external" href="/docs/connection.html#connection.notices"&gt;notices&lt;/a&gt; and &lt;a class="reference external" href="/docs/connection.html#connection.notifies"&gt;notifies&lt;/a&gt; can be
customized replacing them with any object exposing an &lt;tt class="docutils literal"&gt;append()&lt;/tt&gt; method
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/326"&gt;ticket #326&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Adapt network types to &lt;a class="reference external" href="https://docs.python.org/3/library/ipaddress.html#module-ipaddress"&gt;ipaddress&lt;/a&gt; objects when available. When not
enabled, convert arrays of network types to lists by default. The old &lt;tt class="docutils literal"&gt;Inet&lt;/tt&gt;
adapter is deprecated (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/317"&gt;ticket #317&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/343"&gt;ticket #343&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/387"&gt;ticket #387&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;a class="reference external" href="/docs/extensions.html#psycopg2.extensions.quote_ident"&gt;quote_ident()&lt;/a&gt; function (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/359"&gt;ticket #359&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;a class="reference external" href="/docs/connection.html#connection.get_dsn_parameters"&gt;get_dsn_parameters()&lt;/a&gt; connection method (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/364"&gt;ticket #364&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="/docs/cursor.html#cursor.callproc"&gt;callproc()&lt;/a&gt; now accepts a dictionary of parameters (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/381"&gt;ticket #381&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Give precedence to &lt;tt class="docutils literal"&gt;__conform__()&lt;/tt&gt; over superclasses to choose an object
adapter (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/456"&gt;ticket #456&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Using Python C API decoding functions and codecs caching for faster
unicode encoding/decoding (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/473"&gt;ticket #473&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;executemany()&lt;/tt&gt; slowness addressed by
&lt;a class="reference external" href="/docs/extras.html#psycopg2.extras.execute_batch"&gt;execute_batch()&lt;/a&gt; and &lt;a class="reference external" href="/docs/extras.html#psycopg2.extras.execute_values"&gt;execute_values()&lt;/a&gt;
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/491"&gt;ticket #491&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;async_&lt;/tt&gt; as an alias for &lt;tt class="docutils literal"&gt;async&lt;/tt&gt; to support Python 3.7 where
&lt;tt class="docutils literal"&gt;async&lt;/tt&gt; will become a keyword (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/495"&gt;ticket #495&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Unless in autocommit, do not use &lt;tt class="docutils literal"&gt;default_transaction_*&lt;/tt&gt; settings to
control the session characteristics as it may create problems with external
connection pools such as pgbouncer; use &lt;tt class="docutils literal"&gt;BEGIN&lt;/tt&gt; options instead
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/503"&gt;ticket #503&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="/docs/connection.html#connection.isolation_level"&gt;isolation_level&lt;/a&gt; is now writable and entirely separated from
&lt;a class="reference external" href="/docs/connection.html#connection.autocommit"&gt;autocommit&lt;/a&gt;; added &lt;a class="reference external" href="/docs/connection.html#connection.readonly"&gt;readonly&lt;/a&gt;,
&lt;a class="reference external" href="/docs/connection.html#connection.deferrable"&gt;deferrable&lt;/a&gt; writable attributes.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Bug fixes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed error caused by missing decoding &lt;a class="reference external" href="/docs/extras.html#psycopg2.extras.LoggingConnection"&gt;LoggingConnection&lt;/a&gt;
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/483"&gt;ticket #483&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed integer overflow in &lt;tt class="docutils literal"&gt;interval&lt;/tt&gt; seconds (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/512"&gt;ticket #512&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Other changes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Dropped support for Python 2.5 and 3.1.&lt;/li&gt;
&lt;li&gt;Dropped support for client library older than PostgreSQL 9.1 (but older
server versions are still supported).&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="/docs/connection.html#connection.isolation_level"&gt;isolation_level&lt;/a&gt; doesn't read from the database but will return
&lt;a class="reference external" href="/docs/extensions.html#psycopg2.extensions.ISOLATION_LEVEL_DEFAULT"&gt;ISOLATION_LEVEL_DEFAULT&lt;/a&gt; if no value was set on the
connection.&lt;/li&gt;
&lt;li&gt;Empty arrays no more converted into lists if they don't have a type attached
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/506"&gt;ticket #506&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Psycopg 2.7 beta 2 released</title><link href="https://www.psycopg.org/articles/2017/02/17/psycopg-27-beta-2-released/" rel="alternate"/><updated>2017-02-17T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:115f8277-5973-37ff-aa21-b0250328ff16</id><content type="html">&lt;p&gt;Hello,&lt;/p&gt;
&lt;p&gt;we have released psycopg2 version 2.7 beta 2. This version comes two years after the previous major release so it is packed with new features and improvements; among the main points:&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;support for replication protocol&lt;/li&gt;
&lt;li&gt;helpers to build dynamically SQL statements and manipulate connection strings&lt;/li&gt;
&lt;li&gt;speedups in multiple execution and Unicode decoding&lt;/li&gt;
&lt;li&gt;better transactions characteristics support&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A more complete list of changes is available at /docs/news.html#what-s-new-in-psycopg-2-7&lt;/p&gt;
&lt;p&gt;On top of the changes in the code we have addressed the deployment problems found my many inexperienced users, especially on Windows and OSX: psycopg is now distributed as a self-contained wheel package on PyPI, so that:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
pip install psycopg2
&lt;/pre&gt;
&lt;p&gt;will not require the presence of C compiler, headers, pg_config, libpq.&lt;/p&gt;
&lt;p&gt;Because of all these changes, we ask you kindly to test this psycopg beta release, both its installation and its usage, before we release the stable version, which should happen next week if we don't find any problem. The binary packages of the beta release are on the PyPI testing site: you can install it using:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
pip install -U pip # make sure to have an up-to-date pip
pip install -i https://testpypi.python.org/pypi psycopg2==2.7b2
&lt;/pre&gt;
&lt;p&gt;Thank you very much everyone, and I wish to thank especially Oleksandr Shulgin and Craig Ringer for the development of the replication protocol support (the development of which brought along several other features and improvement) and Jason Erickson for the invaluable help with the Windows packages.&lt;/p&gt;
&lt;p&gt;Happy testing!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.6.2 released</title><link href="https://www.psycopg.org/articles/2016/07/07/psycopg-262-released/" rel="alternate"/><updated>2016-07-07T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:ce095754-cfde-30de-8ae8-8c207529b4b5</id><content type="html">&lt;p&gt;Psycopg 2.6.2 has been released. You can get it from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.6.2.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.6.2.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is an interim release, packing together one year of bug fixes, before the release 2.7, intended to deliver several new features. Thank you very much to everybody contributing with reports, code, suggestions.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed inconsistent state in externally closed connections
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/263"&gt;ticket #263&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/311"&gt;ticket #311&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/443"&gt;ticket #443&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Report the server response status on errors (such as &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/281"&gt;ticket #281&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Raise &lt;tt class="docutils literal"&gt;NotSupportedError&lt;/tt&gt; on unhandled server response status
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/352"&gt;ticket #352&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Allow overriding string adapter encoding with no connection (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/331"&gt;ticket #331&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;The &lt;tt class="docutils literal"&gt;wait_select&lt;/tt&gt; callback allows interrupting a
long-running query in an interactive shell using &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;Ctrl-C&lt;/span&gt;&lt;/tt&gt;
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/333"&gt;ticket #333&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;PersistentConnectionPool&lt;/tt&gt; on Python 3 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/348"&gt;ticket #348&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed segfault on &lt;tt class="docutils literal"&gt;repr()&lt;/tt&gt; of an uninitialized connection (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/361"&gt;ticket #361&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Allow adapting bytes using &lt;tt class="docutils literal"&gt;QuotedString&lt;/tt&gt; on Python 3
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/365"&gt;ticket #365&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added support for setuptools/wheel (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/370"&gt;ticket #370&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fix build on Windows with Python 3.5, VS 2015 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/380"&gt;ticket #380&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;errorcodes.lookup&lt;/tt&gt; initialization thread-safety (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/382"&gt;ticket #382&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;read()&lt;/tt&gt; exception propagation in copy_from (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/412"&gt;ticket #412&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed possible NULL TZ decref  (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/424"&gt;ticket #424&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;psycopg2.errorcodes&lt;/tt&gt; map updated to PostgreSQL 9.5.&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Psycopg 2.6.1 released</title><link href="https://www.psycopg.org/articles/2015/06/16/psycopg-261-released/" rel="alternate"/><updated>2015-06-16T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:de7e95fc-3f3a-36db-b633-663fd65c5a2c</id><content type="html">&lt;p&gt;Psycopg 2.6.1 has been released. You can get it from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.6.1.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.6.1.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;The most important bug fixed in this release is the libcrypto
callbacks conflict between libpq and Python (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/290"&gt;ticket #290&lt;/a&gt;), fixed by Jan
Urbański: thank you very much. Other bugs fixed:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Lists consisting of only &lt;tt class="docutils literal"&gt;None&lt;/tt&gt; are escaped correctly (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/285"&gt;ticket #285&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Correctly unlock the connection after error in flush (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/294"&gt;ticket #294&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;MinTimeLoggingCursor.callproc()&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/309"&gt;ticket #309&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Psycopg 2.6 and 2.5.5 released</title><link href="https://www.psycopg.org/articles/2015/02/09/psycopg-26-and-255-released/" rel="alternate"/><updated>2015-02-09T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:3b032d0c-d028-3a29-b3f4-51f3dbfc0601</id><content type="html">&lt;p&gt;We have just released two Psycopg versions: 2.5.5 containing a few bug fixes and 2.6 introducing some new features.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;The main new feature in Psycopg 2.6 is the support for the large objects 64 bits API, allowing access to large objects of more than 2GB size. Note that the feature is only supported on 64 bits clients and with at least PostgreSQL 9.3 server and client library.&lt;/p&gt;
&lt;p&gt;I wish to thank all the people who have contributed to these releases, and especially Blake Rouse and the MAAS Team for the development of the large objects 64 bits API support.&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Psycopg 2.6 &lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.6.tar.gz"&gt;source package&lt;/a&gt;, &lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.6.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Psycopg 2.5.5 &lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.5.tar.gz"&gt;source package&lt;/a&gt;,  &lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.5.tar.gz.asc"&gt;signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is the full list of changes for the new versions:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Psycopg 2.6 new features&lt;/strong&gt;&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added support for large objects larger than 2GB.&lt;/li&gt;
&lt;li&gt;Python &lt;tt class="docutils literal"&gt;time&lt;/tt&gt; objects with a tzinfo specified and PostgreSQL &lt;tt class="docutils literal"&gt;timetz&lt;/tt&gt;
data are converted into each other (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/272"&gt;ticket #272&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Psycopg 2.6 bug fixes&lt;/strong&gt;&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Json apapter's &lt;tt class="docutils literal"&gt;str()&lt;/tt&gt; returns the adapted content instead of the &lt;tt class="docutils literal"&gt;repr()&lt;/tt&gt;
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/191"&gt;ticket #191&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Psycopg 2.5.5 bug fixes&lt;/strong&gt;&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Named cursors used as context manager don't swallow the exception on exit
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/262"&gt;ticket #262&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;cursor.description&lt;/tt&gt; can be pickled (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/265"&gt;ticket #265&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Propagate read error messages in COPY FROM (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/270"&gt;ticket #270&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;PostgreSQL time 24:00 is converted to Python 00:00 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/278"&gt;ticket #278&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A further note for the packagers: the Psycopg 2.6 source tarball doesn't contain the HTML rendered documentation anymore: you will find Makefile targets to easily build it.&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.5.4 released</title><link href="https://www.psycopg.org/articles/2014/08/30/psycopg-254-released/" rel="alternate"/><updated>2014-08-30T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:fb53de94-1978-309e-85d8-e5a375455256</id><content type="html">&lt;p&gt;Psycopg 2.5.4 has been released. You can get it from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.4.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.4.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This version supports the new jsonb PostgreSQL type out-of-the-box. And of course there are a few bug fixed:&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;jsonb&lt;/tt&gt; support for PostgreSQL 9.4 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/226"&gt;ticket #226&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed segfault if COPY statements are passed to &lt;tt class="docutils literal"&gt;execute()&lt;/tt&gt; instead
of using the proper methods (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/219"&gt;ticket #219&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Force conversion of pool arguments to integer to avoid potentially unbounded
pools (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/220"&gt;ticket #220&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Cursors &lt;tt class="docutils literal"&gt;WITH HOLD&lt;/tt&gt; don't begin a new transaction upon move/fetch/close
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/228"&gt;ticket #228&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Cursors &lt;tt class="docutils literal"&gt;WITH HOLD&lt;/tt&gt; can be used in autocommit (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/229"&gt;ticket #229&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;callproc()&lt;/tt&gt; doesn't silently ignore an argument without a length.&lt;/li&gt;
&lt;li&gt;Fixed memory leak with large objects (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/256"&gt;ticket #256&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;The internal &lt;tt class="docutils literal"&gt;_psycopg.so&lt;/tt&gt; module can be imported stand-alone (to
allow modules juggling such as the one described in &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/201"&gt;ticket #201&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Have a good weekend!&lt;/p&gt;
</content></entry><entry><title>Cancelling PostgreSQL statements from Python</title><link href="https://www.psycopg.org/articles/2014/07/20/cancelling-postgresql-statements-python/" rel="alternate"/><updated>2014-07-20T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:0ae5a9b4-9d94-31bb-aaa7-fb996c378597</id><content type="html">&lt;p&gt;Cancelling a long running query from Python is not something that happens automatically: the libpq doesn't react to Python signals so the only way to stop a query is to run a &lt;tt class="docutils literal"&gt;pg_cancel_backend&lt;/tt&gt; from another process. Killing the Python process won't cancel the query: it will run until completion and then rolled back. This makes working with long-running query from the Python interpreter somewhat frustrating.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Using psycopg in &lt;a class="reference external" href="/docs/advanced.html#support-for-coroutine-libraries"&gt;green mode&lt;/a&gt; moves the waiting from the libpq C code to Python: this gives Python some chance of interaction: it is possible for instance to catch a ctrl-c and send a cancel request:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;select&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;select&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg2.extensions&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;POLL_OK&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;POLL_READ&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;POLL_WRITE&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;wait_select_inter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;poll&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;POLL_OK&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;                &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;POLL_READ&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;                &lt;span class="n"&gt;select&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fileno&lt;/span&gt;&lt;span class="p"&gt;()],&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;POLL_WRITE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;                &lt;span class="n"&gt;select&lt;/span&gt;&lt;span class="p"&gt;([],&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fileno&lt;/span&gt;&lt;span class="p"&gt;()],&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;                &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;OperationalError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;                    &lt;span class="s2"&gt;&amp;quot;bad state from poll: &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;KeyboardInterrupt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="c1"&gt;# the loop will be broken by a server error&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;            &lt;span class="k"&gt;continue&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;psycopg2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;extensions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_wait_callback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wait_select_inter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;An interactive session would look like:&lt;/p&gt;
&lt;pre class="code pycon literal-block"&gt;
&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;cnn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cnn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gp"&gt;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select pg_sleep(10)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="go"&gt;^C
&lt;/span&gt;&lt;span class="gt"&gt;Traceback (most recent call last):
&lt;/span&gt;  File &lt;span class="nb"&gt;&amp;quot;&amp;lt;stdin&amp;gt;&amp;quot;&lt;/span&gt;, line &lt;span class="m"&gt;1&lt;/span&gt;, in &lt;span class="n"&gt;&amp;lt;module&amp;gt;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="gr"&gt;QueryCanceledError&lt;/span&gt;: &lt;span class="n"&gt;canceling statement due to user request&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The connection is now in error state, but a &lt;tt class="docutils literal"&gt;cnn.rollback()&lt;/tt&gt; would make it working again.&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.5.3 Released</title><link href="https://www.psycopg.org/articles/2014/05/13/psycopg-253-released/" rel="alternate"/><updated>2014-05-13T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:e429e4db-13fe-3a7b-a56f-5b19e6a9a895</id><content type="html">&lt;p&gt;Psycopg 2.5.3 has been released. You can get it from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.3.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.3.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This version contains several bug fixes over the previous release 2.5.2:&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Work around &lt;a class="reference external" href="https://github.com/pypa/pip/issues/1630"&gt;pip issue #1630&lt;/a&gt;
making installation via &lt;tt class="docutils literal"&gt;pip &lt;span class="pre"&gt;-e&lt;/span&gt; git+url&lt;/tt&gt; impossible (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/18"&gt;ticket #18&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Copy operations correctly set the &lt;tt class="docutils literal"&gt;cursor.rowcount&lt;/tt&gt; attribute
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/180"&gt;ticket #180&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;It is now possible to call &lt;tt class="docutils literal"&gt;get_transaction_status()&lt;/tt&gt; on closed connections.&lt;/li&gt;
&lt;li&gt;Fixed unsafe access to object names causing assertion failures in
Python 3 debug builds (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/188"&gt;ticket #188&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Mark the connection closed if found broken on &lt;tt class="docutils literal"&gt;poll()&lt;/tt&gt; (from &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/192"&gt;ticket #192&lt;/a&gt;
discussion)&lt;/li&gt;
&lt;li&gt;Fixed handling of dsn and closed attributes in connection subclasses
failing to connect (from &lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/192"&gt;ticket #192&lt;/a&gt; discussion).&lt;/li&gt;
&lt;li&gt;Added arbitrary but stable order to &lt;tt class="docutils literal"&gt;Range&lt;/tt&gt; objects, thanks to
Chris Withers (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/193"&gt;ticket #193&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Avoid blocking async connections on connect (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/194"&gt;ticket #194&lt;/a&gt;). Thanks to
Adam Petrovich for the bug report and diagnosis.&lt;/li&gt;
&lt;li&gt;Don't segfault using poorly defined cursor subclasses which forgot to call
the superclass init (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/195"&gt;ticket #195&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Mark the connection closed when a Socket connection is broken, as it
happens for TCP connections instead (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/196"&gt;ticket #196&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed overflow opening a lobject with an oid not fitting in a signed int
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/203"&gt;ticket #203&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed handling of explicit default &lt;tt class="docutils literal"&gt;cursor_factory=None&lt;/tt&gt; in
&lt;tt class="docutils literal"&gt;connection.cursor()&lt;/tt&gt; (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/210"&gt;ticket #210&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed possible segfault in named cursors creation.&lt;/li&gt;
&lt;li&gt;Fixed debug build on Windows, thanks to James Emerton.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Thank you very much to everybody helping with this release!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.5.2 released</title><link href="https://www.psycopg.org/articles/2014/01/07/psycopg-252-released/" rel="alternate"/><updated>2014-01-07T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:8e26aec9-f037-38e1-8e4f-37c5449cac6e</id><content type="html">&lt;p&gt;Psycopg 2.5.2 has been released. You can get it from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.2.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.2.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This version contains a few bug fixes over the previous release 2.5.1:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed segfault pickling the exception raised on connection error
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/170"&gt;ticket #170&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Meaningful connection errors report a meaningful message, thanks to
Alexey Borzenkov (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/173"&gt;ticket #173&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Manually creating &lt;tt class="docutils literal"&gt;lobject&lt;/tt&gt; with the wrong parameter doesn't segfault
(&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/187"&gt;ticket #187&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Thank you very much to all the people who helped during the development!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.5.1 released</title><link href="https://www.psycopg.org/articles/2013/06/23/psycopg-251-released/" rel="alternate"/><updated>2013-06-23T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:1a05b6f7-938b-35bf-9a01-4528466abb41</id><content type="html">&lt;p&gt;Psycopg 2.5.1 has been released. You can get it from:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.1.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.1.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The version contains a few bug fixes over the previous 2.5:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed build on Solaris 10 and 11 where the &lt;tt class="docutils literal"&gt;round()&lt;/tt&gt; function is already
declared (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/146"&gt;ticket #146&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed comparison of &lt;tt class="docutils literal"&gt;Range&lt;/tt&gt; with non-range objects (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/164"&gt;ticket #164&lt;/a&gt;).
Thanks to Chris Withers for the patch.&lt;/li&gt;
&lt;li&gt;Fixed double-free on connection dealloc (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/166"&gt;ticket #166&lt;/a&gt;). Thanks to
Gangadharan S.A. for the report and fix suggestion.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Happy hacking!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.5 released</title><link href="https://www.psycopg.org/articles/2013/04/07/psycopg-25-released/" rel="alternate"/><updated>2013-04-07T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:02bc89eb-f30f-3841-a091-bbab32ab51c8</id><content type="html">&lt;p&gt;We are happy to introduce the release 2.5 of Psycopg, packed with several juicy new features!&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.5.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here are a few highlights of the release:&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="json-adapter"&gt;
&lt;h2&gt;JSON adapter&lt;/h2&gt;
&lt;p&gt;The JSON adapter allows easy exchange of unstructured data with the database:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;     &lt;span class="s2"&gt;&amp;quot;&amp;quot;&amp;quot;select '{&amp;quot;a&amp;quot;:[1,2,3],&amp;quot;b&amp;quot;:[4,5,6]}'::json&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sa"&gt;u&lt;/span&gt;&lt;span class="s1"&gt;'a'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sa"&gt;u&lt;/span&gt;&lt;span class="s1"&gt;'b'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;It works with PostgreSQL 9.2 native &lt;a class="reference external" href="https://www.postgresql.org/docs/current/static/datatype-json.html"&gt;JSON data type&lt;/a&gt;, with previous versions' &lt;a class="reference external" href="http://people.planetpostgresql.org/andrew/index.php?/archives/255-JSON-for-PG-9.2-...-and-now-for-9.1!.html"&gt;extension modules&lt;/a&gt; and, in a pinch, with regular text too. You can feed JSON data to PostgreSQL using &lt;a class="reference external" href="/psycopg/docs/extras.html#psycopg2.extras.Json"&gt;a specific adapter&lt;/a&gt;:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;curs&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;insert into mytable (jsondata) values (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Json&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s1"&gt;'a'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;})])&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;or you can be more aggressive and ask for automatic JSON serialization for every type you want to support, dictionaries for instance:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;psycopg2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;extensions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;register_adapter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Json&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;curs&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;insert into mytable (jsondata) values (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s1"&gt;'a'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;}])&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;
&lt;div class="section" id="range-adapters"&gt;
&lt;h2&gt;Range adapters&lt;/h2&gt;
&lt;p&gt;Other handy adapters for PostgreSQL 9.2 are the &lt;a class="reference external" href="https://www.postgresql.org/docs/current/static/rangetypes.html"&gt;Range data type&lt;/a&gt; adapters family: ranges are returned as &lt;a class="reference external" href="/psycopg/docs/extras.html#psycopg2.extras.Range"&gt;an object&lt;/a&gt; allowing access to their properties:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&amp;quot;select '[10,20)'::int4range&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lower&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;upper_inc&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Of course the Python object can be used as arguments in query parameters:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;DateRange&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2013&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2013&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s1"&gt;'[)'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;select * from events where &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt; &amp;#64;&amp;gt; date&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;Built-in range types are supported out-of-the-box, while the function &lt;a class="reference external" href="/psycopg/docs/extras.html#psycopg2.extras.register_range"&gt;register_range()&lt;/a&gt; allows handling user-defined range types.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="connections-and-cursors-as-context-managers"&gt;
&lt;h2&gt;Connections and cursors as context managers&lt;/h2&gt;
&lt;p&gt;A recent DBAPI extension has standardized the use of connections and cursors as context managers: it is now possible to use an idiom such as:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;psycopg2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;DSN&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;curs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;       &lt;span class="n"&gt;curs&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SQL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;with the intuitive behaviour: when the cursor block exits the cursor is closed; when the connection block exits normally the current transaction is committed, if it exits with an exception instead the transaction is rolled back, in either case the connection is ready to be used again (&lt;strong&gt;FIXED:&lt;/strong&gt; the connection is NOT closed as originally stated).&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="other-new-features-in-psycopg-2-5"&gt;
&lt;h2&gt;Other new features in Psycopg 2.5&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added &lt;a class="reference external" href="/psycopg/docs/extensions.html#psycopg2.extensions.Diagnostics"&gt;Diagnostics&lt;/a&gt; object to get extended info from a database error.  Many thanks to Matthew Woodcraft for the
implementation (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/149"&gt;ticket #149&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added &lt;tt class="docutils literal"&gt;connection.cursor_factory&lt;/tt&gt; attribute to customize the default object returned by &lt;tt class="docutils literal"&gt;cursor()&lt;/tt&gt;.&lt;/li&gt;
&lt;li&gt;Added support for backward scrollable cursors. Thanks to Jon Nelson
for the initial patch (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/108"&gt;ticket #108&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Added a simple way to &lt;a class="reference external" href="/psycopg/docs/extras.html#adapt-composite"&gt;customize casting of composite types&lt;/a&gt; into Python objects other than namedtuples.
Many thanks to Ronan Dunklau and Tobias Oberstein for the feature
development.&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="/psycopg/docs/connection.html#connection.reset"&gt;connection.reset()&lt;/a&gt; implemented using &lt;tt class="docutils literal"&gt;DISCARD ALL&lt;/tt&gt; on server
versions supporting it.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Bug fixes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Properly cleanup memory of broken connections (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/148"&gt;ticket #148&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed bad interaction of &lt;tt class="docutils literal"&gt;setup.py&lt;/tt&gt; with other dependencies in
Distribute projects on Python 3 (&lt;a class="reference external" href="https://github.com/psycopg/psycopg2/issues/153"&gt;ticket #153&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Other changes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added support for Python 3.3.&lt;/li&gt;
&lt;li&gt;Dropped support for Python 2.4. Please use Psycopg 2.4.x if you need it.&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="/psycopg/docs/errorcodes.html#module-psycopg2.errorcodes"&gt;errorcodes&lt;/a&gt; map updated to PostgreSQL 9.2.&lt;/li&gt;
&lt;li&gt;Dropped Zope adapter from source repository. ZPsycopgDA has now its own
project at &amp;lt;&lt;a class="reference external" href="https://github.com/psycopg/ZPsycopgDA"&gt;https://github.com/psycopg/ZPsycopgDA&lt;/a&gt;&amp;gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 2.4.6 released</title><link href="https://www.psycopg.org/articles/2012/12/12/psycopg-246-released/" rel="alternate"/><updated>2012-12-12T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:f90ae131-85a2-3eec-87c2-8e09c4bb30e5</id><content type="html">&lt;p&gt;I'm happy to announce the release of Psycopg 2.4.6: a huge thank you to the many contributors.&lt;/p&gt;
&lt;p&gt;This is a bugfix release, introducing no new feature. There are several small corrections in different areas (copy, adaptation, use of extra cursors, stability). The biggest improvements are with the Zope adapter: Zope users using previous 2.4.x versions are encouraged to update to version 2.4.6 soon.&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.6.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.6.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="what-s-new-in-psycopg-2-4-6"&gt;
&lt;h2&gt;What's new in psycopg 2.4.6&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed 'cursor()' arguments propagation in connection subclasses
and overriding of the 'cursor_factory' argument.  Thanks to
Corry Haines for the report and the initial patch (ticket #105).&lt;/li&gt;
&lt;li&gt;Dropped GIL release during string adaptation around a function call
invoking a Python API function, which could cause interpreter crash.
Thanks to Manu Cupcic for the report (ticket #110).&lt;/li&gt;
&lt;li&gt;Close a green connection if there is an error in the callback.
Maybe a harsh solution but it leaves the program responsive
(ticket #113).&lt;/li&gt;
&lt;li&gt;'register_hstore()', 'register_composite()', 'tpc_recover()' work with
RealDictConnection and Cursor (ticket #114).&lt;/li&gt;
&lt;li&gt;Fixed broken pool for Zope and connections re-init across ZSQL methods
in the same request (tickets #123, #125, #142).&lt;/li&gt;
&lt;li&gt;connect() raises an exception instead of swallowing keyword arguments
when a connection string is specified as well (ticket #131).&lt;/li&gt;
&lt;li&gt;Discard any result produced by 'executemany()' (ticket #133).&lt;/li&gt;
&lt;li&gt;Fixed pickling of FixedOffsetTimezone objects (ticket #135).&lt;/li&gt;
&lt;li&gt;Release the GIL around PQgetResult calls after COPY (ticket #140).&lt;/li&gt;
&lt;li&gt;Fixed empty strings handling in composite caster (ticket #141).&lt;/li&gt;
&lt;li&gt;Fixed pickling of DictRow and RealDictRow objects.&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
</content></entry><entry><title>Prepared statements in Psycopg</title><link href="https://www.psycopg.org/articles/2012/10/01/prepared-statements-psycopg/" rel="alternate"/><updated>2012-10-01T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:7aee00a1-0e80-3b85-82a5-e29956df86bb</id><content type="html">&lt;p&gt;Although the libpq library supports &lt;a class="reference external" href="https://www.postgresql.org/docs/current/static/libpq-exec.html#LIBPQ-PQPREPARE"&gt;prepared statements&lt;/a&gt;, psycopg2 doesn't offer yet a direct way to access the relevant functions. This will probably change in the future, but in the meantime it is possible to use prepared statements in PostgreSQL using the &lt;a class="reference external" href="https://www.postgresql.org/docs/current/static/sql-prepare.html"&gt;PREPARE&lt;/a&gt; SQL command.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Whenever you have a loop where the same parametrized query or command is executed:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="s2"&gt;&amp;quot;select * from tables where i = &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt; and j = &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;do_something_with&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;you can actually ask PostgreSQL to prepare the plan in advance and use it, saving time in the inner loop:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;prepare myplan as &amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;select * from tables where i = $1 and j = $2&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;execute myplan (&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;do_something_with&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The time saved could be relevant for complex queries which are fast to execute; for queries that instead take several seconds to complete, the planning time is probably negligible so there wouldn't be much to save. Note that the query passed to &lt;tt class="docutils literal"&gt;PREPARE&lt;/tt&gt; uses Postgres-style placeholders (&lt;tt class="docutils literal"&gt;$1&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;$2&lt;/tt&gt;...) instead of the familiar Python-style &lt;tt class="docutils literal"&gt;%s&lt;/tt&gt; or &lt;tt class="docutils literal"&gt;%(name)s&lt;/tt&gt;.&lt;/p&gt;
&lt;p&gt;So is psycopg useless in this case? Is the burden of executing &lt;tt class="docutils literal"&gt;PREPARE&lt;/tt&gt; on the poor user? Well, it's actually easy to write a &lt;a class="reference external" href="https://gist.github.com/3797445"&gt;cursor subclass implementing prepared statements&lt;/a&gt;. The proposed &lt;tt class="docutils literal"&gt;PreparingCursor&lt;/tt&gt; doesn't &lt;tt class="docutils literal"&gt;PREPARE&lt;/tt&gt; each statement passed to &lt;tt class="docutils literal"&gt;execute()&lt;/tt&gt;: this would be harmful as it involves an extra roundtrip to the server and the plan for a prepared statements is sometimes less efficient than one calculated with the knowledge of the real parameters used. So the proposed class exposes an explicit &lt;tt class="docutils literal"&gt;prepare()&lt;/tt&gt; method: it takes a query (written with Python-style placeholders, so exactly the one you would have used with &lt;tt class="docutils literal"&gt;execute()&lt;/tt&gt;), replaces it with Postgres-style placeholders and &lt;tt class="docutils literal"&gt;PREPARE&lt;/tt&gt;s it. In further calls to &lt;tt class="docutils literal"&gt;execute()&lt;/tt&gt; the query can be omitted: in this case (or if the query is the one prepared) the prepared statement is executed instead.&lt;/p&gt;
&lt;p&gt;Using the &lt;tt class="docutils literal"&gt;PreparingCursor&lt;/tt&gt; the above example could be written as:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cursor_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;PreparingCursor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prepare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="s2"&gt;&amp;quot;select * from tables where i = &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt; and j = &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;        &lt;span class="n"&gt;do_something_with&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;The preparing cursor also overrides &lt;tt class="docutils literal"&gt;executemany()&lt;/tt&gt;: in this case the query is always prepared as it is assumed that it will be executed more than once. Other extensions are a &lt;tt class="docutils literal"&gt;prepared&lt;/tt&gt; attribute, returning the prepared statement if any, and a &lt;tt class="docutils literal"&gt;deallocate()&lt;/tt&gt; method to release the prepared statement (which would be deleted anyway at the end of the session).&lt;/p&gt;
&lt;p&gt;The preparing cursor may find its way into a future psycopg2 release, but its design is not finalized yet and several details, both in the interface and the implementation, could be done in different ways. So please use it and give us feedback: we'll use it to design the optimal implementation for Psycopg!&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.4.5 released</title><link href="https://www.psycopg.org/articles/2012/03/29/psycopg-245-released/" rel="alternate"/><updated>2012-03-29T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:3aeb292c-0bda-3299-a08a-88d96aa8f7fd</id><content type="html">&lt;p&gt;Many thanks to everybody that contributed with bug reports and comments to this release!&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.5.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.5.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="what-s-new-in-psycopg-2-4-5"&gt;
&lt;h2&gt;What's new in psycopg 2.4.5&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;The &lt;tt class="docutils literal"&gt;close()&lt;/tt&gt; methods on connections and cursors don't raise
exceptions
if called on already closed objects.&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;fetchmany()&lt;/tt&gt; with no argument in cursor subclasses
(ticket #84).&lt;/li&gt;
&lt;li&gt;Use &lt;tt class="docutils literal"&gt;lo_creat()&lt;/tt&gt; instead of &lt;tt class="docutils literal"&gt;lo_create()&lt;/tt&gt; when possible for better
interaction with pgpool-II (ticket #88).&lt;/li&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;Error&lt;/tt&gt; and its subclasses are picklable, useful for multiprocessing
interaction (ticket #90).&lt;/li&gt;
&lt;li&gt;Better efficiency and formatting of timezone offset objects thanks
to Menno Smits (tickets #94, #95).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;rownumber&lt;/tt&gt; during iteration on cursor subclasses.
Regression introduced in 2.4.4 (ticket #100).&lt;/li&gt;
&lt;li&gt;Added support for &lt;tt class="docutils literal"&gt;inet&lt;/tt&gt; arrays.&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;commit()&lt;/tt&gt; concurrency problem (ticket #103).&lt;/li&gt;
&lt;li&gt;Codebase cleaned up using the &lt;a class="reference external" href="https://fedorahosted.org/gcc-python-plugin/"&gt;GCC Python plugin&lt;/a&gt;'s &lt;a class="reference external" href="https://readthedocs.org/docs/gcc-python-plugin/en/latest/cpychecker.html"&gt;static analysis tool&lt;/a&gt;, which has revealed several unchecked return values, possible
NULL dereferences, reference counting problems. Many thanks to David
Malcolm for the useful tool and the assistance provided using it.&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 2.4.4 released</title><link href="https://www.psycopg.org/articles/2011/12/19/psycopg-244-released/" rel="alternate"/><updated>2011-12-19T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:f310d659-f328-33db-bd28-090c24faf257</id><content type="html">&lt;p&gt;After a short discussion on this list we decided to change the definitions of isolation levels to make sure old code using numeric constants (both psycopg1 and psycopg2) continue to works. Other small fixes are included in the release: see below for details.&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.4.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.4.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="what-s-new-in-psycopg-2-4-4"&gt;
&lt;h2&gt;What's new in psycopg 2.4.4&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;tt class="docutils literal"&gt;register_composite()&lt;/tt&gt; also works with the types implicitly defined
after a table row, not only with the ones created by &lt;tt class="docutils literal"&gt;CREATE TYPE&lt;/tt&gt;.&lt;/li&gt;
&lt;li&gt;Values for the isolation level symbolic constants restored to what
they were before release 2.4.2 to avoid breaking apps using the
values instead of the constants.&lt;/li&gt;
&lt;li&gt;Named DictCursor/RealDictCursor honour itersize (ticket #80).&lt;/li&gt;
&lt;li&gt;Fixed rollback on error on Zope (ticket #73).&lt;/li&gt;
&lt;li&gt;Raise &lt;tt class="docutils literal"&gt;DatabaseError&lt;/tt&gt; instead of &lt;tt class="docutils literal"&gt;Error&lt;/tt&gt; with empty libpq errors,
consistently with other disconnection-related errors: regression
introduced in release 2.4.1 (ticket #82).&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 2.4.3 released</title><link href="https://www.psycopg.org/articles/2011/12/12/psycopg-243-released/" rel="alternate"/><updated>2011-12-12T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:aee566c5-cc87-3933-80c4-c08fc9bc3932</id><content type="html">&lt;p&gt;Mostly a bugfix release, with as usual a couple of small feature added:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.3.tar.gz"&gt;Download source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.3.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Here is what's new in this release:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="/docs/module.html#psycopg2.connect"&gt;connect()&lt;/a&gt; supports all the keyword arguments supported by the
database&lt;/li&gt;
&lt;li&gt;Added &lt;a class="reference external" href="/docs/extensions.html#psycopg2.extensions.new_array_type"&gt;new_array_type()&lt;/a&gt; function for easy creation of array
typecasters.&lt;/li&gt;
&lt;li&gt;Added support for arrays of &lt;a class="reference external" href="/docs/extras.html#hstore-data-type"&gt;hstores&lt;/a&gt; and &lt;a class="reference external" href="/docs/extras.html#composite-types-casting"&gt;composite types&lt;/a&gt; (ticket
#66).&lt;/li&gt;
&lt;li&gt;Fixed segfault in case of transaction started with connection lost
(and possibly other events).&lt;/li&gt;
&lt;li&gt;Fixed adaptation of Decimal type in sub-interpreters, such as in
certain mod_wsgi configurations (ticket #52).&lt;/li&gt;
&lt;li&gt;Rollback connections in transaction or in error before putting them
back into a &lt;a class="reference external" href="/docs/pool.html"&gt;pool&lt;/a&gt;. Also discard broken connections (ticket #62).&lt;/li&gt;
&lt;li&gt;Lazy import of the slow uuid module, thanks to Marko Kreen.&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;NamedTupleCursor.executemany()&lt;/tt&gt; (ticket #65).&lt;/li&gt;
&lt;li&gt;Fixed &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;--static-libpq&lt;/span&gt;&lt;/tt&gt; setup option (ticket #64).&lt;/li&gt;
&lt;li&gt;Fixed interaction between &lt;a class="reference external" href="/docs/extras.html#real-dictionary-cursor"&gt;RealDictCursor&lt;/a&gt; and &lt;a class="reference external" href="/docs/usage.html#server-side-cursors"&gt;named cursors&lt;/a&gt;
(ticket #67).&lt;/li&gt;
&lt;li&gt;Dropped limit on the columns length in &lt;a class="reference external" href="/docs/usage.html#using-copy-to-and-copy-from"&gt;COPY&lt;/a&gt; operations (ticket #68).&lt;/li&gt;
&lt;li&gt;Fixed reference leak with arguments referenced more than once
in queries (ticket #81).&lt;/li&gt;
&lt;li&gt;Fixed typecasting of arrays containing consecutive backslashes.&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="/docs/errorcodes.html"&gt;errorcodes&lt;/a&gt; map updated to PostgreSQL 9.1.&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Psycopg 2.4.2 released</title><link href="https://www.psycopg.org/articles/2011/06/12/psycopg-242-released/" rel="alternate"/><updated>2011-06-12T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:64ac7c86-a671-3b97-a385-592cc6195d35</id><content type="html">&lt;p&gt;Psycopg 2.4.2 has been released: it brings a few small but interesting new features, and a lot of bug fixes.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;Transaction control has been overhauled: a new connection method &lt;a class="reference external" href="/docs/connection.html#connection.set_session"&gt;set_session()&lt;/a&gt; allows setting all the session properties affecting the transactions behaviour: the &lt;a class="reference external" href="https://www.postgresql.org/docs/9.1/static/transaction-iso.html"&gt;isolation level&lt;/a&gt; but it can also be used to have auto-commit, read-only, and deferrable transactions. You can use for example:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_session&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'read committed'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_session&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;readonly&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;autocommit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_session&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'serializable'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;                 &lt;span class="n"&gt;readonly&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;deferrable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;There is also a welcome improvement for all the users thinking that&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_isolation_level&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;    &lt;span class="n"&gt;psycopg2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;extensions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ISOLATION_LEVEL_AUTOCOMMIT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;was an excessively verbose syntax: a new read/write property allows to set&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;autocommit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;a much handier syntax for an often used connection property.&lt;/p&gt;
&lt;p&gt;The improvements to the transactions control are not only at interface level: Psycopg doesn't require any more setup queries when connecting to a database. A sequence of statements:&lt;/p&gt;
&lt;pre class="code python literal-block"&gt;
&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;psycopg2&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psycopg2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;curs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;curs&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'SELECT 1'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;curs&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'SELECT 2'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/pre&gt;
&lt;p&gt;in older versions of the library would have resulted in the following commands sent to the backend:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
SHOW default_transaction_isolation
SET DATESTYLE TO 'ISO'
SHOW client_encoding
BEGIN; SET TRANSACTION ISOLATION LEVEL READ COMMITTED
SELECT 1
SELECT 2
COMMIT
&lt;/pre&gt;
&lt;p&gt;In Psycopg 2.4.2 the only commands sent are instead the essential:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
BEGIN
SELECT 1
SELECT 2
COMMIT
&lt;/pre&gt;
&lt;p&gt;with the &lt;tt class="docutils literal"&gt;BEGIN&lt;/tt&gt;/&lt;tt class="docutils literal"&gt;COMMIT&lt;/tt&gt; obviously omitted in autocommit mode (the datestyle and encoding statements were already dropped in 2.3).&lt;/p&gt;
&lt;p&gt;The new release also brings a lot of bug fixes, so we encourage updating soon, particularly if you use Psycopg in multithread programs:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed bug with multithread code potentially causing loss of sync
with the server communication or lock of the client (&lt;a class="reference external" href="https://psycopg.lighthouseapp.com/projects/62710/tickets/55"&gt;ticket #55&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Don't fail import if mx.DateTime module can't be found, even if its
support was built (&lt;a class="reference external" href="https://psycopg.lighthouseapp.com/projects/62710/tickets/53"&gt;ticket #53&lt;/a&gt;): a fix for the &amp;quot;ghost mx.DateTime&amp;quot; issue reported in virtualenv.&lt;/li&gt;
&lt;li&gt;Fixed escape for negative numbers prefixed by minus operator
(&lt;a class="reference external" href="https://psycopg.lighthouseapp.com/projects/62710/tickets/57"&gt;ticket #57&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed GC race condition during copy in multithread programs, potentially resulting in refcount errors.  Fixed by Dave
Malcolm (&lt;a class="reference external" href="https://psycopg.lighthouseapp.com/projects/62710/tickets/58"&gt;ticket #58&lt;/a&gt;, Red Hat &lt;a class="reference external" href="https://bugzilla.redhat.com/show_bug.cgi?id=711095"&gt;Bug 711095&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Trying to execute concurrent operations on the same connection
through concurrent green thread results in an error instead of a
deadlock.&lt;/li&gt;
&lt;li&gt;Fixed detection of pg_config on Windows. Report and fix, plus some
long needed &lt;tt class="docutils literal"&gt;setup.py&lt;/tt&gt; cleanup by Steve Lacy: thanks!&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>Building Psycopg on Windows using MinGW</title><link href="https://www.psycopg.org/articles/2011/06/05/psycopg-windows-mingw/" rel="alternate"/><updated>2011-06-05T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:154164ca-b9da-35ba-9185-dbe2408442ea</id><content type="html">&lt;p&gt;My goal was to install Psycopg on Windows using MinGW and the PostgreSQL binary package.&lt;/p&gt;
&lt;p&gt;I have used the &lt;a class="reference external" href="http://www.develer.com/oss/GccWinBinaries"&gt;MinGW GCC binaries&lt;/a&gt; packaged by Giovanni Bajo. The package takes care of a lot of details, for instance registering MinGW as default compiler for Python, plus some magic I don't even want to know, and makes the entire process simple enough.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;The first step is to ensure that &lt;tt class="docutils literal"&gt;setup.py&lt;/tt&gt; can find &lt;tt class="docutils literal"&gt;pg_config&lt;/tt&gt;. There is a bug causing it not to be found if &lt;em&gt;it is&lt;/em&gt; in the path: it will be fixed in Psycopg 2.4.2. On a few older versions you will have to specify the &lt;tt class="docutils literal"&gt;pg_config&lt;/tt&gt; path in the &lt;tt class="docutils literal"&gt;setup.cfg&lt;/tt&gt; or by the &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;--pg-config&lt;/span&gt;&lt;/tt&gt; option, e.g.&lt;/p&gt;
&lt;pre class="literal-block"&gt;
python setup.py build_ext --pg-config=C:\path\to\pg_config.exe build
&lt;/pre&gt;
&lt;p&gt;The library built depends on &lt;tt class="docutils literal"&gt;libpq.dll&lt;/tt&gt;, so at runtime this library should be available, e.g. on the path or it may be copied in the &lt;tt class="docutils literal"&gt;psycopg2&lt;/tt&gt; directory. The libpq in turn depends on a few other dlls, all found in the PostgreSQL &lt;tt class="docutils literal"&gt;bin&lt;/tt&gt; directory: &lt;tt class="docutils literal"&gt;libeay32.dll&lt;/tt&gt;, &lt;tt class="docutils literal"&gt;ssleay32.dll&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;libintl-8.dll&lt;/span&gt;&lt;/tt&gt;: they should be made available to the client as well. Unfortunately if any of these libraries is missing you will only get an &amp;quot;ImportError: dll load failed&amp;quot;. The problem is very easy to debug using a &lt;a class="reference external" href="http://www.dependencywalker.com/"&gt;dependency walker&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Another problem you may find is building for Python 2.6 and newer: some MinGW versions ship with a broken msvcr90 lib, and the result is again a rather unhelpful ImportError. the dependency walker is useful in this case too, showing the missing &lt;tt class="docutils literal"&gt;localtime&lt;/tt&gt; function in the library. The bug was reported in the &lt;a class="reference external" href="https://bugs.python.org/issue3308"&gt;issue 3308&lt;/a&gt;: my solution was to &lt;a class="reference external" href="http://tdm-gcc.tdragon.net/"&gt;download a newer MinGW&lt;/a&gt; and use the &lt;tt class="docutils literal"&gt;libmsvcr90.a&lt;/tt&gt; from there.&lt;/p&gt;
&lt;p&gt;So, overall, it can be done, but the result still depends on many dlls. I'm torn if the solution could be to have all the dependencies copied in the package directory. Of course you can still use &lt;a class="reference external" href="http://www.stickpeople.com/projects/python/win-psycopg/"&gt;Jason's binary package&lt;/a&gt;: he builds both the libpq and the openssl as static libs and creates a self-contained psycopg, which is probably the handiest result, but can't be obtained only using the PostgreSQL binaries.&lt;/p&gt;
</content></entry><entry><title>Psycopg 2.4.1 released</title><link href="https://www.psycopg.org/articles/2011/05/11/psycopg-241-released/" rel="alternate"/><updated>2011-05-11T00:00:00Z</updated><author><name>Federico Di Gregorio</name></author><id>urn:uuid:d50df4a2-7b6d-3b7e-8f19-5a05160a8443</id><content type="html">&lt;p&gt;Hi *,&lt;/p&gt;
&lt;p&gt;Daniele stacked another round of fixes on the devel branch, so it is
time for another release. So, as always, kudos to Daniele and here are
the direct download links:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.1.tar.gz"&gt;Source package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.1.tar.gz"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Release notes attached, as always. And al always, have fun,&lt;/p&gt;
&lt;p&gt;federico&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="what-s-new-in-psycopg-2-4-1"&gt;
&lt;h2&gt;What's new in psycopg 2.4.1&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Use own parser for bytea output, not requiring anymore the libpq 9.0
to parse the hex format.&lt;/li&gt;
&lt;li&gt;Don't fail connection if the client encoding is a non-normalized
variant. Issue reported by Peter Eisentraut.&lt;/li&gt;
&lt;li&gt;Correctly detect an empty query sent to the backend (&lt;a class="reference external" href="https://psycopg.lighthouseapp.com/projects/62710/tickets/46"&gt;ticket #46&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Fixed a SystemError clobbering libpq errors raised without SQLSTATE.
Bug vivisectioned by Eric Snow.&lt;/li&gt;
&lt;li&gt;Fixed interaction between NamedTuple and server-side cursors.&lt;/li&gt;
&lt;li&gt;Allow to specify --static-libpq on setup.py command line instead of
just in 'setup.cfg'. Patch provided by Matthew Ryan (&lt;a class="reference external" href="https://psycopg.lighthouseapp.com/projects/62710/tickets/46"&gt;ticket #48&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 2.4 released</title><link href="https://www.psycopg.org/articles/2011/02/27/psycopg-24-released/" rel="alternate"/><updated>2011-02-27T00:00:00Z</updated><author><name>Federico Di Gregorio</name></author><id>urn:uuid:10bf757f-5712-36e4-bc92-2b508690969a</id><content type="html">&lt;p&gt;Hi *,&lt;/p&gt;
&lt;p&gt;this is probably one of the best psycopg releases ever. Daniele, Jason
and all the others that sent patches did an impressive work to have
psycopg build and work flawlessly on all the supported platforms (well..
we can probably do a little bit better on MacOS but everything else is
almost perfect). So here it is (followed by NEWS excerpt, as always):&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.tar.gz"&gt;Source code&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="reference external" href="https://pypi.org/packages/source/p/psycopg2/psycopg2-2.4.tar.gz.asc"&gt;Signature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="line-block"&gt;
&lt;div class="line"&gt;Have fun,&lt;/div&gt;
&lt;div class="line"&gt;federico&lt;/div&gt;
&lt;/div&gt;
&lt;!-- CUT-HERE --&gt;
&lt;div class="section" id="what-s-new-in-psycopg-2-4"&gt;
&lt;h2&gt;What's new in psycopg 2.4&lt;/h2&gt;
&lt;p&gt;Added support for Python 3.1 and 3.2. The conversion has also
brought several improvements:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added 'b' and 't' mode to large objects: write can deal with both
bytes strings and unicode; read can return either bytes strings
or decoded unicode.&lt;/li&gt;
&lt;li&gt;COPY sends Unicode data to files implementing 'io.TextIOBase'.&lt;/li&gt;
&lt;li&gt;Improved PostgreSQL-Python encodings mapping.&lt;/li&gt;
&lt;li&gt;Added a few missing encodings: EUC_CN, EUC_JIS_2004, ISO885910,
ISO885916, LATIN10, SHIFT_JIS_2004.&lt;/li&gt;
&lt;li&gt;Dropped repeated dictionary lookups with unicode query/parameters.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Improvements to the named cusors:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;More efficient iteration on named cursors, fetching 'itersize'
records at time from the backend.&lt;/li&gt;
&lt;li&gt;The named cursors name can be an invalid identifier.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Improvements in data handling:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added 'register_composite()' function to cast PostgreSQL
composite types into Python tuples/namedtuples.&lt;/li&gt;
&lt;li&gt;Adapt types 'bytearray' (from Python 2.6), 'memoryview' (from
Python 2.7) and other objects implementing the &amp;quot;Revised Buffer
Protocol&amp;quot; to 'bytea' data type.&lt;/li&gt;
&lt;li&gt;The 'hstore' adapter can work even when the data type is not
installed in the 'public' namespace.&lt;/li&gt;
&lt;li&gt;Raise a clean exception instead of returning bad data when
receiving bytea in 'hex' format and the client libpq can't parse
them.&lt;/li&gt;
&lt;li&gt;Empty lists correctly roundtrip Python -&amp;gt; PostgreSQL -&amp;gt; Python.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Other changes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;'cursor.description' is provided as named tuples if available.&lt;/li&gt;
&lt;li&gt;The build script refuses to guess values if 'pg_config' is not
found.&lt;/li&gt;
&lt;li&gt;Connections and cursors are weakly referenceable.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Bug fixes:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed adaptation of None in composite types (ticket #26). Bug
report by Karsten Hilbert.&lt;/li&gt;
&lt;li&gt;Fixed several reference leaks in less common code paths.&lt;/li&gt;
&lt;li&gt;Fixed segfault when a large object is closed and its connection no
more available.&lt;/li&gt;
&lt;li&gt;Added missing icon to ZPsycopgDA package, not available in Zope
2.12.9 (ticket #30). Bug report and patch by Pumukel.&lt;/li&gt;
&lt;li&gt;Fixed conversion of negative infinity (ticket #40). Bug report and
patch by Marti Raudsepp.&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
</content></entry><entry><title>Psycopg 2.4 beta1 released</title><link href="https://www.psycopg.org/articles/2011/02/06/psycopg-24-beta1-released/" rel="alternate"/><updated>2011-02-06T00:00:00Z</updated><author><name>Federico Di Gregorio</name></author><id>urn:uuid:165c117b-f873-33a1-bd68-f56616701700</id><content type="html">&lt;p&gt;Hi *,&lt;/p&gt;
&lt;p&gt;me and Daniele, we just went through his series of patches (a loooot of
patches) and we're ready to release a beta version of the new psycopg.
Apart the usual changes and enhancements (detailed below) this is the
first version that supports Python 3 (thanks to, guess who? :)&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;The NEWS excerpt:&lt;/p&gt;
&lt;div class="section" id="new-features-and-changes"&gt;
&lt;h2&gt;New features and changes&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Added &lt;cite&gt;register_composite()&lt;/cite&gt; function to cast PostgreSQL composite
types into Python tuples/namedtuples.&lt;/li&gt;
&lt;li&gt;More efficient iteration on named cursors.&lt;/li&gt;
&lt;li&gt;The build script refuses to guess values if pg_config is not found.&lt;/li&gt;
&lt;li&gt;Connections and cursors are weakly referenceable.&lt;/li&gt;
&lt;li&gt;Added 'b' and 't' mode to large objects: write can deal with both
bytes strings and unicode; read can return either bytes strings or
decoded unicode.&lt;/li&gt;
&lt;li&gt;COPY sends Unicode data to files implementing io.TextIOBase.&lt;/li&gt;
&lt;li&gt;The build script refuses to guess values if pg_config is not found.&lt;/li&gt;
&lt;li&gt;Improved PostgreSQL-Python encodings mapping. Added a few
missing encodings: EUC_CN, EUC_JIS_2004, ISO885910, ISO885916,
LATIN10, SHIFT_JIS_2004.&lt;/li&gt;
&lt;li&gt;Dropped repeated dictionary lookups with unicode query/parameters.&lt;/li&gt;
&lt;li&gt;Empty lists correctly roundtrip Python -&amp;gt; PostgreSQL -&amp;gt; Python.&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;div class="section" id="bug-fixes"&gt;
&lt;h2&gt;Bug fixes&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;Fixed adaptation of None in composite types (ticket #26). Bug
report by Karsten Hilbert.&lt;/li&gt;
&lt;li&gt;Fixed several reference leaks in less common code paths.&lt;/li&gt;
&lt;li&gt;Fixed segfault when a large object is closed and its connection no
more available.&lt;/li&gt;
&lt;li&gt;Added missing icon to ZPsycopgDA package, not available in Zope
2.12.9 (ticket #30). Bug report and patch by Pumukel.&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="line-block"&gt;
&lt;div class="line"&gt;Have fun,&lt;/div&gt;
&lt;div class="line"&gt;federico&lt;/div&gt;
&lt;/div&gt;
&lt;/div&gt;
</content></entry><entry><title>psycopg2 porting to Python 3: a report</title><link href="https://www.psycopg.org/articles/2011/01/24/psycopg2-porting-python-3-report/" rel="alternate"/><updated>2011-01-24T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:839f4b37-ba01-3adf-a046-446154c847c8</id><content type="html">&lt;p&gt;I've mostly finished the porting of psycopg2 to Python 3. Here is a
report of what done and what can be improved.&lt;/p&gt;
&lt;!-- CUT-HERE --&gt;
&lt;p&gt;The code is available in the python3 branch of the repository
available on &amp;lt;&lt;a class="reference external" href="https://github.com/dvarrazzo/psycopg"&gt;https://github.com/dvarrazzo/psycopg&lt;/a&gt;&amp;gt;. The code is
compatible with both python2 and python3: the Python code is in py2
syntax: setup.py processes it with 2to3 before installation. The C
code uses a macros portability layer (in &lt;tt class="docutils literal"&gt;psycopg/python.h&lt;/tt&gt;) to have py2
and py3 code unified.&lt;/p&gt;
&lt;p&gt;A big chunk of the porting is by Martin von Löwis (who I thank
wholeheartedly): who provided a big patch back in 2008 (against 2.0.9,
IIRC). Unfortunately psycopg code diverged without the patch being
merged or maintained, so I basically used his macros but re-did the
work from scratch, refactoring the code instead of patching many
repetitive parts. On the pro side, since then, psycopg gained many
more tests.&lt;/p&gt;
&lt;p&gt;Large part of the porting has been mechanical, nothing to say about
that. What has required decisions has been instead the string
processing: Py3 uses extensively Unicode, but the communication with
the libpq is performed in char*; being psycopg a programmable
interface the point in which the conversion happens changes how
adapters and typecasters should be written.&lt;/p&gt;
&lt;div class="section" id="adapters"&gt;
&lt;h2&gt;Adapters&lt;/h2&gt;
&lt;p&gt;they are objects wrapping any Python object and returning a SQL
representation to be passed to the libpq. The adapters may have
returned either a str or an unicode, but a critical step is to pass
through libpq functions to have string and binary data escaped (e.g.
PQescapeStringConn). Because these functions are defined char* -&amp;gt;
char*, what makes sense for me was to force adapters to return bytes:
having them returning unicode would mean that unicode strings should
have been:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;converted to bytes&lt;/li&gt;
&lt;li&gt;escaped by the libpq&lt;/li&gt;
&lt;li&gt;converted back to unicode to be returned from the adapter (but at this point which encoding to use is not clear)&lt;/li&gt;
&lt;li&gt;merged to the query&lt;/li&gt;
&lt;li&gt;converted to bytes again to be sent to the socket&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The double encoding seems unnecessary, so I prefer to have adapters to
return bytes. Having them free to return either bytes or unicode makes
writing composite adapters trickier and more error prone, so my
decision is to raise an exception if after adaptation a non-bytes
object is returned.&lt;/p&gt;
&lt;p&gt;Having adapted objects as bytes means that the arguments must be
merged to the query as bytes: this operation is performed by not much
more than a &lt;tt class="docutils literal"&gt;query % args&lt;/tt&gt;. Unfortunately the % operator is not
available for bytes, so I have ported the &lt;tt class="docutils literal"&gt;PyString_Format&lt;/tt&gt; from Python
2.7 and adapted to work with the bytes (the Python license seems
allowing mixing derived code with the LGPL without problems).&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="typecasters"&gt;
&lt;h2&gt;Typecasters&lt;/h2&gt;
&lt;p&gt;These are function performing the opposite: they take the PostgreSQL
representation of a value and convert it into a Python object. They
receive bytes from the libpq of course. What I have currently
implemented is to convert this string to unicode before passing it to
the Python functions: because in Python the parsing functions mostly
take strings as argument (meaning unicode in py3), passing bytes to the typecasters would have meant that each of them
should have implemented about the same boilerplate, something like:&lt;/p&gt;
&lt;pre class="literal-block"&gt;
def caster(value, curs):
    value = value.decode(
        psycopg.encodings[curs.connection.encoding])
&lt;/pre&gt;
&lt;p&gt;but only in Py3, not in Py2. The current approach (passing an already decoded object) makes writing the typecasters easier, but has the drawback of making
impossible to write a Python typecaster for a binary type (but I don't
think there is really the need for such caster) and it is kinda
inconsistent with the adapters (dealing with bytes). What option would be better?&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="copy"&gt;
&lt;h2&gt;COPY&lt;/h2&gt;
&lt;p&gt;Copy operations deal with Python files or file-like objects. In input
(COPY IN) both unicode and bytes files are accepted; unicode is
converted in the connection encoding. In output (COPY OUT)... oops:
reviewing now I see I've overlooked this part: as it is now the data
(bytes) from the libpq are passed to &lt;tt class="docutils literal"&gt;file.write()&lt;/tt&gt; using
&lt;tt class="docutils literal"&gt;PyObject_CallFunction(func, &lt;span class="pre"&gt;&amp;quot;s#&amp;quot;,&lt;/span&gt; buffer, len)&lt;/tt&gt;. But this implies that
buffer is decoded from utf8 in Py3, so it would break if the
connection encoding was different. I've done a quick check and in Py3
a file open in text mode doesn't accept bytes, while one open in
binary mode doesn't accept unicode. Uhm... what could we pass this
file? Is there an interface in Python 3 to know if a file is binary or
text? Added ticket &lt;a class="reference external" href="https://psycopg.lighthouseapp.com/projects/62710/tickets/36"&gt;#36&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="large-objects"&gt;
&lt;h2&gt;Large Objects&lt;/h2&gt;
&lt;p&gt;These are open using a mode string such as &amp;quot;r&amp;quot;, &amp;quot;w&amp;quot;, &amp;quot;rw&amp;quot;. I have
added a format letter pretty much as the open() function in Py3: it
can be &amp;quot;b&amp;quot; or &amp;quot;t&amp;quot;. In binary mode the file always returns bytes (str
in py2, bytes [edit: not unicode] in py3). In text mode it always returns unicode
(unicode in py2, str in py3). The default is &amp;quot;b&amp;quot; in py2, &amp;quot;t&amp;quot; in py3.
writing to the file accepts both str and unicode. This means that in
Py2 everything is compatible, but there are a few features added
(unicode communication) and it's easy to write portable code by
specifying the mode &amp;quot;b&amp;quot; or &amp;quot;t&amp;quot;.&lt;/p&gt;
&lt;/div&gt;
&lt;div class="section" id="other-random-details"&gt;
&lt;h2&gt;Other random details&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;in py2 psycopg uses basic string as default, and unicode must be chosen specifically (e.g. registering the adapter, passing a unicode=True to certain functions etc.) In py3 there is no such choice and unicode is returned where there used to be a choice.&lt;/li&gt;
&lt;li&gt;bytea fields are returned as MemoryView, from which is easy to get bytes.&lt;/li&gt;
&lt;li&gt;&amp;quot;secondary strings&amp;quot; (notices, notifications, errors...) are decoded in the connection encoding, but I'm not be 100% sure that this will be always right, so the decoding is forgiving: decode(x, 'replace') for them.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This should be pretty much everything about the Py3 porting. Comments
are welcome, above all on the open points (typecasters and COPY OUT),
but if there is anything to point out I'd be happy to know. Best option would be to discuss on the &lt;a class="reference external" href="https://mail.postgresql.org/mj/mj_wwwusr/domain=postgresql.org?func=lists-long-full&amp;amp;extra=psycopg"&gt;mailing list&lt;/a&gt;. See you there!&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Update:&lt;/strong&gt; &lt;a class="reference external" href="/articles/2011/02/06/psycopg-24-beta1-released/"&gt;Psycopg 2.4 beta 1 has been released&lt;/a&gt; with Python 3 support.&lt;/p&gt;
&lt;/div&gt;
</content></entry><entry><title>New Psycopg Mailing List Online!</title><link href="https://www.psycopg.org/articles/2011/01/08/new-psycopg-mailing-list-online/" rel="alternate"/><updated>2011-01-08T00:00:00Z</updated><author><name>Daniele Varrazzo</name></author><id>urn:uuid:2c3a0088-8d9e-393b-a260-f58570ed297d</id><content type="html">&lt;p&gt;After a long while Psycopg has a Mailing List again!&lt;/p&gt;
&lt;p&gt;You are welcome to &lt;a class="reference external" href="https://mail.postgresql.org/mj/mj_wwwusr/domain=postgresql.org?func=lists-long-full&amp;amp;extra=psycopg"&gt;subscribe&lt;/a&gt;, either if you are an user dealing with the first stumbling blocks (albeit a look at the &lt;a class="reference external" href="/docs/"&gt;documentation&lt;/a&gt; or the &lt;a class="reference external" href="/docs/faq.html"&gt;FAQ&lt;/a&gt; wouldn't hurt!) or if you want to contribute to the psycopg2 development, about which there are several upcoming news.&lt;/p&gt;
&lt;p&gt;To post, send mail to &amp;lt;&lt;a class="reference external" href="mailto:psycopg&amp;#64;postgresql.org"&gt;psycopg&amp;#64;postgresql.org&lt;/a&gt;&amp;gt;. Yes, we are proud to be hosted on the mighty shoulders of the &lt;strong&gt;PostgreSQL Infrastructure team&lt;/strong&gt;. I want to thank them and the &lt;strong&gt;PostgreSQL Global Development Group&lt;/strong&gt; wholeheartedly for making this happen.&lt;/p&gt;
&lt;p&gt;See you there!&lt;/p&gt;
</content></entry></feed>