Tanner Rollefson

A Pagination Inefficiency in Flask-SQLAlchemy

At Pioneer I have been building an API that paginates through thousands of land parcel sales records. Under load the responses were slower than expected, so I dug into the flask-sqlalchemy pagination internals.

The issue is that every call to .paginate() runs a full COUNT(*) against the table to calculate the total number of pages. On a large table that count is expensive, and it runs on every single request regardless of whether the caller needs that information.

I opened a pull request to make the count optional. The fix adds a count parameter to .paginate() that defaults to True for backwards compatibility but lets you opt out when you do not need the total page count.

# before, always counts the full table
page = query.paginate(page=2, per_page=25)

# after, skip the COUNT(*) when you don't need it
page = query.paginate(page=2, per_page=25, count=False)

It is a small change, but it meaningfully reduces response times for any endpoint paginating through a large dataset without needing to report back a total page count.

LinkedIn GitHub LeetCode