Handling Cursor and Offset Pagination

List APIs use two types of pagination protocols. Before calling, refer to the parameters and response fields of the current API page. Do not assume that all list APIs follow the same approach.

Cursor Pagination

The cursor is generated by the server. For the first request, use an empty cursor or omit cursor. Subsequent requests always use the nextCursor from the previous response.

def collect_by_cursor(fetch_page):
    records = []
    cursor = ""
    while True:
        page = fetch_page(cursor=cursor, size=100)
        records.extend(page["items"])
        if not page.get("hasMore"):
            return records
        cursor = page["nextCursor"]

Do not calculate the cursor yourself, and do not skip to the next cursor directly after a failed request. Only save nextCursor after the current page has been processed successfully.

Offset Pagination

Offset indicates the starting point of the current page, and limit indicates the number of items per page. Advance offset each time based on the actual number returned, until the number of items read reaches total or the current page is empty.

def collect_by_offset(fetch_page):
    records = []
    offset = 0
    while True:
        page = fetch_page(offset=offset, limit=100)
        items = page["items"]
        records.extend(items)
        offset += len(items)
        if offset >= page["total"] or not items:
            return records

Production Checklist

• Set a timeout, limited retries, and rate limiting for each page request.
• Deduplicate using resource IDs to handle data additions or deletions during pagination.
• For long-running tasks, save completed pages and pagination markers to support resuming after failures.
• Do not set size or limit to a value exceeding the maximum allowed by the API.

Reference: Original example.