# Pagination

Many APIs return data in pages rather than all at once. Weavestream can automatically follow pagination to fetch all your data. You configure pagination in the Mapping step when adding or editing an endpoint.

# Pagination Types

# None

The default. Weavestream makes a single request and uses whatever the API returns. Choose this if:

• The API returns all items in one response
• You only need the first page of results
• The API doesn't support pagination

# Cursor-Based

Cursor-based pagination uses a token or cursor value from each response to request the next page. This is common in modern APIs.

How it works:

1. Weavestream makes the first request
2. It looks for a cursor value in the response at the path you specify
3. If a cursor exists, it makes another request including that cursor
4. This continues until no more cursor values are returned

You'll need to configure:

• Cursor Field — The path in the response where the next-page cursor lives. For example:
  • nextPageToken — Common in Google APIs
  • paging.cursor — Nested cursor field
  • meta.next_cursor — Another common pattern

# Page-Based

Page-based pagination uses incrementing page numbers. Weavestream starts at page 1 and keeps requesting the next page until no more items are returned.

You'll need to configure:

• Page Parameter — The query parameter name for the page number (e.g., page, p)
• Page Size Parameter (optional) — The query parameter for how many items per page (e.g., per_page, limit, count)

# Common Settings

Both cursor-based and page-based pagination share these settings:

# Page Size

How many items to request per page. Adjustable from 10 to 100 in steps of 10. The default depends on your API, but 50 is a common choice.

This value is sent as a query parameter (for page-based pagination) or included in the request to hint at the desired page size.

# Max Items

The maximum total number of items to fetch across all pages. Adjustable from 50 to 500 in steps of 50. This prevents Weavestream from fetching thousands of items from APIs with very large datasets.

Once Weavestream has fetched this many items (across however many pages it takes), it stops paginating.

# Examples

# GitHub Issues (Page-Based)

GitHub's API uses page and per_page parameters:

• Pagination Type: Page-based
• Page Parameter: page
• Page Size Parameter: per_page
• Page Size: 50
• Max Items: 200

# YouTube Videos (Cursor-Based)

YouTube's API uses nextPageToken in the response:

• Pagination Type: Cursor-based
• Cursor Field: nextPageToken
• Page Size: 50
• Max Items: 200

# Stripe Charges (Cursor-Based)

Stripe uses has_more and cursor IDs:

• Pagination Type: Cursor-based
• Cursor Field: has_more (or the ID of the last item, depending on the API)
• Page Size: 100
• Max Items: 500

# Tips

• Start with a small Max Items while testing. You can always increase it later.
• Check your API's rate limits. Each page is a separate request. If you set a large Max Items with a small Page Size, Weavestream may need many requests.
• Not sure which type? Check your API's documentation for terms like "cursor," "nextPageToken," or "offset" (cursor-based) vs. "page," "page_number" (page-based).

# Next Steps

• Configuring Endpoints — Full endpoint setup
• Syncing & Refresh — Control when data is fetched