# Polling events

The event feed reports two facts: a lead received its first message, and a lead replied. The loop below never loses or duplicates events, even after failures and downtime.

## How the feed works

`GET /v1/events` returns events in the order they enter the feed. There are two types:

| Type | When it appears |
|---|---|
| `lead.first_message_sent` | The lead received its first message, including one sent manually. |
| `lead.replied` | The lead replied. |

Feed properties:

- **Ordered by arrival, not by when the fact happened.** The `timestamp` field shows when the fact happened. An event with an earlier `timestamp` can arrive after its neighbor. Don't use `timestamp` as a cursor.
- **At-least-once delivery.** After a failure, a page can be delivered again. The deduplication key is `event.id`: `evt_` followed by 32 hexadecimal characters.
- **IDs match webhooks.** If webhooks are set up in your organization, the same fact has the same `id`. You can use the feed and webhooks together and deduplicate by `id`.
- **There is always a cursor.** `next_cursor` is never `null`, even on an empty page. Every response returns a new cursor, and you need to save it.
- **30-day retention.** Events older than 30 days leave the feed. A cursor is valid for 30 days after it is issued, and each request extends it.

What to know about event data:

- `lead.replied` arrives for every counted reply, so one lead can have several.
- `data.reply.automation_stopped` is currently always `null`: the event doesn't confirm that the sequence stopped. The `data.reply.is_first_reply` field is currently not sent.
- `data.first_message_sent_at` on a reply event is often `null`. For the time of the first message, rely on the `lead.first_message_sent` event or the lead record instead.
- `data.lead.fields.record_id` is currently `null`, and `record_id_status` is `unknown`. To link an event to your CRM, see the [CRM mapping](https://docs.melioralab.io/en/crm-mapping/#record-id) guide.
- `data.lead.username` is lowercase.

## First run

You can start in one of two ways:

- **From now.** A request with `start=now` returns an empty page and a cursor. The feed starts with events that appear after it. Fetch the past state by walking through [campaigns](https://docs.melioralab.io/en/reference/campaigns/) and [leads](https://docs.melioralab.io/en/reference/leads/).
- **From the available history.** A request without `cursor` and without `start` begins at the earliest event the feed holds, but no earlier than 30 days ago and no earlier than when event collection was enabled for your organization.

For a CRM integration, the first option is usually more convenient: get a cursor first, then reconcile the records, then read the feed from the cursor. That way, no change made during reconciliation is lost.

## Reliable loop

1. Load the saved cursor. If there is none, do a first run.
2. Request `GET /v1/events?cursor=<cursor>&limit=100`.
3. In **a single transaction** in your database, apply the events on the page and save the new `next_cursor`. Skip events whose `id` you have already applied.
4. If `has_more` is `true`, request the next page right away.
5. If `has_more` is `false`, wait a minute and repeat from step 2.

Why a transaction: if the process crashes between applying events and saving the cursor, it repeats the same page after a restart, and deduplication by `id` keeps any event from being applied twice. If you save the cursor before applying the events, a failure loses them.

> **Warning.** Don't change filters while polling with a saved cursor. The cursor is bound to the `workspace_id`, `campaign_id`, `type` and `include_text` of the first request. A different set of filters returns 400 `cursor_mismatch`. Use a separate cursor for each set of filters.

## Errors while polling

| Response | What to do |
|---|---|
| 429, 503 | Retry the same request with the same cursor after `Retry-After`. The cursor doesn't advance. |
| Network error, timeout | Retry with the same cursor. |
| 410 `cursor_expired` | The cursor is older than 30 days. Run recovery. |
| 400 `cursor_mismatch` | The key's permissions or scope changed, or the filters changed. Run recovery. |
| 401, 403 | The key was revoked or expired, or access was turned off. Stop polling and check the key. |

## Recovery after downtime

Run recovery after a 410 or `cursor_mismatch` response, after downtime longer than 30 days, and when `meta.coverage.status` reports that data may be incomplete.

1. Record the gap on your side: the time the stream broke.
2. Get a new anchor: `GET /v1/events?start=now`. Save its cursor only after step 3.
3. Walk through current [campaigns](https://docs.melioralab.io/en/reference/campaigns/) and [leads](https://docs.melioralab.io/en/reference/leads/) and update the records in your system. The lead's `first_message_sent_at`, `first_reply_at` and `status` fields show what happened during the gap.
4. Save the anchor cursor and continue the normal loop.

Events that appear during the walk-through arrive in the feed after the anchor. Merge them into your records monotonically: never move a status backward, and never replace a later time with an earlier one.

## Feed completeness

Every page includes `meta`:

- `meta.available_from`: the point from which the feed has events.
- `meta.coverage.status`: `complete`, `partial` or `unknown`. During early access you usually get `unknown` or `partial`: feed completeness is not guaranteed. For important processes, reconcile lead records periodically, for example once a day.
- `meta.coverage.gaps`: intervals during which events may not have been fully collected. After such an interval, reconcile the leads of the affected campaigns.
- `meta.collection.state`: the collection state. `degraded` means the data may be incomplete.

> **Note.** If the feed stays empty while messages and replies are going on in your organization, contact [Support](https://docs.melioralab.io/en/support/): during early access, we enable the event feed for each organization.

## Message text

With `include_text=true`, an event includes text: in `data.first_message` for `lead.first_message_sent`, and in `data.reply` for `lead.replied`. You need the `messages:read` permission and "Message text" enabled on the key. Text is included only if it was saved and proven to belong to the conversation. Otherwise `text` is `null`, and the reason is in `text_unavailable_reason`. A request with text has a weight of 3 instead of 2.

## Node.js example

Replace the storage functions with your own. `meliora()`, with retries on 429 and 503, is described in the [Rate limits and retries](https://docs.melioralab.io/en/rate-limits/#povtor-bezopasen) guide.

```js
const POLL_INTERVAL_MS = 60_000;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function runEventLoop(db) {
  let cursor = await db.loadCursor();
  if (!cursor) cursor = await resync(db);

  for (;;) {
    let page;
    try {
      page = await meliora('/events', { cursor, limit: 100 });
    } catch (err) {
      if (err.status === 410 || err.code === 'cursor_mismatch') {
        cursor = await resync(db);
        continue;
      }
      throw err; // 401 and 403: stop and check the key
    }

    await db.transaction(async (tx) => {
      for (const event of page.data) {
        if (await tx.isApplied(event.id)) continue;
        if (event.type === 'lead.first_message_sent' || event.type === 'lead.replied') {
          await applyToCrm(tx, event);
        }
        await tx.markApplied(event.id); // mark unknown types as applied too
      }
      await tx.saveCursor(page.next_cursor);
    });
    cursor = page.next_cursor;

    if (!page.has_more) await sleep(POLL_INTERVAL_MS);
  }
}

async function resync(db) {
  await db.recordGap(new Date());
  const anchor = await meliora('/events', { start: 'now' });
  await reconcileLeads(db); // walk /v1/leads and update records
  await db.saveCursor(anchor.next_cursor);
  return anchor.next_cursor;
}
```

## Python example

```python
import time
from datetime import datetime, timezone

POLL_INTERVAL = 60

def run_event_loop(db):
    cursor = db.load_cursor() or resync(db)
    while True:
        try:
            page = meliora("/events", {"cursor": cursor, "limit": 100})
        except MelioraError as err:
            if err.status == 410 or err.code == "cursor_mismatch":
                cursor = resync(db)
                continue
            raise  # 401 and 403: stop and check the key

        with db.transaction() as tx:
            for event in page["data"]:
                if tx.is_applied(event["id"]):
                    continue
                if event["type"] in ("lead.first_message_sent", "lead.replied"):
                    apply_to_crm(tx, event)
                tx.mark_applied(event["id"])  # mark unknown types as applied too
            tx.save_cursor(page["next_cursor"])
        cursor = page["next_cursor"]

        if not page["has_more"]:
            time.sleep(POLL_INTERVAL)

def resync(db):
    db.record_gap(datetime.now(timezone.utc))
    anchor = meliora("/events", {"start": "now"})
    reconcile_leads(db)  # walk /v1/leads and update records
    db.save_cursor(anchor["next_cursor"])
    return anchor["next_cursor"]
```

The `meliora()` function and the `MelioraError` class are in the [Rate limits and retries](https://docs.melioralab.io/en/rate-limits/#povtor-bezopasen) guide. To match an event to a CRM record, see the [CRM mapping](https://docs.melioralab.io/en/crm-mapping/) guide.

## How often to poll

Once a minute is enough for most integrations. That uses 2 weight units per minute of the key's limit. You can poll less often, for example once an hour: a cursor is valid for 30 days, and the feed keeps events for 30 days. While `has_more` is `true`, fetch pages back to back without pausing. Polling more often doesn't make events appear sooner: they enter the feed as data is collected, and the length of this delay is not guaranteed.