Polling events

The event feed reports two facts: a lead received its first message, and a lead replied. The loop below never loses or duplicates events, even after failures and downtime.

#How the feed works

GET /v1/events returns events in the order they enter the feed. There are two types:

Feed properties:

• Ordered by arrival, not by when the fact happened. The timestamp field shows when the fact happened. An event with an earlier timestamp can arrive after its neighbor. Don't use timestamp as a cursor.
• At-least-once delivery. After a failure, a page can be delivered again. The deduplication key is event.id: evt_ followed by 32 hexadecimal characters.
• IDs match webhooks. If webhooks are set up in your organization, the same fact has the same id. You can use the feed and webhooks together and deduplicate by id.
• There is always a cursor. next_cursor is never null, even on an empty page. Every response returns a new cursor, and you need to save it.
• 30-day retention. Events older than 30 days leave the feed. A cursor is valid for 30 days after it is issued, and each request extends it.

What to know about event data:

• lead.replied arrives for every counted reply, so one lead can have several.
• data.reply.automation_stopped is currently always null: the event doesn't confirm that the sequence stopped. The data.reply.is_first_reply field is currently not sent.
• data.first_message_sent_at on a reply event is often null. For the time of the first message, rely on the lead.first_message_sent event or the lead record instead.
• data.lead.fields.record_id is currently null, and record_id_status is unknown. To link an event to your CRM, see the CRM mapping guide.
• data.lead.username is lowercase.

#First run

You can start in one of two ways:

• From now. A request with start=now returns an empty page and a cursor. The feed starts with events that appear after it. Fetch the past state by walking through campaigns and leads.
• From the available history. A request without cursor and without start begins at the earliest event the feed holds, but no earlier than 30 days ago and no earlier than when event collection was enabled for your organization.

For a CRM integration, the first option is usually more convenient: get a cursor first, then reconcile the records, then read the feed from the cursor. That way, no change made during reconciliation is lost.

#Reliable loop

1. Load the saved cursor. If there is none, do a first run.
2. Request GET /v1/events?cursor=<cursor>&limit=100.
3. In a single transaction in your database, apply the events on the page and save the new next_cursor. Skip events whose id you have already applied.
4. If has_more is true, request the next page right away.
5. If has_more is false, wait a minute and repeat from step 2.

Why a transaction: if the process crashes between applying events and saving the cursor, it repeats the same page after a restart, and deduplication by id keeps any event from being applied twice. If you save the cursor before applying the events, a failure loses them.

#Errors while polling

#Recovery after downtime

Run recovery after a 410 or cursor_mismatch response, after downtime longer than 30 days, and when meta.coverage.status reports that data may be incomplete.

1. Record the gap on your side: the time the stream broke.
2. Get a new anchor: GET /v1/events?start=now. Save its cursor only after step 3.
3. Walk through current campaigns and leads and update the records in your system. The lead's first_message_sent_at, first_reply_at and status fields show what happened during the gap.
4. Save the anchor cursor and continue the normal loop.

Events that appear during the walk-through arrive in the feed after the anchor. Merge them into your records monotonically: never move a status backward, and never replace a later time with an earlier one.

#Feed completeness

Every page includes meta:

• meta.available_from: the point from which the feed has events.
• meta.coverage.status: complete, partial or unknown. During early access you usually get unknown or partial: feed completeness is not guaranteed. For important processes, reconcile lead records periodically, for example once a day.
• meta.coverage.gaps: intervals during which events may not have been fully collected. After such an interval, reconcile the leads of the affected campaigns.
• meta.collection.state: the collection state. degraded means the data may be incomplete.

#Message text

With include_text=true, an event includes text: in data.first_message for lead.first_message_sent, and in data.reply for lead.replied. You need the messages:read permission and "Message text" enabled on the key. Text is included only if it was saved and proven to belong to the conversation. Otherwise text is null, and the reason is in text_unavailable_reason. A request with text has a weight of 3 instead of 2.

#Node.js example

Replace the storage functions with your own. meliora(), with retries on 429 and 503, is described in the Rate limits and retries guide.

const POLL_INTERVAL_MS = 60_000;
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function runEventLoop(db) {
  let cursor = await db.loadCursor();
  if (!cursor) cursor = await resync(db);

  for (;;) {
    let page;
    try {
      page = await meliora('/events', { cursor, limit: 100 });
    } catch (err) {
      if (err.status === 410 || err.code === 'cursor_mismatch') {
        cursor = await resync(db);
        continue;
      }
      throw err; // 401 and 403: stop and check the key
    }

    await db.transaction(async (tx) => {
      for (const event of page.data) {
        if (await tx.isApplied(event.id)) continue;
        if (event.type === 'lead.first_message_sent' || event.type === 'lead.replied') {
          await applyToCrm(tx, event);
        }
        await tx.markApplied(event.id); // mark unknown types as applied too
      }
      await tx.saveCursor(page.next_cursor);
    });
    cursor = page.next_cursor;

    if (!page.has_more) await sleep(POLL_INTERVAL_MS);
  }
}

async function resync(db) {
  await db.recordGap(new Date());
  const anchor = await meliora('/events', { start: 'now' });
  await reconcileLeads(db); // walk /v1/leads and update records
  await db.saveCursor(anchor.next_cursor);
  return anchor.next_cursor;
}

#Python example

import time
from datetime import datetime, timezone

POLL_INTERVAL = 60


def run_event_loop(db):
    cursor = db.load_cursor() or resync(db)
    while True:
        try:
            page = meliora("/events", {"cursor": cursor, "limit": 100})
        except MelioraError as err:
            if err.status == 410 or err.code == "cursor_mismatch":
                cursor = resync(db)
                continue
            raise  # 401 and 403: stop and check the key

        with db.transaction() as tx:
            for event in page["data"]:
                if tx.is_applied(event["id"]):
                    continue
                if event["type"] in ("lead.first_message_sent", "lead.replied"):
                    apply_to_crm(tx, event)
                tx.mark_applied(event["id"])  # mark unknown types as applied too
            tx.save_cursor(page["next_cursor"])
        cursor = page["next_cursor"]

        if not page["has_more"]:
            time.sleep(POLL_INTERVAL)


def resync(db):
    db.record_gap(datetime.now(timezone.utc))
    anchor = meliora("/events", {"start": "now"})
    reconcile_leads(db)  # walk /v1/leads and update records
    db.save_cursor(anchor["next_cursor"])
    return anchor["next_cursor"]

The meliora() function and the MelioraError class are in the Rate limits and retries guide. To match an event to a CRM record, see the CRM mapping guide.

#How often to poll

Once a minute is enough for most integrations. That uses 2 weight units per minute of the key's limit. You can poll less often, for example once an hour: a cursor is valid for 30 days, and the feed keeps events for 30 days. While has_more is true, fetch pages back to back without pausing. Polling more often doesn't make events appear sooner: they enter the feed as data is collected, and the length of this delay is not guaranteed.