1. Choose one initial support task
Start with a recurring task whose answer is documented, such as shipping policy or product care. Identify the channels and languages you need. Compare the required workflow with the current Gorgias AI Agent or Tidio Lyro product documentation and your store integration.
2. Prepare the knowledge sources
Update shipping, returns, warranty, product, and contact information. Remove contradictory rules and expired promotions. Define when the agent should ask for clarification or hand off to a person. Keep order-specific data out of generic help content.
3. Test a shared set of conversations
Use realistic examples from your support queue with personal information removed. Include normal questions, missing knowledge, conflicting requests, and escalation. Test action permissions separately from answer quality.
- Can it answer a documented question accurately?
- Does it admit when an answer is unavailable?
- Does it reject an action that your policy or permissions do not allow?
- Does a person receive enough context after handoff?
- Are usage and action logs available for review?
Test ChatGPT Business Agent knowledge separately from Shopify storefronts
A Shopify merchant evaluating pre-sale support in ChatGPT should distinguish three surfaces: Shopify Catalog makes eligible products discoverable; the optional ChatGPT Business app can help claim the business profile; and a separately reviewed and published Business Agent answers customer questions using information supplied for it. Connecting a product feed does not publish an agent or guarantee a recommendation. The customer completes a purchase at the merchant’s online-store checkout, not through the Business Agent.
As of November 12, 2026, OpenAI describes Business Agents for eligible US ecommerce and online-service businesses with a website, subject to its industry policy. Shopify specifies eligible US-based stores. This is narrower than Shopify’s ChatGPT storefront requirement: a store can be based outside the US if it sells to US customers and meets the other requirements. Claiming a profile and creating an agent have separate eligibility and verification steps; check the current rules for your business.
Shopify says its agentic-storefront knowledge base has no connection to Business Hub, the profile or the Business Agent. Inventory the approved product facts, policies and instructions for each surface rather than assuming an edit in one updates another. After claim approval and Persona verification, review the information used by any draft agent and preview its answers before the separate publish step. OpenAI says businesses do not receive individual customer prompts or Business Agent chats, so do not assume the pilot has a transcript-review channel.
The cases below are a proposed merchant acceptance set, not results of tests by EcomAgentHub or proof of any store integration. Use synthetic examples, dated source snapshots and authorized preview access. Review answers separately from transactional workflows: the current Business Agent can explain products and policies and link to the website, but OpenAI says it cannot complete purchases or bookings or check a customer’s order.
| Case | Prepare | Accept when | Escalate or stop when |
|---|---|---|---|
| Storefront and agent disagree | Compare answers to the same synthetic return-policy question in an authorized storefront test fixture and draft-agent preview. Retain the approved policy snapshot and timestamp for each surface; do not insert conflicting policies into a live store. | The reviewer identifies which surface supplied the answer and corrects the appropriate source before launch. Recheck both independently. | A correct storefront answer is treated as proof the agent uses the same knowledge, or the source of a conflicting answer cannot be determined. |
| Changed product fact | Update a synthetic product specification in the approved source; record the prior and new version and the time of the change. | The answer reflects the approved fact after the relevant update and review, or clearly signals uncertainty until it can be confirmed. | An obsolete specification is presented confidently. Do not promise an immediate propagation time. |
| Unknown or contradictory policy | Give the preview a question for which the agent’s approved policy is absent or conflicts with another source. | It avoids inventing a rule and directs the shopper to an appropriate merchant contact or policy page. | It states an unsupported return deadline or promises an exception. |
| Order-specific request | Ask a synthetic shopper’s order status or request a refund without providing real customer data. | The answer does not claim to inspect or change the order and points to the merchant’s appropriate support route. | It claims an order was checked, refunded or changed without evidence from an independently authorized workflow. |
| Purchase handoff | Ask to buy a product from a synthetic test catalog without entering payment details. | The response makes the next step clear and does not claim a completed purchase; the merchant tests the checkout separately. | It claims that the Business Agent placed an order or processed a payment. |
Record evidence without implying access to customer chats
For each preview, retain the case ID, date, agent configuration, approved source version, question, observed answer and reviewer decision where your preview process permits. Keep a failed or unavailable attempt as such and repeat after correction. These examples do not authorize a purchase, account connection, publication or use of real customer information.
Before pre-sale guidance goes live, test the buying decision
A shopping assistant should establish what the shopper needs, check the proposed variant against those needs, and explain when it cannot recommend a purchase. A fluent product answer or an add-to-cart event does not establish that the recommendation fits. Use this additional checklist when your support workflow includes pre-sale product guidance.
Visa's November 5, 2026 announcement reports that 42.8% of runs failed its Agentic Commerce Benchmark. This is a shopping-decision failure rate, not a payment failure rate. The October research snapshot covers 22 models, 15 scenarios and three attempts per model and scenario; only 34.2% of model-scenario groups passed all three attempts. These are Visa's reported results, not tests of the tools listed here or a forecast for your store.
The whitepaper uses a fixed simulated market and a simulated customer. Passing requires a justified selection or justified refusal: a lucky product match without establishing the required customer facts does not pass. Checkout, payment and order placement are outside its current scope. Most episodes are private, and the authors identify calibration against human shopping sessions as future work. Sources were reviewed November 6, 2026; EcomAgentHub has not reproduced the benchmark.
The merchant procedure below is our proposed acceptance method, not Visa certification or an experiment we performed. Prepare a small, dated test catalog and approved policy snapshot. Write an answer key listing acceptable variants, required facts, disqualifying constraints and unknown fields. Let a staff member play the shopper from a separate script: reveal a compatibility need, budget or measurement only when the assistant asks a relevant question. Use synthetic customer details and a staging or mocked workflow that cannot place real orders.
| Case | Prepare | Accept only when | Record as a failure or unresolved outcome |
|---|---|---|---|
| An incomplete request | A vague opening request plus a withheld device model, size limit or delivery requirement. | The assistant asks for the deciding fact or requests a measurement before recommending a specific variant. | It treats silence as permission, invents the missing fact, or happens to pick a suitable item without checking. |
| A suitable product exists | One verified compatible variant and one attractive near miss that breaks a single requirement. | The recommendation identifies the exact variant and supports each requirement with approved evidence; applicable cost and policy conditions remain visible. | It recommends the near miss, omits a necessary accessory, or says nothing fits despite verified eligible stock. |
| No suitable product exists | A complete test selection in which every item fails at least one required constraint. | It explains why it cannot recommend an item from this selection, preserves the shopper's constraints and offers a useful handoff. | It pushes a sale, silently relaxes the budget or compatibility requirement, or claims no product exists anywhere. |
| A deciding field is unknown | A product record with no approved evidence for a crucial specification or delivery promise. | It names the uncertainty, asks for verification or hands off, and avoids presenting a purchase as justified. | It invents compatibility or delivery information. Do not mistake a legitimate unresolved handoff for proof that no suitable product exists. |
Repeat the same cases without hiding the failed attempts
Run each case in fresh conversations with the same catalog snapshot, shopper script, assistant version and permissions. Three repetitions can expose inconsistency as an initial check; they are not statistical assurance or a universal launch threshold. Keep every attempt. Label a timeout, unavailable integration and completed but incorrect recommendation separately instead of replacing an inconvenient result with a retry.
Report both passed attempts out of all attempts and cases that passed every repetition out of all cases. Keep expected no-buy outcomes separate from unresolved handoffs and unjustified refusals. A higher purchase rate is not automatically better. For this proposed gate, an unsupported compatibility claim or a purchase recommendation that breaks a required constraint should block expansion of that workflow until corrected and retested.
Keep a review record that explains the decision
For each attempt, record: case and run ID; catalog and policy version; assistant configuration; opening request; required facts and how they were established; exact variant or no-buy decision; evidence URLs or record IDs; unresolved fields; transcript; expected versus observed outcome; reviewer and next action. Save the original failure when retesting a fix.
Evaluate recommendation quality separately from action authority. This checklist grants no permission to change a cart, charge a card or place an order. If those capabilities are later introduced, test consent, authentication, duplicate-action handling and the actual checkout integration in a separately authorized test environment. The Visa results do not validate those stages or any Shopify app.
Before a store-credit action, review the note and the transaction separately
A transaction note can preserve why an approved store-credit adjustment was made; it does not approve the adjustment or prove that it succeeded. Shopify announced optional merchant notes on November 6, 2026 for GraphQL Admin API 2027-01 and later. At our November 8 review, the versioned reference labels 2027-01 a release candidate. Versions 2026-10 and earlier do not expose these fields: verify the integration's actual API version before planning a rollout.
The credit and debit inputs accept a note of up to 5,000 characters. Shopify says customers cannot see it and it cannot be edited after the transaction is created. An overlong note returns NOTE_TOO_LONG and no transaction is created. Notes are optional; this announcement does not require existing integrations to add them. Existing access scopes and permission requirements still apply.
Treat this as a vendor acceptance requirement, not an announced capability of Gorgias AI Agent, Tidio Lyro or another listed tool. Ask the integration owner to demonstrate both note submission and read-back for the configured workflow. EcomAgentHub reviewed the public documentation but has not created store-credit transactions, tested an app integration or verified any vendor's support for this field.
| Case | Evidence to collect | Acceptance boundary |
|---|---|---|
| Approved adjustment with a note | Approved account, currency, amount and case reference; returned transaction ID, note, amount and balance after the transaction; any userErrors. | The stored note and transaction match the approved request. A drafted note or a successful chat reply alone is insufficient. |
| Length boundary | Using synthetic text in an authorized development store, compare the documented 5,000-character limit with a 5,001-character note. Keep the returned errors and transaction result. | For an otherwise valid request, confirm the boundary behavior. The over-limit case must surface NOTE_TOO_LONG without creating the transaction; do not silently cut off the explanation. |
| No note supplied | Record the transaction type and read-back separately from its note value. | A null note alone does not establish failure. Shopify documents null for an unset note and always for expiration transactions; assess the actual transaction result. |
| Response missing or explanation needs correction | Retain the original request reference and known transaction ID, inspect the account history, and link any clarification in the support case. | An unknown outcome is unresolved. Do not repeat a balance-changing action merely to obtain a response or rewrite an immutable note; escalate the accounting decision to an authorized person. |
Keep approval, explanation and execution evidence connected
Use an internal case reference to join the support conversation, approval and resulting transaction. Keep the detailed evidence in the support system; a merchant-only field should not become a copy of sensitive customer conversations. The following record is an editorial recommendation, not a Shopify-required note format.
- Before execution: verify the account and currency, the reason allowed by store policy, the amount and the person or workflow authorized to approve it. A note is not a substitute for these checks.
- Before submission: review the final explanation and its length. A synthetic example is: CASE-EXAMPLE-001; policy exception approved in the linked support case. Do not claim approval that has not occurred.
- After execution: retain the returned transaction ID, created time, amount, balance, stored note and errors with the case reference. Reconcile unresolved results before considering another write.
- If the explanation was wrong: preserve the original transaction evidence and add a dated clarification in the support case. A separate balance correction needs its own authorization; changing money is not a way to edit text.
4. Start with a limited pilot
Enable a narrow workflow for a defined audience or channel where your chosen product supports it. Assign a person to review incorrect answers and, where the product makes them available, transcripts. Business Agent conversations are not shared with businesses. Keep account-changing actions limited to the permissions and identity checks you have verified. Pause or reduce the scope if the agent repeatedly provides incorrect information.
5. Review quality and cost together
Track correct resolutions, escalations, incorrect actions, customer feedback, review time, and total service charges. Estimate both ordinary and peak-month cost. Increase coverage only when the evidence supports it. This checklist is a suggested evaluation method; neither product has been hands-on tested by this directory.

