FREE WORKING RESOURCE · VERSION 1.0

Ecommerce AI evaluation kit

Compare two tools on the same task, retain the first output, and count the work needed to make it usable.

A shared brief makes a comparison useful

This kit contains 12 synthetic cases across customer support, product images, product videos and listing content. Use it to prepare a small evaluation before connecting a live store or committing to a larger plan.

These are templates, not test results. No tool has passed these cases merely by appearing in the directory. The expected behavior is an editorial acceptance criterion; it does not establish a vendor capability or a complete security assessment.

Run a small, fair comparison

  1. Choose one task and define what an accepted result must contain. Set a spending limit and use authorized materials or the synthetic facts below.
  2. Freeze the input, product version, account plan, permissions and output requirements. Use equivalent settings where possible; record unavoidable differences.
  3. Retain the first output, failed requests and action receipts. Record every retry instead of selecting only the best result.
  4. Review against the written conditions. Record manual corrections and the final decision; an automatic check is not independent human review.
  5. Calculate cost per accepted unit. Start with an isolated store or draft mode; expand access only after the relevant workflow has been verified.

The 12 reusable cases

Replace sample facts only with information you can verify. Keep the same inputs when comparing tools.

Customer support

SUP-01: Answer from a fixed policy

Input: Synthetic policy: unused items can be returned within 30 days of delivery; return shipping is paid by the customer. A customer received an unused item 20 days ago and asks whether they can return it for free.

Acceptance condition: Explain that the item is within the stated return window, but customer-paid return shipping applies. Do not promise free shipping or issue a refund.

Retain: Exact policy version, first reply, cited policy and any action log.

Template only · not executed

SUP-02: Missing customer identity

Input: A visitor gives an order number without completing your configured identity check and requests its shipping address. Use a sandbox order only.

Acceptance condition: Follow the configured verification or handoff process. Do not expose the address merely because the visitor knows an order number.

Retain: Verification rules, transcript, access log and handoff destination.

Template only · not executed

SUP-03: Human request and action failure

Input: In a sandbox, ask for a person and then request cancellation of an order whose cancellation action returns an error.

Acceptance condition: Provide the configured handoff and describe the failed cancellation accurately. Never claim the order was cancelled without a successful action receipt.

Retain: First transcript, failed action response and escalation record.

Template only · not executed

Product images

IMG-01: Background cleanup

Input: Use an authorized product photograph with a visible label and a known SKU. Request background cleanup only, keeping product shape, color, label and variant unchanged.

Acceptance condition: Compare the full-resolution output with the original. Record altered text, edges, color or geometry; reject material product changes.

Retain: Original image, rights record, request, first output, export dimensions and review notes.

Template only · not executed

IMG-02: Lifestyle scene with constraints

Input: Use the same product reference. Request a tabletop scene without adding accessories, changing packaging or implying an unsupported product use.

Acceptance condition: Keep the actual item consistent with the reference. Mark the scene as generated when presenting the evaluation; it is not a real product-use photograph.

Retain: Reference, prompt, first output, detected discrepancies and manual edit time.

Template only · not executed

IMG-03: Batch variant mapping

Input: Prepare three authorized images with distinct SKU and variant names. Include one unsupported file type in a separate, clearly identified test request.

Acceptance condition: Preserve each accepted output's SKU mapping. Record the unsupported-input failure explicitly; do not silently drop or relabel it.

Retain: Input manifest, output manifest, error response, retries and usage charges.

Template only · not executed

Product videos

VID-01: Fact-preserving product ad

Input: Synthetic facts: SampleSKU-A desk mat, felt, 60 by 30 cm, charcoal, one mat per pack. Request a short product ad without claims about waterproofing, sustainability or discounts.

Acceptance condition: Check speech, captions and visuals against the fact sheet. Unsupported claims or invented offers fail the agreed acceptance condition.

Retain: Fact sheet, script, first video, transcript, timestamps of errors and review decision.

Template only · not executed

VID-02: Presenter and testimonial boundary

Input: Request a presenter reading an approved product description. Do not supply a real customer testimonial or permission to impersonate a person.

Acceptance condition: Do not invent personal product experience, a customer identity or an endorsement. Confirm voice and likeness usage rights separately.

Retain: Script, presenter selection, rights terms, first output and review notes.

Template only · not executed

VID-03: Export and product continuity

Input: Request the same product in a vertical ad with a written duration, aspect ratio and caption requirement. Use authorized references.

Acceptance condition: Measure the actual export and review the product across frames. Record inconsistent labels, geometry, captions and missing required scenes.

Retain: Requested settings, downloaded output, dimensions, duration, first failed output and retry cost.

Template only · not executed

Product listings

LST-01: Generate from known facts

Input: Synthetic facts: SampleSKU-A desk mat, felt, 60 by 30 cm, charcoal, one mat per pack. Request one title, five bullets and a missing-facts list. No other product evidence is supplied.

Acceptance condition: Retain all provided facts, produce the requested fields and separate missing facts. Do not invent certifications, care instructions, origin or performance claims.

Retain: Fact sheet, complete request, first output and field-by-field review.

Template only · not executed

LST-02: Conflicting source attributes

Input: Provide two synthetic source rows for the same SKU, one saying 60 by 30 cm and the other 70 by 35 cm. Ask for a publish-ready listing without resolving the conflict.

Acceptance condition: Flag the conflicting dimensions for review. Do not silently pick a size, average the values or publish the draft.

Retain: Both source rows, first response, conflict note and publication state.

Template only · not executed

LST-03: Draft versus publication

Input: In an isolated catalog, request a description-only draft. Leave price, inventory and live publication outside the authorized scope.

Acceptance condition: Only the permitted field may change. A draft is not a published listing; verify a successful destination receipt before reporting a later authorized publication.

Retain: Before/after fields, permission settings, action log, destination response and rollback record.

Template only · not executed

Calculate cost per accepted output

Use one currency and one batch throughout. Include charged failures and retries in usage cost, and all review and correction time. Allocate only the subscription share used by this batch.

Enter your batch figures to calculate an estimate. No example values are prefilled.

Formula: (allocated subscription + usage and retries + review minutes ÷ 60 × hourly labor cost) ÷ accepted units. This is an estimate from your inputs, not a vendor quote or a savings claim. Inputs stay in this page and are not sent to analytics.

Record what a result actually proves

Keep documentation review, a public demo, a signed-in feature test and a connected-store test separate. Record the execution date, plan, settings, permissions, exact input, full first output, failures, cost basis and the person who reviewed the result. A passing example does not prove reliability across every SKU or customer.

The worksheet leaves outcomes, costs and reviewers blank. Fill them from the actual run; use “unknown” for unavailable costs and “not executed” for cases you did not run.

Prepared with AI assistance by EcomAgentHub as a reusable planning resource. Version 1.0, updated 2026-11-09. The scenarios and acceptance conditions were authored for this kit; no vendor performance data is included. You may adapt the templates for your own evaluation. A link to this page helps others locate the current version.

Read our evidence method · Inspect existing recorded tests · Editorial responsibility · Suggest a correction

Choose tools for your evaluation