OpenAI and Ex-Shopify Product Lead Reveals How He Builds AI Products

Shopify began its generative AI work after leadership called for keeping merchants on the cutting edge. The team chose product descriptions because many of Shopify’s billions of products lacked them, then shipped a feature in the Winter ’23 Edition to learn from merchant use. Positive feedback led merchants to ask for AI capabilities in more places.

Shopify’s GSD process moves through proposal, prototype, build, and release. For product descriptions, the team tried different interface patterns, tested prompts and retrieval approaches, and created a quality evaluation before release. Stakeholders reviewed demo videos asynchronously every few days, while merchants gave feedback on mockups and generated samples.

Sidekick can answer Shopify questions, help merchants navigate settings, create products and discounts, and edit storefront themes. Because AI errors are hard to predict, the team planned for mistakes and outages from the start. It kept actions in client state so merchants could review them, with the principle that AI should augment rather than act without their control.

Jaffer describes a stack spanning the model, adapters, APIs, interface, and admin integration, with data supporting the system throughout. Sidekick does not train on personally identifiable data; merchant context is added at query time and stays within the session. Subject matter experts handwrite hundreds of tests per scenario, including multi-turn conversations, because automated evaluations can miss what a correct answer should be.

Jaffer says some hallucinations may move a user forward, while harmful commands or suggestions are unacceptable; merchants remain responsible for approving what goes out. Shopify did not set adoption targets, focusing instead on neutral or positive merchant outcomes. Jaffer also describes principles, open decision-making, separate crafter and manager career tracks, and broad internal access to AI tools.