Quiz Funnel A/B Test — In-Product Behavior

match_me  vs  creator_funnel_v2  ·  which acquisition funnel produces better in-product behavior
Cohort window: May 28 – Jun 3, 2026 · both arms identified by the quizVariant identify trait (like-for-like)
Source: BigQuery stxt-490006 · dbt marts d2c_prod_dbt_stxt_analytics
Data as of Jun 4, 2026
One-line read: across every in-product step we can measure like-for-like, creator_funnel_v2 outperforms match_me — from 1.2× (starting a chat) to 3.5× (reaching checkout). Both funnels have 0 purchases so far (cohort is ≤7 days old; paid lands a median ~35 days after signup). On click→signup (Everflow paid-click proxy) the funnels are similar — match_me ~1.5% (3,202 clicks → 47) vs creator_funnel_v2 ~1.6% (3,377 → 54); creator_funnel_v2’s edge shows up after signup (see Part 1).
Part 1 · At a glance

How the two funnels compare, step by step

Each row scales the longer bar to full width, so the gap you see is the gap in the data. The chip on the right is the multiple (e.g. “2.6× higher” = creator_funnel_v2’s rate is 2.6 times match_me’s). Rates are % of each funnel’s signups (match_me n=47, creator_funnel_v2 n=54).

match_me (n=47) creator_funnel_v2 (n=54)
Click → signup · Everflow paid-click proxy≈1.1× · ~tied
match_me
1.5%
creator_funnel_v2
1.6%
Started a chat1.2× higher
match_me
47%
creator_funnel_v2
57%
Received an in-chat photo2.2× higher
match_me
28%
creator_funnel_v2
61%
Hit the demo paywall1.7× higher
match_me
9%
creator_funnel_v2
15%
Clicked “Subscribe”2.6× higher
match_me
6%
creator_funnel_v2
17%
Reached checkout3.5× higher
match_me
2%
creator_funnel_v2
7%
Came back on a later day2.6× higher
match_me
2%
creator_funnel_v2
6%
Paid (subscription or tokens)tied · 0
match_me
0%
creator_funnel_v2
0%
Visitor → signup — via Everflow click proxy (covers both funnels). On-site page views can’t do this comparison because the V2 lander page view (landerHomeV2) isn’t firing (1 recorded visitor vs 54 signups — GTM-1988). Since all paid traffic routes through Everflow, we use Everflow clicks per Offer URL as the visitor proxy: match_me (Main Page #413) = 3,202 clicks → 47 signups ≈ 1.5%; creator_funnel_v2 (quiz_v2 #422) = 3,377 clicks → 54 signups ≈ 1.6%. Near-identical click volume and click→signup rate (~1.1×); creator_funnel_v2’s advantage emerges after signup (the steps above). Caveat: Everflow clicks are paid traffic only; for reference, the on-site match_me lander (landerHome) shows 1,538 page views → 47 (3.1%).
Part 2 · The details

AWhat was tested

Two acquisition funnels run on stxtai.com: a visitor goes through a quiz and is sent to justsext.com to create an account. match_me is the original path; creator_funnel_v2 is the new /quiz_v2 path. Both are new and were tested concurrently.

Cohort: users who created an account May 28–Jun 3, keyed on user_id, with the funnel taken from the quizVariant identify trait (dim_user_quiz_preferences) — the one signal recorded identically for both funnels. Behavior is read from fct_user_journey_events.

BWhere the data comes from & how it was validated

The app emits events through Segment (analytics.js). Segment writes to BigQuery (d2c_prod); dbt builds the marts behind the team dashboards. The same stream also feeds Mixpanel.

Validation: reproduced the team funnel dashboard for Jun 3 (/quiz_v2=18, /quiz/default=11, unknown=3); cohort sizes match the day-level signup mart (47 / 54–55) and Mixpanel (42 / 53) within ~5–10%; paid = 0 for both in Mixpanel and the warehouse.

CSignup volume & engagement

Metricmatch_mecreator_funnel_v2Gap
Signups in window (count)47541.15×
Click → signup (Everflow paid-click proxy)~1.5% (3,202→47)~1.6% (3,377→54)≈1.1×
Started a chat47% (22)57% (31)1.2×
Received in-chat photo28% (13)61% (33)2.2×
Chat-start events / signup2.53.51.4×

What this means: the two funnels brought a similar number of signups (47 vs 54 — driven by traffic split, not conversion). Once inside, creator_funnel_v2 users engage more: a higher share start a chat, more than twice the share receive an in-chat photo from the AI, and they generate more chat activity per user. (Signup counts are volume, not a conversion rate; chat-start events repeat on reconnect; in-chat photo is a server delivery signal.)

DMonetization & retention

Stepmatch_mecreator_funnel_v2Gap
Hit demo paywall9% (4)15% (8)1.7×
Clicked “Subscribe”6% (3)17% (9)2.6×
Reached checkout2% (1)7% (4)3.5×
Paid subscription00
Token purchase00
Returned on a later day2% (1)6% (3)2.6×

What this means: creator_funnel_v2 users move further down the paid funnel at every step (paywall → Subscribe click → checkout), and the relative gap widens deeper in the funnel (1.7× → 2.6× → 3.5×). Neither funnel has produced a paid conversion yet; in this data paid lands a median ~35 days after signup, and this cohort is ≤7 days old. Return-visits favor creator_funnel_v2 (6% vs 2%) but on tiny counts (3 vs 1).

EData notes & known limitations

1. Small samples. 47 and 54 signups; paywall/checkout/retention are single-digit counts. Read as directional, not statistically conclusive.

2. Both arms use the identify trait. The funnels are instrumented differently — match_me stamps its variant onto events; creator_funnel_v2 writes it only to the user profile. To compare like-for-like, both are keyed on the quizVariant identify trait (the only signal recorded the same way for both). Tracked in GTM-1985.

3. Visitor→signup uses an Everflow click proxy. On-site page views can’t compare the funnels because the V2 lander page view (landerHomeV2) isn’t firing (1 visitor vs 54 signups — GTM-1988, fix in progress). Proxy via Everflow clicks per Offer URL (paid traffic): match_me (#413) 3,202 clicks → 47 signups ≈ 1.5%; creator_funnel_v2 (#422) 3,377 → 54 ≈ 1.6% — near-identical. For reference, the on-site match_me lander (landerHome) shows 1,538 page views → 47 (3.1%); Everflow conversions per Offer URL are also in BigQuery (everflow.conversion_report).

4. Purchases need more time. Median signup→paid is ~35 days; a ≤7-day cohort can’t show paid yet. 0 paid is expected at this age.

5. Retention is right-censored. Most signups landed in the final 2 days of the window, and the journey-event log excludes some in-app actions (e.g. message_sent, image_generated). Treat retention as early/partial.

Appendix — reproducible SQL (re-run with a new window to refresh)
-- Cohort + behavior, both arms keyed on the quizVariant identify trait
WITH acct AS (
  SELECT user_id, MIN(event_date) AS signup_date
  FROM d2c_prod_dbt_stxt_analytics.fct_user_journey_events
  WHERE event_name='account_created' AND event_date BETWEEN '2026-05-28' AND '2026-06-03' AND user_id IS NOT NULL
  GROUP BY user_id
),
coh AS (
  SELECT a.user_id, a.signup_date, d.quiz_variant
  FROM acct a JOIN d2c_prod_dbt_stxt_analytics.dim_user_quiz_preferences d USING (user_id)
  WHERE d.quiz_variant IN ('match_me','creator_funnel_v2')
),
ev AS (
  SELECT coh.quiz_variant, coh.user_id, coh.signup_date, e.event_name, e.event_date
  FROM coh JOIN d2c_prod_dbt_stxt_analytics.fct_user_journey_events e ON e.user_id = coh.user_id
)
SELECT quiz_variant,
  COUNT(DISTINCT user_id) AS cohort,
  COUNT(DISTINCT IF(event_name='chat_initiated', user_id, NULL)) AS chat,
  COUNTIF(event_name='chat_initiated') AS chat_events,
  COUNT(DISTINCT IF(event_name='escalation_photo_sent', user_id, NULL)) AS photo,
  COUNT(DISTINCT IF(event_name='onboarding_paywall_hit', user_id, NULL)) AS paywall_hit,
  COUNT(DISTINCT IF(event_name='onboarding_paywall_click_subscribe', user_id, NULL)) AS subscribe_click,
  COUNT(DISTINCT IF(event_name='checkout_form_rendered', user_id, NULL)) AS checkout,
  COUNT(DISTINCT IF(event_name='subscription_paid', user_id, NULL)) AS paid,
  COUNT(DISTINCT IF(event_date>signup_date, user_id, NULL)) AS returned_later
FROM ev GROUP BY quiz_variant ORDER BY quiz_variant;

-- Visitor -> signup by funnel, from lander page views (page_name + quiz_variant)
SELECT page_name, quiz_variant, COUNT(DISTINCT anonymous_id) AS visitors
FROM d2c_prod.page_view
WHERE DATE(timestamp) BETWEEN '2026-05-28' AND '2026-06-03'
GROUP BY 1,2 ORDER BY visitors DESC;
-- landerHome / match_me      = 1538 visitors -> 47 signups (~3.1%)
-- landerHomeV2 / creator_funnel_v2 = 1 visitor (page view not firing) despite 54 signups -> unmeasurable (GTM-1988)

-- Visitor proxy via Everflow clicks (paid traffic) — covers both funnels
-- Everflow Reporting API: POST https://api.eflow.team/v1/networks/reporting/entity
--   {"from":"2026-05-28","to":"2026-06-04","columns":[{"column":"offer_url"}],"query":{"filters":[]}}
-- total_click by Offer URL: #413 Main Page (match_me) = 3,202 -> 47 signups (~1.5%)
--                           #422 quiz_v2 (creator_funnel_v2) = 3,377 -> 54 signups (~1.6%)

Prepared by GTM analytics · single source: BigQuery stxt-490006 via dbt marts. Renders in light & dark. To refresh: re-run the appendix SQL with a new date window and update the figures + the “Data as of” badge.