Title: FAQ
Locale: en
URL: https://sensorswave.com/en/docs/experiments/faq/
Description: Frequently asked questions about A/B experiments
This article compiles frequently asked questions and answers about A/B experiments, covering experiment design, technical integration, data analysis, and the relationship with Feature Gates.

## Experiment Design Questions

### Q: How long should an experiment run?

**A**: At least meet the following conditions:

- **Minimum Duration**: At least 1 week (covering weekdays and weekends)
- **Sample size**: At least 1,000 users per group (5,000+ recommended)
- **Statistical significance**: p < 0.05

**Actual Duration depends on**:
- Traffic volume: High traffic 1–2 weeks, medium traffic 2–4 weeks, low traffic 4–8 weeks
- Conversion rate: Lower conversion rates require larger sample sizes
- MDE: Smaller Minimum Detectable Effect requires larger sample sizes

**Do not end early**:
- Even if the Test Group appears to lead, wait until the predetermined sample size is reached
- Short-term fluctuations may not represent long-term effects

### Q: How large should the sample size be?

**A**: Use the sample size formula:

```
Sample per group ≈ 16 × p × (1-p) / (MDE)²
```

**Examples**:

| Baseline Conversion Rate | MDE (Relative Lift) | Sample per Group | Total Sample |
|-----------|----------------|----------|---------|
| 20% | 10% (2% absolute lift) | 6,400 | 12,800 |
| 20% | 20% (4% absolute lift) | 1,600 | 3,200 |
| 5% | 20% (1% absolute lift) | 15,200 | 30,400 |

**Online calculators**:
- https://www.evanmiller.org/ab-testing/sample-size.html
- https://www.optimizely.com/sample-size-calculator/

### Q: How to choose a Primary Metric?

**A**: The Primary Metric should:

1. **Be directly related to business goals**:
   - E-commerce: Payment conversion rate, total revenue
   - Content platform: Content consumption duration, user retention
   - SaaS: Paid conversion rate, user activity

2. **Be accurately measurable**:
   - Has a clear Event definition
   - SDK or backend can accurately track it

3. **Be sensitive to changes**:
   - Can reflect the experiment's impact
   - Not overly affected by external factors

**Avoid vanity Metrics**:
- Page views (PV): Users may be casually clicking with no real engagement
- Registered user count: If users aren't active after registration, limited business value

### Q: How to allocate traffic for multi-variant experiments?

**A**: Multi-variant experiment Allocation strategies:

**Even Allocation** (recommended):
```
3-variant experiment:
- Control: 34%
- Test Group A: 33%
- Test Group B: 33%

4-variant experiment:
- Control: 25%
- Test Group A: 25%
- Test Group B: 25%
- Test Group C: 25%
```

**Control with majority** (conservative strategy):
```
3-variant experiment:
- Control: 50%
- Test Group A: 25%
- Test Group B: 25%
```

**Considerations**:
- Keep the number of Variants manageable (recommended ≤ 4)
- Each Variant should have at least 10% traffic
- Multi-variant experiments need longer Duration (sample size is distributed)

---

## Technical Integration Questions

### Q: How to ensure exposure logs are being reported properly?

**A**: Check the following steps:

**1. Check SDK configuration**:

JavaScript SDK:
```javascript
SensorsWave.init('YOUR_SOURCE_TOKEN', {
  enableAB: true  // Must be set to true
});
```

Go SDK:
```go
cfg := sensorswave.Config{
  AB: &sensorswave.ABConfig{
    ProjectSecret: "your-project-secret",  // Must be configured
    MetaEndpoint:  "https://your-ab-endpoint.com",
  },
}
```

**2. Check experiment status**:
- Experiment must be in "Running" status
- Draft status does not log exposures

**3. Check code integration**:
```javascript
// Correct: call getExperiment
const experiment = await sensorswave.getExperiment('experiment_key');

// SDK automatically logs $ABImpress Event
```

**4. Verify exposure data**:

Query in Segmentation:
```
Event: $ABImpress
Filter: experiment_key = 'your_experiment_key'
Time range: Last 1 hour
```

### Q: What if experiment variable retrieval fails?

**A**: Common causes and solutions for failed variable retrieval:

**Cause 1: Network error**

```javascript
try {
  const experiment = await sensorswave.getExperiment('experiment_key');
  const value = experiment.get('variable_name', defaultValue);
} catch (error) {
  console.error('Failed to get experiment:', error);
  // Use default value to ensure normal functionality
  const value = defaultValue;
}
```

**Cause 2: Experiment does not exist or is not released**

SDK returns default values:
```javascript
const experiment = await sensorswave.getExperiment('non_existent_experiment');
const value = experiment.get('variable_name', 'default');
// Returns 'default'
```

**Cause 3: User does not meet Targeting Rules**

If the user is not in the experiment's target audience, SDK returns default values.

**Best practices**:
- Always provide default values
- Add error handling
- Log errors for troubleshooting

### Q: Can experiment configuration be modified dynamically?

**A**: Yes, but not recommended.

**Impact of modifying configuration**:
- Breaks experiment stability
- Affects result reliability
- Makes data difficult to analyze

**If changes are necessary**:

1. Stop the current experiment
2. Create a new experiment (new experiment_key)
3. Apply new configuration
4. Re-collect data

**What can be modified** (without affecting the experiment):
- Display name
- Description
- Experiment Metric monitoring configuration

**What should not be modified**:
- Allocation ratios
- Variable values
- Targeting Rules

### Q: Will the same user participating in multiple experiments cause conflicts?

**A**: No, split across different experiments is independent.

**Independent split**:
```
User A in Experiment 1 (button color): Assigned to Control (blue)
User A in Experiment 2 (recommendation algorithm): Assigned to Test Group (deep learning)
```

**Considerations**:
- Ensure experiments don't interfere with each other (feature conflicts)
- Plan traffic reasonably to avoid shortages

---

## Data Analysis Questions

### Q: How to determine if experiment results are Significant?

**A**: Use statistical significance testing (P-Value):

**Criteria**:
- **p < 0.05**: Result is statistically Significant — unlikely to be produced by chance
- **p ≥ 0.05**: Result may be produced by chance — more data is needed or abandon Hypothesis

**Example**:

| Variant | Conversion Rate | Lift | P-Value | Conclusion |
|------|--------|---------|---------|------|
| Control | 24.0% | - | - | - |
| Test Group | 28.0% | +16.7% | 0.002 | Significant, Success |

**Calculate P-Value**:
- Use Sensors Wave's statistical testing feature
- Or use online tools: https://www.evanmiller.org/ab-testing/chi-squared.html

### Q: Why are Metrics declining for both Test Group and Control?

**A**: Possible reasons:

**1. External factors**:
- Holiday impact (e.g., overall traffic decline during Chinese New Year)
- Marketing campaign ended (conversion rate was high during the campaign and dropped after)
- Competitor promotions (users diverted)

**Solutions**:
- Extend experiment Duration to cover a full cycle
- Compare relative differences between Test Group and Control, not absolute values
- Exclude time periods affected by external factors

**2. Technical issues**:
- Experiment code has a bug affecting all users
- Server performance degradation, slower page loads

**Solutions**:
- Check technical Metrics (error rate, response time)
- Pause experiment, investigate issues

**3. Novelty effect wearing off**:
- Users are curious about new features early in the experiment, Metrics improve
- After some time, novelty fades, Metrics regress

**Solutions**:
- Extend experiment Duration to observe long-term effects
- Monitor long-term Metrics like retention rate

### Q: What if experiment results are unstable?

**A**: Possible causes and solutions for unstable results:

**Cause 1: Insufficient sample size**

```
Problem: Checking data daily, results fluctuate wildly
- Day 1: Test Group leads by 20%
- Day 2: Control leads by 15%
- Day 3: Test Group leads by 5%

Solution:
- Wait until predetermined sample size is reached
- Don't check data daily (easily misled by short-term fluctuations)
- Set minimum Duration (at least 1 week)
```

**Cause 2: External factor interference**

```
Problem: Large difference between weekdays and weekends
- Weekdays: Conversion rate 25%
- Weekends: Conversion rate 15%

Solution:
- Run for at least one full week (covering weekdays and weekends)
- Exclude impact of holidays and marketing campaigns
```

**Cause 3: Unbalanced Cohorts**

```
Problem: User characteristics differ significantly between Control and Test Group
- Control: 60% new users
- Test Group: 40% new users

Solution:
- Verify the split algorithm is correct
- With sufficient sample size, Cohorts will tend toward balance
```

---

## Relationship with Feature Gates

### Q: When to use Feature Gates vs A/B Experiments?

**A**: Choose based on purpose:

**Feature Gates**:
- **Purpose**: Control feature rollout, reduce risk
- **When to use**: After feature development, ready for release
- **Focus Metrics**: Technical Metrics (error rate, response time, crash rate)
- **Typical scenarios**: New feature gradual rollout, feature degradation, access control

**A/B Experiments**:
- **Purpose**: Validate feature effectiveness, optimize decisions
- **When to use**: After feature stabilization, validate effectiveness
- **Focus Metrics**: Business Metrics (conversion rate, retention rate, revenue)
- **Typical scenarios**: Solution comparison, optimization validation, impact assessment

**Recommended combined use**:
1. First use Feature Gates to verify technical stability (1–2 weeks)
2. Then use A/B experiments to validate business impact (2–4 weeks)
3. Full release of the winning solution

See [Feature Gates vs A/B Testing](../feature-gates/gates-vs-experiments.mdx) for details.

### Q: Can Feature Gates and A/B experiments be used together?

**A**: Yes, and it's recommended.

**Combined approach**:

```javascript
// First use Feature Gate to filter VIP users
const isVIP = await sensorswave.checkFeatureGate('vip_users_only');

if (isVIP) {
  // Run A/B experiment among VIP users
  const experiment = await sensorswave.getExperiment('vip_pricing_test');
  const price = experiment.get('price', 299);
  renderCheckoutPage(price);
} else {
  // Non-VIP users use standard pricing
  renderCheckoutPage(299);
}
```

**Advantages**:
- Feature Gates control experiment scope
- A/B experiments validate effectiveness within specific user groups
- They complement each other for more powerful capabilities

### Q: How to transition from Feature Gates to A/B Experiments?

**A**: Transition in phases:

**Phase 1: Feature Gate gradual rollout (1–2 weeks)**

```javascript
const isNewFeatureEnabled = await sensorswave.checkFeatureGate('new_feature_enabled');

if (isNewFeatureEnabled) {
  renderNewFeature();  // New feature
} else {
  renderOldFeature();  // Legacy feature
}
```

**Gradual rollout plan**:
- 1%: Internal testing
- 10%: Small-scale validation
- 50%: Large-scale testing
- 100%: Full rollout

**Validation Metrics**: Error rate, response time, crash rate

**Phase 2: A/B Experiment validation (2–4 weeks)**

After Feature Gate reaches 100%, create an A/B experiment:

```javascript
// Feature Gate stays at 100% enabled
const isNewFeatureEnabled = await sensorswave.checkFeatureGate('new_feature_enabled');

if (!isNewFeatureEnabled) {
  renderOldFeature();
  return;
}

// Run A/B experiment within the new feature
const experiment = await sensorswave.getExperiment('new_feature_optimization');
const variant = experiment.get('variant', 'v1');

if (variant === 'v2') {
  renderNewFeatureV2();  // Optimized version
} else {
  renderNewFeatureV1();  // Base version
}
```

**Validation Metrics**: Conversion rate, retention rate, revenue

**Phase 3: Clean up code**

After the experiment ends, apply the winning solution and clean up Feature Gate and experiment code.

---

## Related Documentation

- [A/B Experiment Overview](overview.mdx): Understand core A/B experiment capabilities
- [Quick Start](quick-start.mdx): Complete your first experiment in 20 minutes
- [Best Practices](best-practices.mdx): Master experiment design and execution best practices

---

**Last updated**: January 29, 2026