Strategy Game Market Study

The question

A team planning a turn-based strategy game wants to know three things about comparable games: what players praise, what makes them leave a negative review, and whether that changes with how a game resolves turns. Some games let one side move and then the other. Some alternate unit by unit. Some have both sides plan at once and resolve together. Steam reviews are the largest free source of answers, so I built a study to measure which themes the review text supports and which it can't settle.

This is the second phase of my Steam review pipeline. I directed the study and set its rules before any results existed. AI coding agents built much of the pipeline under a review protocol I designed.

Source code and dashboard

The source code is on GitHub, with both studies committed as SQLite seeds. The live dashboard opens on this study and may need a moment to wake up.

---

The data

• 29 games in three groups. 22 turn-based tactics titles split by turn structure (7 side-by-side, 7 alternating, 8 simultaneous), 4 historical grand strategy games, and 3 large anchors: Civilization VI, Stellaris, and Total War: WARHAMMER III.
• 209,223 English reviews pulled from Steam on September 26, 2026. 22 games were pulled in full. The seven largest hit a 20,000-review cap and cover a recent window.
• A plain random sample. Up to 500 reviews per game gave 12,315 reviews, and 9,622 survived cleaning and a language filter. I didn't stratify by the recommend flag, so every game keeps its real recommend rate. Across the 22 fully pulled games, the sample sits 0.3 points below Steam's own positive share on average.

What separates negative reviews from positive ones

Crashes, by the widest margin. About one not-recommended review in sixteen lands in the crash topic, against one recommended review in two hundred. Across the 18 eligible turn-based titles, that's a complaint lift of 12.8 (95% interval 7.2 to 29.3).

Pacing and the interface. Grind, tedium, late-game drag, and micromanagement come up two to three times as often in negative reviews as in positive ones. So does the interface. Both patterns hold in all three turn-structure groups.

The turn system itself is rarely discussed. Waiting through enemy turns, initiative, and combat previews each appear in under 3% of reviews per group. That's too rare in this sample to tell the groups apart.

Players write more critically than they vote. In every group, the sentiment model scores 4.8 to 9.9 points fewer reviews positive than the share that recommend the game.

The study ran 459 comparisons and produced 24 findings, all but one on the complaint side. The intervals are per comparison and uncorrected, so I treat every finding as exploratory: a strong lead for design and marketing work. Genre, franchise, and price travel with turn structure, so none of the findings is a claim about the turn rule itself.

---

How a theme becomes a finding

I wrote the rule down before running the statistics. A theme counts as a finding only when:

1. its group has at least three eligible games,
2. the complaint lift interval sits entirely above 1, or the praise interval entirely below 1,
3. the result holds when any single game is dropped,
4. the theme has at least five mentions on each side, and
5. for phrase-lexicon themes, the lexicon passed its precision audit.

Group averages weight each game equally, with a two-stage cluster bootstrap that resamples games and then reviews. That keeps Civilization VI from outvoting a niche title with a few hundred reviews.

The rule changed twice after results came in, and the README says so. The first run read 13 empty intervals as praise findings, so I added the five-mention floor. Later, an independent review found the stability check ranked topics and lexicon themes together instead of separately, as the written plan required. Fixing the code to match the plan moved the finding count from 14 to 24.

Replacing VADER with RoBERTa

VADER struggles with long, critical strategy reviews. A player praises the game for four paragraphs, then explains why they can't recommend it. VADER reads the praise.

I wrote the adoption rule before reading the test half of the data. VADER got its threshold tuned in its favor on the development half. The better transformer on the development half went forward. The test half was read once, and the transformer would be adopted only if the bootstrap interval of its gain over VADER sat above zero at both VADER thresholds.

Scorer	Balanced accuracy	Not-recommended reviews scored positive
VADER, default threshold	70.3%	42.4%
VADER, tuned threshold	71.3%	32.2%
DistilBERT SST-2	84.8%	9.5%
RoBERTa-large	91.2%	8.1%

On 4,783 held-out reviews, RoBERTa-large gained 19.9 points of balanced accuracy over tuned VADER (paired interval 18.1 to 21.6), so it became the study's sentiment measure. On the April reviews, which nothing was tuned on, it scored 87.8%. I report balanced accuracy because calling every review "recommended" would score 80.9% plain accuracy on this sample.

The findings didn't move when the scorer changed. The finding rule uses only the recommend flag, so sentiment can describe a theme but can't create one.

Sentence sentiment took three attempts

I also wanted sentiment per sentence, so the dashboard could show how players feel about each design theme. The gate was a three-class macro-F1 of 0.70, fixed before any label existed. Each attempt was written down in advance and read a fresh labeled test set exactly once.

1. 0.52. The binary RoBERTa-large model never answers "neither," so every neutral sentence counted as an error.
2. 0.69. A three-class RoBERTa-base model, on 200 fresh sentences. One point short.
3. 0.707. The same model with a neutral-probability cut tuned on the 300 earlier labels, on a third fresh set of 200. It passed.

The gate never moved. The README reports all three attempts and tells readers to weigh three tries against one gate. No finding depends on sentence sentiment. The dashboard shows it as description.

Auditing the phrase lexicon

Topics catch broad themes, but a design team also asks about named mechanics. I built a phrase lexicon for seven design themes and four controls, then had an independent model labeler check 433 tagged sentences against a written guide. Seven of eleven themes passed an 80% precision floor. The four that failed make no claims.

The worst was first-strike advantage at 27.5%. Most of its hits were "alpha strike" in the mech-combat sense: firing every weapon at once, not acting first. I froze the lexicon after the audit. Fixing it at that point would have tuned it to the reported numbers. A second version will get its own audit on fresh sentences.

Running any Steam game set

The tool now runs a study on any group of Steam games:

• a study command that writes a configuration file from Steam store lookups and refuses name or path collisions,
• an idempotent pipeline runner that reports each stage as ran, no-op, skipped, or not applicable,
• a cohort comparison view in the dashboard, and
• frozen shipped studies that no command can overwrite without an explicit unfreeze flag.

The April baseline stays reproducible from its own tagged commit. The project has 195 tests, and the committed databases total 22.2 MiB, under a 25 MiB budget.

Tools: Python, SQLite, spaCy, lingua, NLTK (VADER), Hugging Face Transformers, PyTorch, BERTopic, sentence-transformers, NumPy, SciPy, Streamlit, Plotly, GitHub Actions, Playwright, pytest