Strategy Game Market Study

The question

A team planning a turn-based strategy game wants to know three things about comparable games: what players praise, what makes them leave a negative review, and whether that changes with how a game resolves turns. Some games let one side move and then the other. Some alternate unit by unit. Some have both sides plan at once and resolve together. Steam reviews are the largest free source of answers, so I built a study to measure which themes the review text supports and which it can't settle.

This is the second phase of my Steam review pipeline. I designed the study, set its rules before any results existed, and directed the AI coding agents that built the pipeline.

Source code and dashboard

The source code is on GitHub, with both studies committed as SQLite seeds. The live dashboard opens on this study and may need a moment to wake up.

---

The data

• 29 games in three groups. 22 turn-based tactics titles split by turn structure (7 side-by-side, 7 alternating, 8 simultaneous), 4 historical grand strategy games, and 3 large anchors: Civilization VI, Stellaris, and Total War: WARHAMMER III.
• 209,223 English reviews pulled from Steam on September 26, 2026. 22 games were pulled in full. The seven largest hit a 20,000-review cap and cover a recent window.
• A plain random sample. Up to 500 reviews per game gave 12,315 reviews, and 9,622 survived cleaning and a language filter. I didn't stratify by the recommend flag, so every game keeps its real recommend rate. Across the 22 fully pulled games, the sample sits 0.3 points below Steam's own positive share on average.

What separates negative reviews from positive ones

Crashes, by the widest margin. About one not-recommended review in sixteen lands in the crash topic, against one recommended review in two hundred. Across the 18 eligible turn-based titles, that's a complaint lift of 12.8 (95% interval 7.2 to 29.3).

Pacing and the interface. Grind, tedium, late-game drag, and micromanagement come up two to three times as often in negative reviews as in positive ones. So does the interface. Both patterns hold in all three turn-structure groups.

The turn system itself is rarely discussed. Waiting through enemy turns, initiative, and combat previews each appear in under 3% of reviews per group. That's too rare in this sample to tell the groups apart.

Players write more critically than they vote. In every group, the sentiment model scores 4.8 to 9.9 points fewer reviews positive than the share that recommend the game.

The study ran 459 comparisons and produced 24 findings, all but one on the complaint side. With that many comparisons, some findings could be chance, so I tested each one again on reviews the study had never used. 21 of the 24 held up (see Confirming the findings on fresh reviews). Genre, franchise, and price travel with turn structure, so none of the findings is a claim about the turn rule itself.

---

How a theme becomes a finding

I wrote the rule down before running the statistics. A theme counts as a finding only when:

1. its group has at least three eligible games,
2. the complaint lift interval sits entirely above 1, or the praise interval entirely below 1,
3. the result holds when any single game is dropped,
4. the theme has at least five mentions on each side, and
5. for phrase-lexicon themes, the lexicon passed its precision audit.

Group averages weight each game equally, with a two-stage cluster bootstrap that resamples games and then reviews. That keeps Civilization VI from outvoting a niche title with a few hundred reviews.

The rule changed twice after results came in, and the README says so. The first run read 13 empty intervals as praise findings, so I added the five-mention floor. Later, an independent review found the stability check ranked topics and lexicon themes together instead of separately, as the written plan required. Fixing the code to match the plan moved the finding count from 14 to 24.

Confirming the findings on fresh reviews

The 24 findings were exploratory when found. To confirm them, I committed the test plan, the code that runs it, and a fingerprint of the exact review sample to the public repository and tagged it replication-protocol-v1 before any result existed. Anyone can check that the rules came before the answer. The read refuses to run unless the code is unchanged at that tag and no result exists yet, so it ran once.

• Fresh sample. 36,774 reviews that were in neither the original sample nor any labeled audit set, drawn from the same games and time windows without looking at the recommend flag. 28,200 passed the text filters.
• Nothing retuned. The same cleaning rules, phrase lexicon, saved topic model, and bootstrap.
• Pass rule and power, set first. A finding replicates when its new lift points the same way and its 95% interval rules out "no difference" on that side. The power analysis expected 21.8 replications if the original effects were real and 14.8 if every game's effect were half as strong, and it set the pass line at 12.

Result: 21 of 24 replicated, and 7 of 8 theme groups, in line with the power analysis. All 24 pointed the same way as before. Crash complaints replicated in all three game groupings, at about 15.9 times the rate in negative reviews as in positive ones (9.2 to 29.8). All five bugs-and-performance findings and all five interface findings replicated. The three misses still pointed the same way, but their intervals now include "no difference": a praise topic in the matched alternating-turn group, pacing in side-by-side turn games, and price in simultaneous-turn games, which the power analysis had ranked among the weakest. There were no deviations from the protocol.

The replication used the same games and windows, so it shows the findings hold on other reviews of these games, not on other games or later periods.

Replacing VADER with RoBERTa

VADER struggles with long, critical strategy reviews. A player praises the game for four paragraphs, then explains why they can't recommend it. VADER reads the praise.

I wrote the adoption rule before reading the test half of the data. VADER got its threshold tuned in its favor on the development half. The better transformer on the development half went forward. The test half was read once, and the transformer would be adopted only if the bootstrap interval of its gain over VADER sat above zero at both VADER thresholds.

Scorer	Balanced accuracy	Not-recommended reviews scored positive
VADER, default threshold	70.3%	42.4%
VADER, tuned threshold	71.3%	32.2%
DistilBERT SST-2	84.8%	9.5%
RoBERTa-large	91.2%	8.1%

On 4,783 held-out reviews, RoBERTa-large gained 19.9 points of balanced accuracy over tuned VADER (paired interval 18.1 to 21.6), so it became the study's sentiment measure. On the April reviews, which nothing was tuned on, it scored 87.8%. I report balanced accuracy because calling every review "recommended" would score 80.9% plain accuracy on this sample.

The findings didn't move when the scorer changed. The finding rule uses only the recommend flag, so sentiment can describe a theme but can't create one.

Sentence sentiment took three attempts

I also wanted sentiment per sentence, so the dashboard could show how players feel about each design theme. The gate was a three-class macro-F1 of 0.70, fixed before any label existed. Each attempt was written down in advance and read a fresh labeled test set exactly once.

1. 0.52. The binary RoBERTa-large model never answers "neither," so every neutral sentence counted as an error.
2. 0.69. A three-class RoBERTa-base model, on 200 fresh sentences. One point short.
3. 0.707. The same model with a neutral-probability cut tuned on the 300 earlier labels, on a third fresh set of 200. It passed.

The gate never moved. The README reports all three attempts and tells readers to weigh three tries against one gate. No finding depends on sentence sentiment. The dashboard shows it as description.

Auditing the phrase lexicon

Topics catch broad themes, but a design team also asks about named mechanics. I built a phrase lexicon for seven design themes and four controls, then had an independent model labeler check 433 tagged sentences against a written guide. Seven of eleven themes passed an 80% precision floor. The four that failed make no claims.

The worst was first-strike advantage at 27.5%. Most of its hits were "alpha strike" in the mech-combat sense: firing every weapon at once, not acting first. I froze the lexicon after the audit. Fixing it at that point would have tuned it to the reported numbers. A second version will get its own audit on fresh sentences.

A second study: real-time-with-pause versus turn-based combat

The same pipeline answered a new question in October 2026. A team choosing a combat model for a party RPG or a strategy game wants to know what players complain about when combat runs live and the player pauses to give orders (real-time-with-pause), compared with turn-based combat.

• 26 games in five groups: real-time-with-pause and turn-based CRPGs, pausable real-time grand strategy, turn-based strategy, and six games that offer both modes. Each game's combat type is backed by its store page or a developer post. 239,850 English reviews loaded.
• Split before exploring. A hash of each review ID put it in a development half or a test half. All exploration, lexicon building, and topic modeling (196 topics) used the development half only, and a test confirms no test-half text can reach it.
• Lexicon checked by a fresh labeler. A model labeler that saw only the labeling guide and the sentences judged 40 test-half sentences per aspect. Three of nine aspects passed the 80% precision floor on the first try. One allowed revision, built from development-half sentences only and checked on 240 new sentences, passed the other six.
• Pre-registered read. The finding rule turned 1,845 comparisons into 66 candidates. The protocol, the word lists, and the exact list of 119,485 test-half reviews were tagged rtwp-turn-protocol-v1 before the read. The power analysis expected 60.0 replications if the effects were real and set the pass line at 43.

Result: 59 of 66 replicated, in line with the power analysis, and every candidate pointed the same way. Slow or too many fights showed up about six times as often in negative CRPG reviews as in positive ones (6.35, 3.09 to 14.0). Pausing and micromanagement replicated in 5 of 6 groups at about twice the rate. Bugs and performance, the interface, crashes, and multiplayer desync replicated everywhere they were tested, under both combat types, and bugs and performance is named in about a quarter of negative CRPG reviews. Turn-based mode as a praise theme didn't replicate. Five of the seven misses were combat themes, including the two the power analysis had ranked weakest.

In games that offer both modes, reviews that mention only the turn-based mode recommend the game more often than reviews that mention neither: 81.1% against 73.3% in Pathfinder: Kingmaker, and 92.4% against 85.8% in Pillars of Eternity II: Deadfire. That comparison is descriptive.

Era, studio, and genre travel with combat type here. Three of the real-time-with-pause CRPGs are enhanced editions from one studio, and every real-time-with-pause strategy game comes from one publisher. So the study says what players of each kind of game complain about. It doesn't rank one combat model over the other.

Running any Steam game set

The tool now runs a study on any group of Steam games:

• a study command that writes a configuration file from Steam store lookups and refuses name or path collisions,
• an idempotent pipeline runner that reports each stage as ran, no-op, skipped, or not applicable,
• a cohort comparison view in the dashboard, and
• frozen shipped studies that no command can overwrite without an explicit unfreeze flag.

The April baseline stays reproducible from its own tagged commit. The project has 332 tests, and the committed databases total 22.2 MiB, under a 25 MiB budget.

Tools: Python, SQLite, spaCy, lingua, NLTK (VADER), Hugging Face Transformers, PyTorch, BERTopic, sentence-transformers, NumPy, SciPy, Streamlit, Plotly, GitHub Actions, Playwright, pytest