n.Newfeed

Literature · 6 min read

What Stylometry Can and Cannot Establish About Literary Authorship

A practical account of stylometric attribution, validation, corpus bias, and the limits of a nearest-author result.

What Stylometry Can and Cannot Establish About Literary Authorship

Turning writing style into measurements

Stylometry measures recurring features of language and uses them to compare texts. In literary authorship research, the aim may be to decide which of several candidate authors most resembles an anonymous work, to test whether a book contains stylistic segments, or to map changes across a writer’s career. The method replaces an impression such as “this sounds like Austen” with a reproducible set of counts and comparisons.

Its most useful features are often unremarkable. Frequencies of common function words—and, of, to, by—can be more informative than rare thematic vocabulary because writers choose them repeatedly and only partly consciously. Researchers also measure character sequences, punctuation, sentence length, word length, grammatical patterns, and vocabulary distribution. No feature is an authorial fingerprint in the forensic sense. Attribution comes from a pattern across features and from how that pattern behaves in a defined comparison set.

This distinction limits the conclusion. A model can find that one candidate is the nearest among those supplied. It cannot thereby show that the true author was included, that the text was not collaborative, or that editorial intervention left the signal untouched. Stylometry produces conditional evidence, strongest when the corpus and alternative explanations are carefully controlled.

How a closed-set attribution is built

A common workflow begins with texts of known authorship. Researchers normalize spelling and markup, decide whether to remove dialogue or paratext, divide long works into comparable samples, and count selected features. Those decisions are analytical, not clerical. Modernized spelling may erase historical habits; retaining chapter headings may capture an editor; cutting texts into short blocks creates more samples without creating more independent language.

John Burrows’s Delta, introduced in 2002, offered a widely adopted procedure. For each frequent word, a text’s frequency is converted to a z-score relative to the corpus mean and variation. The distance between a test text and each author profile is then summarized across words. The closest profile is a candidate attribution. Later variants change the distance measure, feature weighting, or normalization, while machine-learning classifiers learn boundaries among authors from labeled examples.

Evaluation must keep test material separate from training material. If passages from the same novel appear on both sides, the model may learn that novel’s topic, characters, or editorial quirks rather than a transferable authorial pattern. Cross-validation estimates performance by repeatedly withholding known samples and asking the model to classify them. A confusion matrix shows which authors are mistaken for which others.

Accuracy is meaningful only for the task constructed. Distinguishing two contemporaries writing in the same genre is harder and more relevant than distinguishing a poet’s letters from a scientist’s textbook. A model evaluated on 20 known candidates does not inherit that accuracy when thousands of unknown writers are possible. This is the difference between closed-set attribution, where the author is assumed to be among the candidates, and open-set verification, where “none of these” must remain possible.

Cases where stylometry is informative

The classic demonstration predates modern machine learning. Frederick Mosteller and David Wallace analyzed disputed essays in The Federalist using function-word probabilities. The leading candidates, Alexander Hamilton and James Madison, had both written undisputed papers in the collection. Topic, period, and genre were unusually well matched. Their Bayesian analyses strongly favored Madison for the disputed essays. The result was persuasive not because a computer replaced historical scholarship, but because the candidate set and comparison material were unusually suitable.

Stylometry can also challenge a simple authorship story. Rolling analyses move a window through a text and look for changes in similarity, which may identify candidate boundaries for collaboration or revision. These patterns are hypotheses to compare with manuscript evidence, publication history, and close reading. A stylistic shift may reflect a different author, but it may also mark a new narrator, genre, source text, or phase of composition.

Recent work on Frankenstein illustrates both value and care. Researchers have compared the 1818 text, Mary Shelley’s known writing, and Percy Bysshe Shelley’s material with Burrows’s Delta and support-vector machines. Such work can test claims about relative stylistic affinity at passage level. It does not convert Percy’s documented editorial contributions into authorship of the novel, nor can a classifier reconstruct every exchange in a collaborative household.

Results are most informative when multiple reasonable feature sets and methods converge, known texts classify correctly under withheld testing, and the disputed text is not an outlier to every candidate. Reporting only the best-performing configuration after trying many alternatives exaggerates certainty. Robust studies disclose preprocessing, parameters, corpus contents, and sensitivity analyses so others can reproduce the result.

Corpus, genre, and model limitations

Text length is a basic constraint. Short samples contain fewer repeated choices, so estimates of rare features become unstable. Research on literary corpora has found that performance generally declines as test samples and training material shrink. There is no universal minimum word count: the threshold depends on language, number of candidates, genre similarity, period, features, and method. A figure established in one experiment should not become a rule for all texts.

Topic can masquerade as style. If one author’s training texts concern ships and another’s concern gardens, content words may let a classifier identify subject matter. Restricting analysis to frequent function words reduces this risk but does not remove it; function-word use can vary by genre, dialogue, or syntax. Matching comparison texts by date and genre is often more valuable than adding distant material simply to increase volume.

Textual transmission adds another layer. Spelling may belong to a compositor. Punctuation may be imposed by an editor. Translation carries patterns from both source author and translator. Optical character recognition introduces errors that may cluster by edition or typeface. If each candidate is represented by a different publisher’s digital edition, a model can classify production pipelines instead of writers.

Authorship itself may not be singular. Plays, serialized fiction, journalism, and correspondence can involve collaborators, secretaries, editors, and source reuse. Training labels that assign an entire known work to one name can conceal the same complexity the analysis is supposed to detect. A model forced to select one candidate will still produce an answer.

Statistical dependence is another risk. Multiple chunks from one book share plot, characters, and compositional history. Treating them as independent evidence makes confidence intervals too narrow. Researchers can hold out entire works rather than random chunks and can test whether results survive changes in chunk size and vocabulary selection.

How to combine computation with literary evidence

A responsible attribution states the question conditionally: among these candidates, using these texts and features, which attribution is best supported, and how often does the method recover known authors in comparable tests? It reports uncertainty and negative diagnostics. If the disputed text lies far from every profile, “unrepresented author or transformed text” may be the most informative result.

External evidence answers questions stylometry cannot. Paper, watermarks, handwriting, and ink can date a manuscript. Letters and contracts can establish access or collaboration. Provenance can reveal when an attribution emerged. Philology can trace sources and revisions. Close reading can explain why a computationally unusual passage differs. Convergence among independent kinds of evidence is stronger than repeating one classifier with several settings.

What is established is that relative frequencies of common linguistic features can carry author-associated signals and that well-constructed models can recover known authors at useful rates in bounded corpora. What is contested is how far accuracy transfers from curated tests to historical disputes with short, edited, translated, or collaborative texts.

What remains unknown in many cases is whether the comparison corpus contains the right alternatives and preserves the relevant authorial signal. Stylometry can rank hypotheses, locate anomalies, and make intuition auditable. It cannot certify a name without assumptions. The sound conclusion is therefore comparative: the measured text is more consistent with one model than another, within a documented field of candidates.

Sources

4 sources, none of them with a link, so treat them as reading, not proof.

  1. Frederick Mosteller and David L. Wallace, Inference and Disputed Authorship: The Federalist (Addison-Wesley, 1964; later reprint editions).
  2. John Burrows, “‘Delta’: A Measure of Stylistic Difference and a Guide to Likely Authorship,” Literary and Linguistic Computing 17, no. 3 (2002): 267–287.
  3. Kim Luyckx and Walter Daelemans, “Authorship Attribution and Verification with Many Authors and Limited Data,” Proceedings of COLING 2008: 513–520.
  4. Michael Dalvean, “Did Mary Shelley Write Frankenstein? A Stylometric Analysis,” Digital Scholarship in the Humanities 38, no. 2 (2023): 750–765.

Want the reader to remember this spot? Sign in — it is free.

Next How Zora Neale Hurston's Barracoon Reached Print in 2018

The app

Keep going, or take it with you.

Free accounts read twenty pieces a day. Pro is US$1.99 a month or US$11.99 a year for unlimited reading.