<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0">
<channel>
<title>Coder Legion - Recent posts tagged digital-humanities</title>
<link>https://coderlegion.com/tag/digital-humanities</link>
<description>Powered by Question2Answer</description>
<item>
<title>Digital Humanities in Russia: From Bely to Big Data – A Complete History</title>
<link>https://coderlegion.com/25490/digital-humanities-in-russia-from-bely-to-big-data-a-complete-history</link>
<description>&lt;p&gt;If you've ever wondered how a country with a deep literary and historical tradition embraced the digital revolution – and what that journey reveals about the broader relationship between technology and culture – this episode is for you. In this installment of &lt;em&gt;In-Depth with Academia&lt;/em&gt;, we take a sweeping tour through the history, achievements, and challenges of digital humanities in Russia. From early 20th-century experiments with quantitative analysis by pioneers like Andrei Bely and Nikolai Morozov, to the computer-driven projects of the 1960s and 70s, to today's sophisticated digital archives and stylometric tools, the story is one of continuity, adaptation, and occasional friction. The video traces how Russian scholars have long blended mathematics with literary and historical inquiry – long before &quot;digital humanities&quot; became a global buzzword. Researchers like B.I. Yarkho and M.L. Gasparov continued this tradition, applying statistical rigor to poetry and style, setting the stage for the computational turn. With the arrival of computers, projects focused on machine-readable databases, automated text analysis, and new statistical tools, enabling questions that were previously impossible due to the sheer volume of data. The institutionalization of the field came through the creation of research centers, commissions, and academic journals, as well as landmark resources like the National Corpus of the Russian Language and the poetic subcorpora. Groundbreaking projects such as Tolstoy.Digital, Prozhito, Textograf, and Chekhov Digital have transformed how Russian literature and history are studied, both domestically and internationally. Stylometry became a major focus, with Russian scholars adopting and adapting methods like Delta to participate in global conversations about authorship attribution, poetic meter, and linguistic patterns. Historians, too, embraced digital tools – GIS, network analysis, 3D modeling – to uncover patterns in social, economic, and political history that were previously hidden. The video also addresses the challenges: ongoing debates about methodology, institutional fragility, political and economic pressures, data standards, and sustainability. Some research centers have closed or restructured, and there are concerns about falling behind international standards. Yet the field's history of resilience, interdisciplinary collaboration, and openness to new ideas offers hope. Conferences, summer schools, and joint publications have built a vibrant community, and Russian projects are increasingly recognized globally. Education plays a crucial role – universities now offer courses in digital methods and programming for humanists, equipping the next generation with essential skills. The episode concludes with a balanced reflection: digital humanities in Russia is neither a triumph nor a failure, but a living, evolving field shaped by creativity, constraint, and the enduring human desire to understand culture through every available lens.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Why this matters for developers:&lt;/strong&gt;&lt;br&gt;
this is a case study in how technical infrastructure – corpora, annotation pipelines, search interfaces, visualization tools – gets built, maintained, and sometimes lost in a non-commercial, academically driven environment. The challenges of data standardization, long-term preservation, and cross-institutional collaboration are familiar to anyone who has worked on open-source projects or research software. The video also highlights the importance of domain expertise: the most sophisticated NLP pipeline is useless if it doesn't address meaningful research questions or if its outputs can't be interpreted by literary scholars. For developers interested in digital archives, historical data, or cultural analytics, this is a reminder that technical choices (like lemmatization schemes, metadata schemas, or statistical thresholds) have real consequences for scholarship. The tension between global best practices and local traditions – a recurring theme – mirrors the challenges of building software that must serve diverse user communities with different needs and expectations.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;What you'll take away:&lt;/strong&gt;&lt;br&gt;
a broad historical and institutional perspective on how a national research community adopted and adapted computational methods over more than a century. You'll see why the Russian case is both unique (due to its strong formalist tradition) and representative (facing the same pressures of funding, recognition, and technological change as digital humanities elsewhere). You'll learn about specific resources and projects that you might explore or cite in your own work – from the National Corpus to Prozhito to Tolstoy.Digital. Most importantly, you'll gain an appreciation for the non-technical factors that determine whether a digital humanities project succeeds or fails: institutional support, community building, open access policies, and the ability to tell compelling stories about why the data matters.&lt;/p&gt;
&lt;p&gt;Watch the full episode and ask yourself: if a century of quantitative literary analysis can survive revolutions, wars, and institutional upheaval, what does that say about the longevity of your own codebases and data pipelines? When you build a tool for scholars, are you designing for sustainability or just for the next publication? And if Russian digital humanities can maintain global relevance despite political and economic headwinds, what barriers in your own field are truly insurmountable – and which ones are just waiting for the right collaboration?&lt;br&gt;
Stay curious – and maybe take a moment to document your data sources and preprocessing steps as if someone in 2125 will need to understand them.&lt;/p&gt;
</description>
<category>Videos</category>
<guid isPermaLink="true">https://coderlegion.com/25490/digital-humanities-in-russia-from-bely-to-big-data-a-complete-history</guid>
<pubDate>Sun, 01 Nov 2026 02:57:00 +0000</pubDate>
</item>
<item>
<title>When Stylometry Meets Intellectual History: What &quot;The Bakhtin Circle&quot; Teaches Us About Authorship</title>
<link>https://coderlegion.com/25027/when-stylometry-meets-intellectual-history-what-bakhtin-circle-teaches-about-authorship</link>
<description>&lt;p&gt;Every developer who has worked with stylometry knows the drill: you feed a corpus of texts with known authors into a machine learning pipeline, extract a feature set (usually the most frequent words), apply some dimensionality reduction (PCA, anyone?), and watch as the unknown text confidently clusters with its true author. It's elegant, it's powerful, and for most practical cases—from disputed Federalist Papers to identifying anonymous blog posts—it works remarkably well.&lt;/p&gt;
&lt;p&gt;But what happens when you point this finely-tuned machinery at a &lt;em&gt;collective&lt;/em&gt; intellectual project? What if the &quot;author&quot; isn't an individual at all, but a circle of thinkers who read each other's drafts, argued over concepts, and published under each other's names? This is exactly the fascinating problem tackled by Orekhov and Vasilenko in their recent paper &quot;&lt;a rel=&quot;nofollow&quot; href=&quot;https://nevmenandr.github.io/portfolio/assets/pdf/bakhtin_delta.pdf&quot;&gt;Quantitative Authorship Attribution and the Bakhtin Circle: The Problem of Authorship in Intellectual Communities&lt;/a&gt;&quot; (2026).&lt;/p&gt;
&lt;h3 id=&quot;the-core-problem-attribution-in-a-collaborative-context&quot;&gt;The Core Problem: Attribution in a Collaborative Context&lt;/h3&gt;
&lt;p&gt;The Bakhtin Circle—that legendary group of Soviet thinkers including Mikhail Bakhtin, Valentin Voloshinov, and Pavel Medvedev—presents a nightmare scenario for traditional authorship attribution. These scholars didn't just influence each other; they actively collaborated, shared ideas, and in some cases, published works that may have been jointly authored or even ghost-written. The question &quot;Who really wrote &lt;em&gt;Marxism and the Philosophy of Language&lt;/em&gt;?&quot; has haunted literary scholars for decades.&lt;/p&gt;
&lt;p&gt;The paper, which I'll cite properly in a moment, approaches this not as a simple &quot;who wrote what&quot; puzzle, but as a methodological challenge: &lt;strong&gt;how do we test authorship attribution methods when the ground truth itself is contested, and when the very concept of &quot;individual authorship&quot; breaks down?&lt;/strong&gt;&lt;/p&gt;
&lt;h3 id=&quot;the-technical-approach-stylo-and-delta&quot;&gt;The Technical Approach: Stylo and Delta&lt;/h3&gt;
&lt;p&gt;The researchers used the excellent R package &lt;a rel=&quot;nofollow&quot; href=&quot;https://github.com/computationalstylistics/stylo&quot;&gt;&lt;code&gt;stylo&lt;/code&gt;&lt;/a&gt; (Eder et al., 2016)—a toolkit that has become the workhorse of computational stylistics. If you haven't used it, imagine &lt;code&gt;scikit-learn&lt;/code&gt; for text analysis but with a specialized focus on authorship problems. It implements the classic &quot;Delta&quot; method (Burrows' distance) along with various machine learning classifiers.&lt;/p&gt;
&lt;p&gt;Their pipeline is refreshingly straightforward:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Feature extraction&lt;/strong&gt;: They used the most frequent words (MFW) as features—a common and effective choice in stylometry&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Distance calculation&lt;/strong&gt;: They computed the Delta distance between texts&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Visualization&lt;/strong&gt;: They used Principal Component Analysis (PCA) and cluster analysis to visualize textual relationships&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Validation&lt;/strong&gt;: They performed cross-validation to test the stability of their attributions&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;What makes this study particularly interesting for developers is &lt;strong&gt;the data strategy&lt;/strong&gt;. Instead of relying on contested attributions, they built their training set from texts with &lt;em&gt;uncontroversial&lt;/em&gt; authorship—works that everyone agrees were written by Bakhtin, Voloshinov, or Medvedev individually. The test set then included the disputed works.&lt;/p&gt;
&lt;p&gt;This approach is worth remembering. In real-world attribution problems, we rarely have perfect ground truth. We often have to construct it from a subset of cases where attribution is clear, then test our methods on the ambiguous ones.&lt;/p&gt;
&lt;h3 id=&quot;the-surprising-and-cautionary-result&quot;&gt;The Surprising (and Cautionary) Result&lt;/h3&gt;
&lt;p&gt;Here's where it gets interesting for practitioners. The stylometric analysis did not produce the clean, unambiguous clusters that we often see in textbook examples. The disputed texts didn't neatly align with a single author. Instead, they showed varying degrees of similarity to different circle members.&lt;/p&gt;
&lt;p&gt;Now, a less careful researcher might have concluded &quot;the method doesn't work.&quot; But Orekhov and Vasilenko make a much more sophisticated argument: &lt;strong&gt;the ambiguous results actually reflect the collaborative reality of the Bakhtin Circle&lt;/strong&gt;. When Voloshinov and Bakhtin worked closely together, exchanging ideas and editing each other's prose, the resulting texts naturally contain elements of multiple writing styles. The &quot;noise&quot; in our stylometric analysis isn't a bug—it's a feature that tells us about the social dynamics of intellectual production.&lt;/p&gt;
&lt;p&gt;This is a crucial insight for developers: &lt;strong&gt;our tools are never neutral&lt;/strong&gt;. They encode assumptions about authorship (e.g., that style is stable and unique to individuals) that may not hold in all contexts. When we see ambiguous results, we should ask: is this a methodological failure, or is the data telling us something about the nature of authorship in this particular case?&lt;/p&gt;
&lt;h3 id=&quot;practical-implementation-r-code-for-stylometric-analysis&quot;&gt;Practical Implementation: R Code for Stylometric Analysis&lt;/h3&gt;
&lt;p&gt;To make this concrete, here's how you can perform a similar analysis using the &lt;code&gt;stylo&lt;/code&gt; package in R. The following code demonstrates the key steps used in the paper—from loading the corpus to performing cluster analysis and cross-validation.&lt;/p&gt;
&lt;h4 id=&quot;setting-up-the-environment&quot;&gt;Setting Up the Environment&lt;/h4&gt;
&lt;p&gt;First, install and load the necessary packages:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-r&quot;&gt;# Install stylo from CRAN if you haven't already
# install.packages(&quot;stylo&quot;)

# Load the package
library(stylo)
&lt;/code&gt;&lt;/pre&gt;
&lt;h4 id=&quot;basic-stylometric-analysis&quot;&gt;Basic Stylometric Analysis&lt;/h4&gt;
&lt;p&gt;The main workhorse function is &lt;code&gt;stylo()&lt;/code&gt;. Here's how to run a basic analysis on your own corpus:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-r&quot;&gt;# --- Basic stylometric analysis ---
# Place your plain text files (.txt) in a folder, e.g., &quot;bakhtin_corpus&quot;
# Each file should be named with author information, e.g., &quot;Bakhtin_text1.txt&quot;

results &amp;lt;- stylo(
    gui = FALSE,                        # Set to TRUE for interactive mode
    corpus.dir = &quot;bakhtin_corpus&quot;,      # Path to your text files
    mfw.min = 100,                      # Minimum number of most frequent words
    mfw.max = 100,                      # Maximum number (fixed set here)
    mfw.incr = 1,                       # Increment step (ignored if min=max)
    analysis.type = &quot;CA&quot;,               # &quot;CA&quot; for Cluster Analysis
    distance.measure = &quot;delta&quot;,         # Burrows' Delta distance
    write.png = TRUE                    # Save plots as PNG files
)

# View the distance matrix (first 10 rows and columns)
print(&quot;Delta distance matrix (first 10 texts):&quot;)
print(results$distance.matrix[1:10, 1:10])
&lt;/code&gt;&lt;/pre&gt;
&lt;h4 id=&quot;advanced-manual-feature-selection-and-cross-validation&quot;&gt;Advanced: Manual Feature Selection and Cross-Validation&lt;/h4&gt;
&lt;p&gt;For more control over the process—mirroring the approach in Orekhov and Vasilenko—you can manually extract features and perform cross-validation:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-r&quot;&gt;# --- Manual feature extraction ---
# Load all texts from the corpus directory
corpus &amp;lt;- load.corpus(files = &quot;all&quot;, corpus.dir = &quot;bakhtin_corpus&quot;)

# Create a table of frequencies for all words
all_freq &amp;lt;- make.frequencies(corpus)

# Select the top N most frequent words as features
n_features &amp;lt;- 150
mfw_list &amp;lt;- frequent.features(all_freq, cutoff = n_features)

# Create a table of relative frequencies for these features
# This is your feature matrix: rows = texts, columns = word frequencies
feature_table &amp;lt;- make.table.of.frequencies(corpus, mfw_list)

# --- Delta distance calculation ---
# Calculate Delta distance matrix manually
delta_matrix &amp;lt;- dist.delta(feature_table)

# Perform PCA for visualization
pca_results &amp;lt;- prcomp(delta_matrix, scale. = TRUE)

# Create a PCA plot with labels
plot(pca_results$x[,1], pca_results$x[,2],
     type = &quot;n&quot;,
     xlab = &quot;PC1&quot;, ylab = &quot;PC2&quot;,
     main = &quot;PCA of Bakhtin Circle Corpus&quot;)
text(pca_results$x[,1], pca_results$x[,2],
     labels = rownames(pca_results$x),
     cex = 0.7)

# --- Cross-validation to test classification accuracy ---
# Use the classify() function to perform leave-one-out cross-validation
# This tests how well the model can predict authorship for unknown texts

# First, create a vector of known author labels
# These should match the filenames or be provided separately
# For example: author_labels &amp;lt;- c(&quot;Bakhtin&quot;, &quot;Bakhtin&quot;, &quot;Voloshinov&quot;, ...)

# Perform cross-validation using the Delta distance
cv_results &amp;lt;- classify(
    corpus.dir = &quot;bakhtin_corpus&quot;,      # Path to corpus
    mfw = n_features,                   # Number of features to use
    distance.measure = &quot;delta&quot;,         # Distance measure
    cv.folds = 10,                      # Number of cross-validation folds
    classification.method = &quot;knn&quot;       # k-nearest neighbors classification
)

# Print classification accuracy
print(&quot;Cross-validation accuracy:&quot;)
print(paste(&quot;Overall accuracy:&quot;, 
            round(cv_results$accuracy * 100, 2), &quot;%&quot;))

# View the confusion matrix
print(&quot;Confusion matrix:&quot;)
print(cv_results$confusion.matrix)
&lt;/code&gt;&lt;/pre&gt;
&lt;h4 id=&quot;processing-the-bakhtin-circle-dataset&quot;&gt;Processing the Bakhtin Circle Dataset&lt;/h4&gt;
&lt;p&gt;If you want to replicate the analysis from the paper, you would modify the code to handle multiple feature sets and visualize the ambiguous cases:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-r&quot;&gt;# --- Iterative feature selection to find optimal MFW ---
# Test different numbers of most frequent words
# to find the range that yields the best separation

mfw_range &amp;lt;- seq(from = 50, to = 500, by = 50)
accuracy_results &amp;lt;- data.frame(mfw = mfw_range, accuracy = NA)

for (i in seq_along(mfw_range)) {
    cat(&quot;Testing MFW =&quot;, mfw_range[i], &quot;...\n&quot;)
    cv_result &amp;lt;- classify(
        corpus.dir = &quot;bakhtin_corpus&quot;,
        mfw = mfw_range[i],
        distance.measure = &quot;delta&quot;,
        cv.folds = 10,
        classification.method = &quot;knn&quot;
    )
    accuracy_results$accuracy[i] &amp;lt;- cv_result$accuracy
}

# Plot accuracy by number of features
plot(accuracy_results$mfw, accuracy_results$accuracy,
     type = &quot;b&quot;,
     xlab = &quot;Number of Most Frequent Words&quot;,
     ylab = &quot;Cross-Validation Accuracy&quot;,
     main = &quot;Feature Selection for Bakhtin Circle Attribution&quot;)

# --- Visualizing the ambiguous cases ---
# In the paper, disputed texts showed mixed affiliations
# You can highlight these in your plots

# Identify disputed texts (e.g., those with uncertain authorship)
# For demonstration, let's assume you have a vector of disputed text names
# disputed_texts &amp;lt;- c(&quot;Marxism_and_Philosophy_of_Language.txt&quot;, ...)

# In PCA plot, color disputed texts differently
is_disputed &amp;lt;- rownames(pca_results$x) %in% disputed_texts
plot_colors &amp;lt;- ifelse(is_disputed, &quot;red&quot;, &quot;blue&quot;)

plot(pca_results$x[,1], pca_results$x[,2],
     col = plot_colors,
     pch = ifelse(is_disputed, 17, 16),
     xlab = &quot;PC1&quot;, ylab = &quot;PC2&quot;,
     main = &quot;PCA with Disputed Texts Highlighted&quot;)
text(pca_results$x[,1], pca_results$x[,2],
     labels = rownames(pca_results$x),
     pos = 3, cex = 0.6)
&lt;/code&gt;&lt;/pre&gt;
&lt;h3 id=&quot;the-open-dataset-a-resource-for-the-community&quot;&gt;The Open Dataset: A Resource for the Community&lt;/h3&gt;
&lt;p&gt;One of the most valuable contributions of this paper is the release of their data on &lt;a rel=&quot;nofollow&quot; href=&quot;https://osf.io/r9wbz/&quot;&gt;OSF&lt;/a&gt;. The dataset (Orekhov &amp;amp; Vasilenko, 2025) contains:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The full corpus of texts used in the analysis&lt;/li&gt;
&lt;li&gt;Metadata about authorship attributions (both certain and disputed)&lt;/li&gt;
&lt;li&gt;The feature tables generated during analysis&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is a significant resource for developers working in stylometry. It provides a test case for authorship attribution in a &quot;difficult&quot; context—one where the usual assumptions about individual style don't straightforwardly apply. If your algorithm can handle the Bakhtin Circle, it can probably handle most real-world attribution problems.&lt;/p&gt;
&lt;h3 id=&quot;key-takeaways-for-developers&quot;&gt;Key Takeaways for Developers&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;1. Always validate your assumptions.&lt;/strong&gt; Stylometry works best when authorship is individual and style is consistent. Before applying it, ask: does the text corpus meet these assumptions? If not, how might that affect the results?&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;2. Embrace ambiguity.&lt;/strong&gt; Not every classification problem yields a clear answer. Sometimes the &quot;noise&quot; contains important information about the data generation process. In the Bakhtin Circle case, the ambiguous classifications revealed patterns of intellectual collaboration that traditional scholarship had debated for decades.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;3. Use the right tools for the right problem.&lt;/strong&gt; &lt;code&gt;stylo&lt;/code&gt; is a mature, well-documented package that handles the core tasks of authorship attribution elegantly. Its GUI interface makes it accessible to non-programmers, while its R backend allows for deep customization. The paper demonstrates its effectiveness even on challenging datasets.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;4. Share your data.&lt;/strong&gt; The authors' decision to release their dataset on OSF is a model for reproducible research. If you're working on stylometric problems, consider doing the same—it allows others to validate your findings and build upon your work.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;5. Tune your parameters carefully.&lt;/strong&gt; The choice of the number of most frequent words (MFW) can significantly impact results, as shown in the feature selection plot above. Always test a range of values and consider the stability of your classifications across different feature sets.&lt;/p&gt;
&lt;h3 id=&quot;the-full-citation&quot;&gt;The Full Citation&lt;/h3&gt;
&lt;p&gt;For those who want to dig deeper, here's the proper reference (paper in Russian):&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Orekhov, B.V., &amp;amp; Vasilenko, A.G. (2026). Quantitative Authorship Attribution and the Bakhtin Circle: The Problem of Authorship in Intellectual Communities. &lt;em&gt;New Philological Bulletin&lt;/em&gt;, (1), 40-55. DOI: &lt;a rel=&quot;nofollow&quot; href=&quot;https://doi.org/10.54770/20729316-2026-1-40&quot;&gt;10.54770/20729316-2026-1-40&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;And the dataset:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Orekhov, B.V., &amp;amp; Vasilenko, A.G. (2025). Quantitative Authorship Attribution and the Bakhtin Circle: the Problem of Authorship in Intellectual Communities (stylometry data). OSF. DOI: &lt;a rel=&quot;nofollow&quot; href=&quot;https://doi.org/10.17605/OSF.IO/R9WBZ&quot;&gt;10.17605/OSF.IO/R9WBZ&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
</description>
<category>Articles</category>
<guid isPermaLink="true">https://coderlegion.com/25027/when-stylometry-meets-intellectual-history-what-bakhtin-circle-teaches-about-authorship</guid>
<pubDate>Fri, 25 Sep 2026 03:31:00 +0000</pubDate>
</item>
<item>
<title>Open-Source Python Tools for Russian Text Processing in Digital Humanities</title>
<link>https://coderlegion.com/25025/open-source-python-tools-for-russian-text-processing-in-digital-humanities</link>
<description>&lt;p&gt;&lt;em&gt;Practical tools for OCR, accentuation, and text analysis that every developer should know about&lt;/em&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;h3 id=&quot;the-problem&quot;&gt;The Problem&lt;/h3&gt;
&lt;p&gt;If you're working with Russian texts in the humanities, you've probably faced this situation: you need to process a pre-1917 text, but every NLP tool is trained on modern Russian. Or you want to analyze verse, but stress marks are missing. Or you have a TEI XML document that needs conversion, but existing tools are either proprietary or dead.&lt;/p&gt;
&lt;p&gt;While large language models are everywhere, they're surprisingly bad at these specific tasks. As Orekhov (2025) points out in his review, &quot;modern versions of artificial intelligence cannot handle subtle nuances tied to specific subject domains.&quot;&lt;/p&gt;
&lt;p&gt;Worse, even when good solutions exist, nobody knows about them. Orekhov mentions at least two separate attempts to rebuild functionality that was already available in open-source Python packages — a clear sign that the real problem isn't technical, but informational.&lt;/p&gt;
&lt;hr&gt;
&lt;h3 id=&quot;what-s-actually-available&quot;&gt;What's Actually Available&lt;/h3&gt;
&lt;p&gt;Here are the tools Orekhov covers. All are open-source, Python-based, and created by researchers at HSE University's School of Linguistics.&lt;/p&gt;
&lt;h4 id=&quot;1-pre-1917-russian-orthography-support&quot;&gt;1. Pre-1917 Russian Orthography Support&lt;/h4&gt;
&lt;h5 id=&quot;ocr-model-for-old-orthography&quot;&gt;OCR Model for Old Orthography&lt;/h5&gt;
&lt;p&gt;&lt;strong&gt;Repository&lt;/strong&gt;: &lt;a rel=&quot;nofollow&quot; href=&quot;https://huggingface.co/Serovvans/trocr-preferform-orthography&quot;&gt;Serovvans/trocr-preferform-orthography&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The model uses Tesseract-OCR infrastructure and handles the pre-reform Russian alphabet (including Ѣ, Ѳ, Ѵ, and the final Ъ). It's actually trained on modern Russian text in old orthography (not Old Church Slavonic — the model's description incorrectly labels it as &quot;Old Russian&quot;).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Quality metrics&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;CER (Character Error Rate): 0.095&lt;/li&gt;
&lt;li&gt;WER (Word Error Rate): 0.298&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This means only about 70% of words are correctly recognized. Not great, but it's free and works well enough when combined with GPT-4 for correction, as Orekhov demonstrates with an example where GPT-4o successfully cleans up OCR errors while preserving the old orthography.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Example usage&lt;/strong&gt;:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-python&quot;&gt;from recognize_page import recognize_page

page_path = &quot;page.png&quot;
text = recognize_page(page_path, text_output_path=&quot;output/file.txt&quot;)
print(f&quot;Text from page:\n{text}&quot;)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Dependencies&lt;/strong&gt;: tesseract-ocr, poppler, pytesseract, Pillow, transformers, torch, numpy, tqdm, pdf2image&lt;/p&gt;
&lt;p&gt;⚠️ &lt;strong&gt;Note&lt;/strong&gt;: For 18th-century texts with that weird 'm'-shaped 'т', this model doesn't work well. You'll need proprietary Transkribus models (which Orekhov warns against due to their dependence on a private company with an uncertain future).&lt;/p&gt;
&lt;h5 id=&quot;old-to-new-orthography-converter&quot;&gt;Old to New Orthography Converter&lt;/h5&gt;
&lt;p&gt;&lt;strong&gt;Repository&lt;/strong&gt;: &lt;a rel=&quot;nofollow&quot; href=&quot;https://pypi.org/project/preferom2modern/&quot;&gt;preferom2modern on PyPI&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Simple, dependency-free string conversion. Can output plain text or TEI XML with &lt;code&gt;&amp;lt;choice&amp;gt;&lt;/code&gt; tags preserving both versions:&lt;/p&gt;
&lt;pre&gt;&lt;code class=&quot;language-xml&quot;&gt;&amp;lt;choice&amp;gt;
  &amp;lt;reg&amp;gt;пример&amp;lt;/reg&amp;gt;
  &amp;lt;orig&amp;gt;примѣръ&amp;lt;/orig&amp;gt;
&amp;lt;/choice&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Works from both command line and Python. Orekhov notes it's been unmaintained for years, but that's actually fine — it worked correctly from day one.&lt;/p&gt;
&lt;hr&gt;
&lt;h4 id=&quot;2-literary-and-poetic-text-analysis&quot;&gt;2. Literary and Poetic Text Analysis&lt;/h4&gt;
&lt;h5 id=&quot;accentuation-module-for-russian-verse&quot;&gt;Accentuation Module for Russian Verse&lt;/h5&gt;
&lt;p&gt;&lt;strong&gt;Repository&lt;/strong&gt;: &lt;a rel=&quot;nofollow&quot; href=&quot;https://pypi.org/project/ru-accent-poet/&quot;&gt;ru-accent-poet on PyPI&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;This is a hybrid approach: dictionary-based + recurrent neural network. It's designed specifically for poetic texts where stress placement is crucial for meter analysis.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Performance&lt;/strong&gt;: The main downside is speed. A pre-annotated dataset of classical Russian prose was published in 2024 to let researchers skip the processing step.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Citation&lt;/strong&gt;: Короткова Ю. О. (2022). Комбинированный словарно-нейросетевой акцентуатор для разметки русского поэтического текста. &lt;em&gt;Труды Института русского языка им. В. В. Виноградова&lt;/em&gt;, 3(33), 181–190.&lt;/p&gt;
&lt;h5 id=&quot;direct-speech-detector&quot;&gt;Direct Speech Detector&lt;/h5&gt;
&lt;p&gt;&lt;strong&gt;Repository&lt;/strong&gt;: &lt;a rel=&quot;nofollow&quot; href=&quot;https://github.com/neiriz/direct-speech-detector&quot;&gt;neiriz/direct-speech-detector&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;This isn't just a rule-based approach — it's a full pipeline combining regex (for direct speech marks) with morphological parsing, parsing, and text segmentation. It handles cases where direct speech and authorial narration are mixed.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Use cases&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Comparing dialogue density across novels (there's previous work by Sobchuk, 2016)&lt;/li&gt;
&lt;li&gt;Analyzing character speech patterns&lt;/li&gt;
&lt;li&gt;Extracting dialogue for stylistic analysis&lt;/li&gt;
&lt;/ul&gt;
&lt;h5 id=&quot;formulaicity-assessment-module&quot;&gt;Formulaicity Assessment Module&lt;/h5&gt;
&lt;p&gt;&lt;strong&gt;Repository&lt;/strong&gt;: &lt;a rel=&quot;nofollow&quot; href=&quot;https://github.com/evgenilatyshev/formulaicity&quot;&gt;evgenilatyshev/formulaicity&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;This is for folklore studies. It detects formulaic expressions — repeated phrases characteristic of oral traditions. The algorithm combines:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Vocabulary variability coefficient (VocD)&lt;/li&gt;
&lt;li&gt;N-gram coefficients&lt;/li&gt;
&lt;li&gt;Binomial expressions&lt;/li&gt;
&lt;li&gt;Fixed phrases&lt;/li&gt;
&lt;li&gt;Interjections&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Example output&lt;/strong&gt; (from a folk song analyzed by the module's author, V. Sidnenko):&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;Found n-grams: 
- черный ворон

Formulaicity coefficient: 1.207
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Dependencies&lt;/strong&gt;: spacy, pymorphy2, numpy, sklearn&lt;/p&gt;
&lt;h5 id=&quot;tei-xml-converter&quot;&gt;TEI XML Converter&lt;/h5&gt;
&lt;p&gt;&lt;strong&gt;Repository&lt;/strong&gt;: &lt;a rel=&quot;nofollow&quot; href=&quot;https://pypi.org/project/TEItransformer/&quot;&gt;TEItransformer on PyPI&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;TEI is the standard XML format for digital humanities. This module, developed by Kostyanitsyna and Skorinkin (2024), converts TEI XML to:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;DOCX&lt;/strong&gt; (editable format)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;HTML&lt;/strong&gt; (with search functionality)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;JSON&lt;/strong&gt; (for data analysis)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Under the hood, it's actually an XSLT wrapper — which Orekhov candidly calls an &quot;outdated and unreliable technology&quot; for XML processing, but acknowledges that building a pure Python alternative proved too complex.&lt;/p&gt;
&lt;p&gt;The generated HTML includes a search interface that lets you query by character, speaker, or any TEI-annotated field.&lt;/p&gt;
&lt;hr&gt;
&lt;h4 id=&quot;3-journal-publication-preprocessing&quot;&gt;3. Journal Publication Preprocessing&lt;/h4&gt;
&lt;h5 id=&quot;cgi-processor&quot;&gt;cgi-processor&lt;/h5&gt;
&lt;p&gt;&lt;strong&gt;Repository&lt;/strong&gt;: &lt;a rel=&quot;nofollow&quot; href=&quot;https://github.com/nevmenandr/cgi_processor&quot;&gt;nevmenandr/cgi_processor&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;This is a niche tool for the journal &quot;Цифровые гуманитарные исследования&quot; (Digital Humanities Research). It handles LaTeX preprocessing:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Non-breaking spaces between initials and surnames&lt;/li&gt;
&lt;li&gt;Em-dashes in number ranges&lt;/li&gt;
&lt;li&gt;Other typographic rules specific to Russian academic publishing&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The journal uses LaTeX for production, and manual typographic cleanup is error-prone. This automates it, saving editors from the tedious process of manually inserting special characters.&lt;/p&gt;
&lt;hr&gt;
&lt;h3 id=&quot;why-this-matters&quot;&gt;Why This Matters&lt;/h3&gt;
&lt;p&gt;Here's the thing: these tools solve real problems that LLMs can't handle well. They're infrastructure. They're the kind of boring, practical code that enables interesting research.&lt;/p&gt;
&lt;p&gt;The bigger issue, as Orekhov emphasizes, is visibility. Researchers often don't even check if their problem has already been solved. There's a mental block: &quot;my problem is so specific, surely nobody else has built this.&quot;&lt;/p&gt;
&lt;p&gt;But they have. And it's open source.&lt;/p&gt;
&lt;hr&gt;
&lt;h3 id=&quot;reference&quot;&gt;Reference&lt;/h3&gt;
&lt;p&gt;Orekhov, B. V. (2025). Otkrytye komp'yuternye instrumenty dlya resheniya zadach otsifrovki i analiza russkoyazychnogo teksta v oblasti Digital Humanities [Open computer tools for digitization and analysis of Russian-language text in Digital Humanities]. &lt;em&gt;Tsifrovye gumanitarnye issledovaniya&lt;/em&gt;, (2), 71–83. DOI: &lt;a rel=&quot;nofollow&quot; href=&quot;https://doi.org/10.31860/cgi-2025-2-71-83&quot;&gt;10.31860/cgi-2025-2-71-83&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Available (in Russian) at: &lt;a rel=&quot;nofollow&quot; href=&quot;https://nevmenandr.github.io/portfolio/assets/pdf/cgi2025-2-tools.pdf&quot;&gt;https://nevmenandr.github.io/portfolio/assets/pdf/cgi2025-2-tools.pdf&lt;/a&gt;&lt;/p&gt;
</description>
<category>Articles</category>
<guid isPermaLink="true">https://coderlegion.com/25025/open-source-python-tools-for-russian-text-processing-in-digital-humanities</guid>
<pubDate>Tue, 15 Sep 2026 08:52:00 +0000</pubDate>
</item>
<item>
<title>Code, Poetry &amp; Neural Ghosts: What AI Teaches Us About Verse</title>
<link>https://coderlegion.com/22850/code-poetry-neural-ghosts-what-ai-teaches-us-about-verse</link>
<description>&lt;p&gt;As the author of &quot;Electronic Lyre&quot; (Электронная лира), I want to share why this book is essential reading for developers working at the intersection of code and creativity. For years, we've treated poetry generation as a side project – a fun way to test RNNs or transformers. But when I started training models on Russian poets like Mandelstam and Pasternak, I realized we're not just building &quot;poetry machines.&quot; We're building statistical mirrors of style itself. The book dives into how moving from combinatorial text shuffling (the 20th-century approach) to predictive neural networks didn't just change the output quality; it changed &lt;em&gt;what&lt;/em&gt; the machine is actually doing. It's no longer playing dice with words; it's learning the fingerprint of a poet's syntax, rhythm, and even their typical errors.&lt;/p&gt;
&lt;p&gt;From an engineering perspective, the most fascinating part was running the experiments that blurred the line between human and machine writing. I used LSTM-based models trained character-by-character and, later, ruGPT transformers. The goal wasn't to pass the Turing test – that's boring for both coders and poets. Instead, I asked a critical question: can a professional philologist interpret a poem generated by a neural network as a meaningful text? The results were paradoxical. Statistically, the generated texts were often &lt;em&gt;more&lt;/em&gt; like the original author than the author's own poems (when tested with methods like Burrows's Delta). To a human reader, however, they felt broken, uncanny – like a face in the &quot;uncanny valley.&quot; But it's precisely this imperfection, these &quot;glitches&quot; in the statistical distribution, that create new, unexpected poetic images. It taught me that our code, when it fails gracefully, can produce something more creatively interesting than when it succeeds perfectly.&lt;/p&gt;
&lt;p&gt;This book isn't a manual, but it's a field guide for navigating the literary landscape we've helped build. It tackles the hard questions we often avoid: can we ever truly own the output of a model? Will we be replaced? My argument is that because a machine generates text without effort or personal cost, its product will never hold the same value as human work – just as a printed book didn't make manuscripts worthless. The real future isn't replacement, but augmentation: using the AI as a prosthetic for exploring style, creating &quot;neural readings&quot; of poetic epochs. If you're a developer who works with language models, this book will challenge you to think about your code not as a tool that produces text, but as a participant in a very old conversation about what it means to write and read.&lt;/p&gt;
&lt;p&gt;Let me expand on why this matters for developers and what you'll actually find inside.&lt;/p&gt;
&lt;p&gt;When I started this project, I wasn't interested in another &quot;AI writes poetry&quot; demo. We've all seen those. Instead, I wanted to understand what neural networks reveal about the &lt;em&gt;structure&lt;/em&gt; of poetic language when we train them not on generic text but on specific, stylistically dense corpora. The technical core of the book is built around two distinct approaches: first, the old-school combinatorial methods (random word selection from a thesaurus with metrical constraints), and second, the modern LSTM and transformer architectures that learn patterns at the character and word level. I walk through the actual training process – from the first chaotic epoch where the model outputs nothing but random Cyrillic letters, to the moment when it begins to grasp morphological patterns like &quot;-ение&quot; or &quot;-ость&quot;, to the astonishing point where it can generate grammatically correct but semantically broken lines that somehow &lt;em&gt;feel&lt;/em&gt; like Russian poetry. This isn't magic; it's statistics applied to sequences, but the emergent properties are genuinely surprising.&lt;/p&gt;
&lt;p&gt;One experiment I describe in detail is particularly revealing for developers working on generative models. I trained a network on the complete poetic works of Nekrasov and then generated hundreds of new four-line stanzas. Then I asked professional philologists – people who literally memorize poetry for a living – to identify which fragments were original and which were synthetic. The results were counterintuitive: when I used high &quot;temperature&quot; settings to encourage creativity, the experts could often tell something was wrong, but they couldn't agree on &lt;em&gt;what&lt;/em&gt; was wrong. The generated texts had the right vocabulary distribution, the right rhythm patterns (mostly iambic), and even the right thematic clusters (peasants, suffering, Russian landscapes). Yet the texts felt &quot;off&quot; – like a voice that sounds familiar but isn't quite your friend's. The statistical methods, however, told a different story: Burrows's Delta, a standard authorship attribution algorithm, consistently misclassified the generated texts as &lt;em&gt;more&lt;/em&gt; typical of Nekrasov than Nekrasov's own poems. The machine had become a statistical caricature of the author, amplifying his most frequent patterns while missing the rare, surprising choices that make human poetry human.&lt;/p&gt;
&lt;p&gt;This leads to a deeper point about what our models actually learn. We often talk about &quot;understanding&quot; in AI, but what we really have is pattern compression. A neural network trained on a poet's work doesn't understand love, death, or winter; it understands that certain words (like &lt;em&gt;stikhotvorenie&lt;/em&gt; for poem, &lt;em&gt;niy&lt;/em&gt; for grief, or &lt;em&gt;nebo&lt;/em&gt; for sky) co-occur with certain others in predictable ratios. It has no lived experience of weather or emotion. Yet, and this is the crux of the book, that doesn't make its output meaningless. Meaning, I argue, is not intrinsic to the text; it's constructed by the reader. When I read a generated line like &quot;Ветер знакомый ребенком&quot; (The wind familiar like a child), my brain desperately tries to find coherence. I supply the missing context, the emotional weight, the possible backstory. The machine didn't intend that meaning, but I, as a reader, created it. This is exactly what happens when we read human poetry too – we just forget that we're doing it. The difference is that with human authors, we can point to a conscious mind that &lt;em&gt;might&lt;/em&gt; have intended something similar. With AI, we have no such anchor, so we're forced to confront our own role in the meaning-making process.&lt;/p&gt;
&lt;p&gt;From a developer's perspective, this raises practical and ethical questions. When we deploy a language model for creative writing, are we building a tool, a collaborator, or a substitute? I argue for the &quot;collaborator&quot; model. The book explores the concept of &quot;computational creativity&quot; and shows how developers can use generative models not to replace writers but to act as &quot;idea generators&quot; or &quot;style explorers.&quot; For example, if you're designing a writing assistant, you could use a model to suggest unexpected word combinations based on the statistical style of a particular author, rather than just the most probable next token. This moves beyond autocomplete into creative inspiration. I also discuss the legal and philosophical issues: if a model generates a commercial poem, who owns it? My argument is that because the machine has no conscious intent and exerts no effort, the creator is not the algorithm but the human who selects, curates, and contextualizes its output – just as Marcel Duchamp claimed ownership of his readymade urinal not because he made it, but because he &lt;em&gt;presented&lt;/em&gt; it as art.&lt;/p&gt;
&lt;p&gt;Another theme I delve into is the historical continuity between AI poetry and earlier avant-garde movements. The combinatorial approach of the 1960s (randomly recombining words from a dictionary) was not a technological necessity but a &lt;em&gt;cultural&lt;/em&gt; one. It was part of a broader artistic protest against rationalism and control – a legacy of Dada and surrealism. When I write code that shuffles words, I'm unknowingly continuing a century-long conversation about randomness and creativity. This is something developers often miss when we focus solely on metrics like BLEU scores or perplexity. The book connects the dots between early experiments like the &quot;Muses&quot; of the 1950s, the Soviet &quot;Skol&quot; programs of the 1980s, and today's GPT-based systems. It shows that each generation of poets and programmers has used the tools available to question what art is and who gets to make it. That historical perspective, I hope, will help developers think more critically about the social and cultural implications of their code.&lt;/p&gt;
&lt;p&gt;For the more technically inclined, I include a chapter on the architecture of the models I used, including specific hyperparameters, training data preparation, and the challenges of generating verse in a highly inflected language like Russian. I also share code snippets and practical lessons learned: why character-level training initially fails but eventually captures morphological nuances, why temperature sampling is a crude but effective control for creativity, and why transformers, despite their advantages, still struggle with narrative coherence beyond a few lines. The book isn't a tutorial, but it provides enough detail for a skilled developer to reproduce the experiments and adapt them to other languages and literatures.&lt;/p&gt;
&lt;p&gt;Finally, I want to stress that this book is written for a specific audience: developers who are curious about the humanities, and humanists who want to understand code. It's intentionally low on math and high on metaphor. I explain backpropagation and attention mechanisms through analogies to reading and interpretation. The goal is to create a conversation, not a technical manual. If you're a coder, you'll find familiar concepts presented in an unfamiliar light. If you're a poet, you'll see your craft through the lens of statistics and probability. And if you're both – or aspiring to be – I believe you'll find this book a valuable companion in navigating the emerging landscape where software and literature increasingly overlap.&lt;/p&gt;
&lt;p&gt;I hope you'll pick it up and join the discussion.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;strong&gt;Publisher's Page:&lt;/strong&gt; &lt;a rel=&quot;nofollow&quot; href=&quot;https://eupress.ru/books/index/item/id/480&quot;&gt;https://eupress.ru/books/index/item/id/480&lt;/a&gt;&lt;/p&gt;
</description>
<category>Articles</category>
<guid isPermaLink="true">https://coderlegion.com/22850/code-poetry-neural-ghosts-what-ai-teaches-us-about-verse</guid>
<pubDate>Sat, 15 Aug 2026 22:31:00 +0000</pubDate>
</item>
</channel>
</rss>