research/near-duplicate-repos-by-description
Detecting Near-Duplicate GitHub Repos by Description, and Where Prefix Matching Fails
Engineering Note · November 6, 2026 · 6 minute read · Node.js · Text Matching · GitHub
Normalize the descriptions first, then compare them. Both common comparisons have a weak spot. Matching the first 20 characters catches simple copies, but it misses a description with reordered wording and wrongly pairs different projects that open the same way. Comparing the overlap of words fixes the reordering problem, but in our test it still could not separate two real duplicates from a different product. The honest answer is that description text alone is not enough.
Everything below was run with Node.js 22.23.4 on Linux. The eight repository descriptions are hypothetical ones we wrote for this test, not real repositories. The outputs are the real ones from this run.
Why does a small tool need this?
A weekly list of trending repositories can include the same project twice: a fork, a rename, or a copy with a lightly reworded description. If you rank or announce those lists, you want one entry per project. The description is the cheapest signal to compare, so it is where most people start.
What does normalizing do?
Raw descriptions differ in ways that do not matter: capital letters, punctuation, emoji, filler words. We strip those out.
const STOP = new Set(['a','an','the','and','or','for','to','of','in','on','with','your','my','that','is','are','by','using','built']);
export function normalize(s) {
return s.toLowerCase()
.replace(/[^\p{L}\p{N}\s]/gu, ' ') // drop punctuation and emoji
.split(/\s+/).filter(w => w && !STOP.has(w)).join(' ');
}
export const prefixKey = (s, n = 20) => normalize(s).slice(0, n);
export function jaccard(a, b) {
const A = new Set(normalize(a).split(' ')), B = new Set(normalize(b).split(' '));
const inter = [...A].filter(w => B.has(w)).length;
return inter / (A.size + B.size - inter);
}The \p{L} and \p{N} patterns match letters and numbers in any language, so the emoji and punctuation are replaced by spaces and everything else stays.
How did we test it?
We wrote eight descriptions. Two sets are true duplicates written differently (an emoji and a hyphen, reordered words, and repos against repositories), and one pair is a trap: different products with the same opening.
import { normalize, prefixKey, jaccard } from './dupes.mjs';
// Hypothetical repo descriptions, written for this test.
const repos = [
{ name: 'a/todo-cli', d: 'A fast command-line todo app written in Rust' },
{ name: 'b/todo-cli', d: '✨ A fast command line todo app, written in Rust!' }, // true dupe: emoji, hyphen, comma
{ name: 'c/rusty-todo', d: 'Command-line todo app written in Rust, fast' }, // true dupe: reordered
{ name: 'd/todo-web', d: 'A fast todo app for the browser written in TypeScript' }, // different
{ name: 'e/sched', d: 'A weekly countdown reel generator for GitHub repos' },
{ name: 'f/sched2', d: 'A weekly countdown reel generator for GitHub repositories' }, // true dupe: repos/repositories
{ name: 'g/gen-a', d: 'A tool that generates weekly reels from trending GitHub data' },
{ name: 'h/gen-b', d: 'A tool that generates a weekly report from trending GitHub data' } // different product, same opening
];
console.log('Normalized:');
for (const r of repos) console.log(' ', r.name.padEnd(12), JSON.stringify(normalize(r.d)));
function pairs(fn, label) {
console.log(`\n${label}`);
for (let i = 0; i < repos.length; i++) for (let j = i + 1; j < repos.length; j++) {
const v = fn(repos[i].d, repos[j].d);
if (v) console.log(` ${repos[i].name} ~ ${repos[j].name}${typeof v === 'number' ? ' ' + v.toFixed(2) : ''}`);
}
}
pairs((a, b) => prefixKey(a) === prefixKey(b), 'Prefix match (first 20 normalized chars):');
pairs((a, b) => { const j = jaccard(a, b); return j >= 0.6 ? j : 0; }, 'Token overlap >= 0.6:');Normalized:
a/todo-cli "fast command line todo app written rust"
b/todo-cli "fast command line todo app written rust"
c/rusty-todo "command line todo app written rust fast"
d/todo-web "fast todo app browser written typescript"
e/sched "weekly countdown reel generator github repos"
f/sched2 "weekly countdown reel generator github repositories"
g/gen-a "tool generates weekly reels from trending github data"
h/gen-b "tool generates weekly report from trending github data"
Prefix match (first 20 normalized chars):
a/todo-cli ~ b/todo-cli
e/sched ~ f/sched2
g/gen-a ~ h/gen-b
Token overlap >= 0.6:
a/todo-cli ~ b/todo-cli 1.00
a/todo-cli ~ c/rusty-todo 1.00
b/todo-cli ~ c/rusty-todo 1.00
e/sched ~ f/sched2 0.71
g/gen-a ~ h/gen-b 0.78Where does prefix matching fail?
It fails in both directions.
- It misses reordered text.
c/rusty-todois the same description with words moved, and its first 20 characters differ, so prefix matching never pairs it with the other two. - It flags different products.
g/gen-amakes weekly reels andh/gen-bmakes a weekly report. They share the opening “tool generates weekly”, so their first 20 characters match.
Where does token overlap fail?
Token overlap found the reordered duplicate, with a score of 1.00. It did not solve the second trap. We ran the three interesting pairs on their own:
import { jaccard } from './dupes.mjs';
const pairs = [
['reordered dupe (a,c)', 'A fast command-line todo app written in Rust', 'Command-line todo app written in Rust, fast'],
['repos/repositories dupe (e,f)', 'A weekly countdown reel generator for GitHub repos', 'A weekly countdown reel generator for GitHub repositories'],
['different product (g,h)', 'A tool that generates weekly reels from trending GitHub data', 'A tool that generates a weekly report from trending GitHub data'],
];
for (const [n, a, b] of pairs) console.log(n.padEnd(32), jaccard(a, b).toFixed(2));reordered dupe (a,c) 1.00
repos/repositories dupe (e,f) 0.71
different product (g,h) 0.78The different product scores higher than a true duplicate. No threshold can keep the 0.71 pair and reject the 0.78 pair. Raise the bar to 0.8 and you lose the repos duplicate. Lower it to 0.7 and you keep the false match. The cause is simple: swapping one word is a large change when the word is the product’s purpose, and a small one when it is a plural.
What would we try next?
We have not tested these, so treat them as ideas.
- Add a second signal. Require the primary language or the repository name to be similar as well as the description.
- Stem words. Treating
reposandrepositoriesas one word would raise that pair’s score. - Review borderline pairs by hand. If a list has a few dozen entries a week, a person can look at the pairs between 0.6 and 0.9.
Limitations
- Eight made-up descriptions is a demonstration, not a measurement. We did not test against real repositories, so we cannot say how often either method fails in practice.
- The stop-word list is short and English only.
- Forks and renames are not always visible in the description. We did not use any GitHub fields other than the text we supplied.
- The 20-character prefix and the 0.6 cutoff are our choices. Other values change the results.
Related reading
- Comparing Version Strings Without a Library: A 30-Line Semver Compare, Fuzzed Against the Reference
- Fuzzing an LLM JSON Extractor: What Breaks When You Parse Between the First and Last Brace
- Guarding an Automated Social Post: Four Checks Before Anything Goes Out
Frequently asked questions
How do you detect near-duplicate repository descriptions?
Normalize each description by lowercasing it, removing punctuation and emoji, and dropping filler words. Then compare the results. Comparing the first N characters is quick. Comparing the overlap of the words, called Jaccard similarity, handles reordered wording better.
What is Jaccard similarity?
It is the number of words two texts share divided by the number of distinct words in either. A score of 1.00 means the same set of words, and 0 means none in common. Order does not matter.
Why does prefix matching fail?
It only sees the start of the text. In our test, a description with the same words in a different order did not match, and two different products that began with the same phrase did match.
Can a single similarity threshold separate duplicates from different projects?
Not in our test. A real duplicate that swapped one word scored 0.71, while a different product with the same opening scored 0.78. Any threshold that catches the first also flags the second, so we would add a second signal, such as the repository name or the programming language.
Do you need a library for this?
Not for a first pass. Our matcher is about 15 lines of Node.js with no dependencies. A library becomes useful when you need stemming, so that repos and repositories count as the same word.