Andrew Trettel's website

Wosp: advanced full-text search on the command line

2025-11-04T00:00:00+00:00

Summary</h2>
Wosp is a command-line program that performs full-text search on text documents. Wosp stands for word-oriented search</em> and print</em>. It is designed for advanced searchers. It works differently than line-oriented search tools like grep, so it can search for matches spanning multiple lines. Wosp supports an expressive query language that contains both Boolean and proximity operators. It also supports nested queries, truncation, wildcard characters, and fuzzy searching.</p>

Git repository</h2>
The source code is available on GitHub</a>. I have licensed the project under the GPL version 3. The README file in the repository covers the build and installation instructions.</p>
Motivations and requirements</h2>
I am a scientist and a researcher. As part of my research, I often search for obscure information and hard-to-define concepts. Some of my searches occur locally. Over the years, I have accumulated a substantial amount of documents that are unavailable online, so I also depend on being able to search for information locally.</p>
Search engines often appear magical. Searchers, like wizards, summon the relevant information with the right incantation (well, query). Of course, search is not magic, but the seemingly miraculous nature of search makes it difficult at times to grasp what went wrong when a searcher cannot find something. Should I narrow this search more? Did I use the wrong term? How does a search engine work anyway?</p>
In the last few years, people have often discussed how "search is broken" in some way. This statement usually applies to online search engines, especially Google. Several factors have contributed to the drop in search quality: extensive search engine optimization</a>, alleged attempts by Google to increase advertising revenue</a>, or the recent introduction of AI overviews hindering deeper searching. Searching on Google now feels adversarial at times, like you have to trick the system to get good results.</p>
The common issue in all three of these factors is that searchers have little control over the results Google returns. That realization made me wonder if I can create some search tool that puts me and other searchers firmly in control of our own searches.</p>
Searchers need repeatable tools that work reliably and do not degrade over time as external incentives change. Ideally, these tools should work locally to search private information. Search tools should also be easy to understand, so that searchers can "look under the hood" and improve the tools as needed. That removes the magical feeling, but it was never magic to begin with. Searchers do not have to accept broken products that operate like magical and untouchable black boxes.</p>
Initially, I did not want to create another program, so I searched for something. I found several interesting programs but concluded that no existing program satisfies my requirements.</p>
I need a search program with the following features:</p>

Command-line operation.</em> The program must work according to the Unix philosophy</a>.</p> </li>

Simple and clear program architecture.</em> The program must have a well-defined scope to avoid feature creep. I want a simple and clear program architecture that follows guidelines like the Suckless philosophy</a> among others.</p> </li>

Boolean and proximity operators.</em> The program must have sentence and paragraph proximity operators. IBM and Data Central (now LexisNexis) independently invented proximity search</a> in the 1960s (Bourne 2003</a>). Proximity search ensures that a document discusses two concepts in the same context, which matters when searching especially long documents.</p> </li>

Wildcard search.</em> Wildcard search</a> enables searching for a variety of terms compactly.</p> </li>

Fuzzy search.</em> Fuzzy search</a> helps search poorly OCRed documents and locates relevant documents even if they have typos.</p> </li>

Nested queries.</em> Support for nested queries means that the query language should allow for the arbitrary combination of all operations. For example, some programs only allow proximity search for words and not for phrases or groups of words. Nested queries make the query language expressive</em> by enabling searchers to search for more precise concepts.</p> </li>

No stored index required.</em> The program must not require a stored index, so it can be run on any arbitrary collection of documents. I need to use the program on any bit of text rather than on a limited library of documents.</p> </li> </ul>
I have considered various local search tools, but none met all of these requirements. agrep</a>, pdfgrep</a>, ripgrep</a>, ag</a>, and ugrep</a> all follow the basic model provided by grep</a> of searching for regular expressions on the command line without any stored indices. qgrep</a> adds stored indices to speed up the search. ack</a> is similar to grep and includes Boolean operations and line-based proximity search, which is too limited for my purposes. Recoll</a>, Aleph</a>, Open Semantic Search</a>, and DocFetcher</a> operate using a graphical-user interface and require stored indices.</p>

A Venn diagram comparing the features of various search programs, including their support for fuzzy or proximity search and their use of stored indices.</figcaption> </figure>
The above Venn diagram compares the features of these local search tools. The Venn diagram excludes several variants of grep due to them lacking fuzzy or proximity search. Aleph, DocFetcher, and Recoll are the most capable of these local search tools, but all require a stored index and Aleph requires a client-server architecture, making it better suited for organizations than individuals. Aleph and Recoll both have the weak form of proximity search that disallows nested proximity operations (I cannot find much information about DocFetcher's proximity search).</p>
Since no existing program met my requirements, I decided to write my own full-text search program designed for advanced searchers. The next section goes into how to use Wosp.</p>
How to use Wosp</h2>
The basic format of a Wosp command is as follows:</p>
`$ wosp "QUERY" file1.txt file2.txt ... </span></code></pre>`Queries are how searchers find information using Wosp's query language. The query language is influenced by the query language used at the United States Patent and Trademark Office</a>. The USPTO's query language is an implementation of the query language used in BRS/Search</a>, which was a copy of the query language used in IBM STAIRS</a>. That said, Wosp's query language includes many advanced features not found in many other systems.</p> To search for a keyword in some documents, a searcher just needs to use the word:</p> $ wosp "detective" A_Study_in_Scarlet.txt </span> </span>A_Study_in_Scarlet.txt:816:am the only one in the world. I'm a consulting detective, if you can understand </span>... </span></code></pre> All of my examples come from the first Sherlock Holmes book, A Study in Scarlet</a>.</p> The wildcard character is ?</code>. That character matches any character in the text. Wosp also supports truncation using either the dollar sign $</code> or the number sign #</code> followed optionally by a number for the maximum number of characters to include. Searchers should include the number or else performance may suffer, though. Wildcards and truncation allow for powerful queries that can match many different words with one wildcard given:</p> $ wosp "#2m?n" A_Study_in_Scarlet.txt </span> </span>A_Study_in_Scarlet.txt:286:free as an income of eleven shillings and sixpence a day will permit a man to </span>... </span></code></pre> #2m?n</code> matches "man", "men", "woman", "women", and "human" all in a single term.</p> Wosp supports different case sensitivity options and fuzzy search with a given edit distance. The edit distance is the number of errors that Wosp can correct. It widens the search for keywords and can find typos, but too much edit distance may find too many results.</p> The default is case insensitive search with an edit distance of zero. Searchers control the case sensitivity and edit distance using the search option commands ICASE</code> (insensitive case), SCASE</code> (sensitive case), LCASE</code> (lowercase), UCASE</code> (uppercase), and TCASE</code> (title case or more specifically start case, where the initial letter is capitalized). These commands must precede either a wildcard term or parentheses.</p> $ wosp "TCASE holmes" A_Study_in_Scarlet.txt </span> </span>... </span>A_Study_in_Scarlet.txt:425:"Dr. Watson, Mr. Sherlock Holmes," said Stamford, introducing us. </span>... </span></code></pre> To specify the edit distance with one of these commands, add the edit distance number to the end of the command:</p> $ wosp "ICASE1 detective" A_Study_in_Scarlet.txt </span> </span>... </span>A_Study_in_Scarlet.txt:878:months or so. It might be made a text-book for detectives to teach them what to </span>... </span></code></pre> In this case, Wosp returned the word "detectives" since it is one error from "detective". A better way to include plurals is to use truncation like with detective#1</code>, since it is more explicit about where the extra character goes.</p> Wosp supports Boolean operations. Boolean operations return all matches for documents that meet certain conditions. The four Boolean operators are OR</code>, AND</code>, NOT</code>, and XOR</code>.</p> OR</code> returns all matches for any term given in all documents. Like all Boolean operations, it takes two arguments. Searchers can chain together multiple OR</code> statements to produce long lists of terms to search all at once.</p> $ wosp "detective#1 OR officer#1" A_Study_in_Scarlet.txt </span> </span>A_Study_in_Scarlet.txt:260:however, with many other officers who were in the same situation as myself, and </span>A_Study_in_Scarlet.txt:816:am the only one in the world. I'm a consulting detective, if you can understand what that is. Here in London we have lots of Government detectives and lots of </span>... </span></code></pre> Searchers can string together many OR</code> operations in a row.</p> $ wosp "detective#1 OR officer#1 OR police OR policem?n OR constable#1" A_Study_in_Scarlet.txt </span> </span>A_Study_in_Scarlet.txt:260:however, with many other officers who were in the same situation as myself, and </span>A_Study_in_Scarlet.txt:487:might start a paper on those lines. Call it the Police News of the Past." </span>... </span></code></pre> AND</code> returns results from files that contain both terms searched for. It is good for limiting the number of files to be searched. NOT</code> returns results from files that contain the first term but not the second. XOR</code> works similarly to OR</code> but only returns documents with one of the terms given in the sequence of XOR</code> operations.</p> Wosp supports OR</code> as the default operator. Default operators are a feature of some previous search systems, but I personally do not use them since I prefer explicit queries. Due to its support for default operators, Wosp treats queries like (first second third)</code> as (first OR second OR third)</code>. It does not raise an error for a missing operator. It is possible to change the default operator but that requires re-compiling Wosp at the moment.</p> Wosp supports using parentheses to nest operations. Parentheses are on the highest precedence level, so they are evaluated first, just like in arithmetic operations. Parentheses allow for searchers to create expressive and precise queries that join many different ideas into a single query. This feature saves advanced searchers time by preventing the needless repetition of many nearly identical queries.</p> Wosp supports many different proximity operations. Proximity search</a> is a search technique that allows searchers to find terms that are close to each other in the text. Closeness typically implies that the terms are related, so proximity search ensures that searchers find relevant terms in the same context rather than in different parts of a long document. In that sense, proximity search works like a more limiting form of the Boolean AND</code>.</p> The most basic form of proximity search is phrase search in quotes, which is typically the only form of proximity search most search engines support today.</p> $ wosp "'police constable'" A_Study_in_Scarlet.txt </span> </span>A_Study_in_Scarlet.txt:1070:and against this wall was leaning a stalwart police constable, surrounded by a </span></code></pre> This query is the same as police ADJ constable</code>. ADJ</code> is the adjacency operator. It finds neighboring words in the order given. One advantage of using ADJ</code> over quotes is that queries can support Boolean operations as needed. For example, using parentheses to group terms, a searcher can search for police ADJ (constable#1 OR officer#1 OR detective#1 OR m?n)</code> to broaden their search without having to type police</code> multiple times.</p> The other proximity operators do not have a given order like ADJ</code>. Instead, they search in both directions within a given series of language elements. NEAR</code> searches neighboring words, AMONG</code> searches in the same clause, WITH</code> searches in the same sentence, ALONG</code> searches in the same line, and SAME</code> searches in the same paragraph.</p> $ wosp "detective#1 WITH (case#1 OR evidence)" A_Study_in_Scarlet.txt </span> </span>A_Study_in_Scarlet.txt:1360:thing which may help you in the case," he continued, turning to the two detectives. "There has been murder done, and the murderer was a man. He was </span>A_Study_in_Scarlet.txt:2217:"When Mrs. Charpentier paused," the detective continued, " I saw that the whole case hung upon one point. Fixing her with my eye in a way which I always found </span>... </span></code></pre> Like the search option operators, all proximity operators support added numbers at the end, and those numbers specify how many neighboring language elements to search. The default is one language element. The following example searches for several terms to within 5 words of each other:</p> $ wosp "(private OR police OR government) NEAR5 (constable#1 OR officer#1 OR detective#1)" A_Study_in_Scarlet.txt </span> </span>A_Study_in_Scarlet.txt:817:what that is. Here in London we have lots of Government detectives and lots of private ones. When these fellows are at fault they come to me, and I manage to </span>A_Study_in_Scarlet.txt:1070:and against this wall was leaning a stalwart police constable, surrounded by a </span>A_Study_in_Scarlet.txt:2006:"It's the Baker Street division of the detective police force," said my </span>... </span></code></pre> Some systems support proximity search, but only between neighboring words. In these systems, it is impossible to nest proximity searches using parentheses. Wosp allows nested proximity search without limitation.</p> For example, searchers can search for proximity matches in proximity to other terms or other proximity matches.</p> $ wosp "night#1 WITH (bod#3 NEAR4 victim#1)" A_Study_in_Scarlet.txt </span> </span>A_Study_in_Scarlet.txt:2264:killed him without leaving any mark. The night was so wet that no one was about, so Charpentier dragged the body of his victim into the empty house. As </span></code></pre> Wosp's query language is designed to be expressive like this. It is designed for advanced users who want power and precision.</p> Wosp also supports negated proximity operators (NOT WITH</code>, etc.). These are useful on occasion but I often avoid them unless I have a compelling reason to use them.</p> How Wosp works: inverted indices (technical discussion)</h2> On a high level, Wosp performs the following steps:</p> Read all documents into doubly-linked lists where each list element is a word.</p> </li> Create a full inverted index</a>.</p> </li> Lex a query, parse it using a recursive descent parser</a>, and evaluate it using the full inverted index.</p> </li> Output the results of the query.</p> </li> </ol> The first step is the easiest to understand. All words in the doubly-linked list then have known neighbors and positions in each document, as depicted below.</p> A document represented as a doubly-linked list of words with given positions.</figcaption> </figure> The second step requires more explanation. An inverted index is an index for every word used in a series of documents, akin to the index in the back of books. Given a particular word, the index allows searchers to find every instance of that word. Once a searcher has built an inverted index, it is trivial to find any given word. The index especially speeds up queries that require finding many different words, since it stores all possible word locations. With an inverted index, the rest of the program only needs to perform various operations on any searched terms. Wosp performs these operations using an interpreter that turns search queries into a list of operations to perform in a tree of operations that ends with wildcard searches, which just look up terms in the inverted index. The search option commands modify how wildcard search occurs.</p> The doubly-list list of all words means that every word has a specified position in each document. Building the inverted index then is just a matter of going through each document again and noting the location of each word. Wosp does this using a trie data structure, as depicted below.</p> An inverted index going from a given word to all of its positions in a document.</figcaption> </figure> A trie is a special type of tree data structure where each node contains a letter of a word and all edges lead to the next possible letter. For example, the word "works" starts at "w", continues does the edge for "o", then "r", then takes the edge "k" instead of "d", then "s", and then finally arrives at a list of all positions of all words "works" in all documents. Wosp does not implement these positions as integers but instead as pointers, but the same principle applies.</p> There are certainly more efficient ways to create an inverted index, but I stuck with a trie due to its simplicity.</p> Wosp does not support stop words</a>, so all words are indexed (including extremely common words). I made this decision to keep Wosp lean.</p> Indexed search is entirely different than the sequential search methods used by programs like grep. On a high level, grep looks for a pattern in each line of text in sequence (I realize that some implementations</a> do not implement this idea literally). Sequential search does not require an overarching data structure for organizing that search. Both approaches have their advantages and disadvantages. The primary advantage of sequential search methods is speed, since it may be possible to read each document once and return results while reading the document. It is possible to implement proximity search</a> using sequential search methods, but it is easier to implement nested proximity search using an indexed search method. This observation is easy to see once you realize that nested queries may require multiple passes over the same document anyway, which defeats the purpose of a sequential search algorithm.</p> How Wosp works: query language (technical discussion)</h2> Previous work</a> has divided query languages into two classes: languages that are powerful for advanced searchers but difficult for most searchers, and languages that are weak for advanced searchers but easy for most searchers. I designed Wosp for advanced users who know what they want to search. To that end, Wosp queries do exactly what the user requests --- no more and no less. Wosp does not engage in any query expansion</a>, so it does not find any plurals or synonyms, does not perform any stemming (unless explicitly requested using trailing truncation), and does not correct any spelling errors (unless explicitly requested using a higher edit distance). This design choice keeps Wosp lean and grants searchers more power.</p> I developed the query language's order of operations (precedence) slowly. I initially intended to follow previous efforts like BRS/Search and IBM STAIRS, but despite extensive searching on my part, I never found any formal documentation for the order of operations for the query language of those programs. I did find three references that provided incomplete information about the order of operations of these systems, but the information was contradictory. For example, The USPTO's Patent Public Search</a> notes that the the precedence of proximity operators goes from the largest language elements (lowest precedence) to the smallest language elements (highest precedence), but it does not comment on the precedence of any Boolean operators. I found two other third-party documents that list the order of operations for IBM STAIRS (Gagjokov et al. 1976</a> and Salton and McGill 1983</a>), but their reported Boolean operator precedence contradicts each other.</p> In the end, I implemented the same order of operations for the Boolean operators as for logical connectives</a> in mathematics. I credit my mother for recommending this. This choice means that Wosp may not behave the same as previous systems, but users at least know how it does behave</em> given its explicit order of operations.</p> The operator precedence levels are given below.</p> Atoms: wildcards, parentheses (nested queries).</p> </li> Search options: ICASE</code>, LCASE</code>, SCASE</code>, TCASE</code>, UCASE</code>.</p> </li> Adjacency operations: ADJ</code>, NOT ADJ</code>, phrases (quotes).</p> </li> Word proximity operations: NEAR</code>, NOT NEAR</code>.</p> </li> Clause proximity operations: AMONG</code>, NOT AMONG</code>.</p> </li> Sentence proximity operations: WITH</code>, NOT WITH</code>, ALONG</code>, NOT ALONG</code>.</p> </li> Paragraph proximity operations: SAME</code>, NOT SAME</code>.</p> </li> Boolean negation: NOT</code>.</p> </li> Boolean conjunction: AND</code>.</p> </li> Boolean disjunction: OR</code>, XOR</code>.</p> </li> </ol> Future work</h2> As much as I want to keep Wosp clean and minimal in its implementation, there are still many features that I want to include.</p> The most immediate features that I want to add:</p> Different character encodings.</em> Wosp currently only supports ASCII well. Searchers can search for words with international characters, and use them with Boolean and proximity operations, but wildcards do not work properly with international characters. Eventually Wosp needs to support international characters properly from the ground up.</p> </li> Command-line options.</em> I have endeavored to put as many of the options into the query language itself rather than into command line arguments, but it should also be possible to control several options at the command line too. I have preferred extending the query language because this provides much more expressiveness overall, since command line options may be limited to global options.</p> </li> Highlighted keywords.</em> At the moment, Wosp does not highlight keywords, but it should. The match data structures already contain all of the information for highlighting.</p> </li> Input options.</em> Wosp must include options for how to split words and treat punctuation. At the moment, Wosp defines words as anything between whitespace, but some systems support additional ways to define word boundaries. For example, I should add an option to determine whether hyphenated words count as one word or two.</p> </li> Output options.</em> I have included some additional output options already, but these are hardcoded and require re-compilation to enable at the moment. That is obviously quite limiting. At the moment, Wosp does not support ranking the results in any form at all. It should be possible to specify how to rank the results, especially since this grants the user far more control than what many web search engines grant at the moment.</p> </li> </ul> Other features that I may add eventually:</p> Additional operations.</em> I have some ideas for additional operations, but I am wary of adding any that do not have a particular use case. Wosp already supports page numbers but lacks any operator for page-based proximity search. This would be trivial to add if needed. I have considered UPON</code> as the syntax for this.</p> </li> Field support.</em> Fields are information like titles, abstracts, the list of authors, and other information about a document. Wosp already divides each document up into individual fields, but it lacks any facility to select individual fields. At the moment, the only field loaded is the text itself. One issue is that text files typically do not have any additional fields (other than the filename, file extension, and other file properties), but PDF files or Markdown files might have some, though I am debating whether direct support for PDF files is in the scope of the program or not.</p> </li> Interactive REPL.</em> Given all of the overhead involved in loading documents, I would appreciate the ability to perform multiple searches without having to re-create the inverted index, especially to refine the results. A REPL</a> allows that. A REPL should also introduce the ability to modify previous searches using variables, like what the USPTO's Patent Public Search</a> allows with L numbers</a>.</p> </li> </ul> Discussion elsewhere</h2> Show HN</a> thread.</p> </li> Reddit thread</a> on r/commandline</code>.</p> </li> </ul>



Research
2025-04-09T00:00:00+00:00
Links</h2>

Google Scholar</a></li>
ORCiD</a></li>
</ul>
Peer-reviewed journal articles</h2>


Andrew Trettel (2026).  "Interface-fitted coordinate systems for
two-dimensional imploding shell problems".  In preparation for submission to a
peer-reviewed journal.  LA-UR-25-29021.</p>
</li>

Andrew Trettel and Johan Larsson (2016).  "Mean velocity scaling for
compressible wall turbulence with heat
transfer"</a>.
Physics of Fluids 28.2, DOI:
10.1063/1.4942022</a>.</p>
</li>

Ponnuthurai Gokulakrishnan, Richard Joklik, Darin Viehe, Andrew Trettel,
Esteban Gonzalez-Juez, and Michael Klassen (2014).  "Optimization of Reduced
Kinetic Models for Reactive Flow
Simulations"</a>.
Journal of Engineering for Gas Turbines and Power 136.1. ISSN: 0742-4795. DOI:
10.1115/1.4025265</a>.</p>
</li>
</ol>
Patents</h2>

Ponnuthurai Gokulakrishnan, Darin Viehe, Andrew Trettel, and Michael S.
Klassen (2024). Computationally Efficient Reduced Kinetics Methodologies For
Jet Fuels. English. US 11914930
B1</a>.</li>
</ol>
Other publications</h2>


Andrew Trettel (June 2019). "Transformations for variable-property
turbulent boundary layers"</a>.
PhD dissertation. University of California, Los Angeles.</p>
</li>

Samet Demircan and Andrew Trettel (2016).  "Direct Numerical Simulations of
Multi-Species Variable-Density
Turbulence"</a>.
Variable-Density Turbulence”. 2016 Final Reports from the Los Alamos National
Laboratory Computational Physics Student Summer Workshop. Los Alamos National
Laboratory, pp. 125–132.  LA-UR-16-27258.
10.2172/1327993</a></p>
</li>

Andrew Trettel (May 2015).  "Velocity transformation for compressible wall
turbulence with heat transfer"</a> MS thesis.
University of Maryland, College Park. DOI:
10.13016/M2F33V</a>.</p>
</li>
</ol>


Welcome to my website
2024-09-07T00:00:00+00:00
I created a website when I was in high school, but abandoned it after a few
years.  In the intervening years, I realized that I need some presence on the
Internet, especially to serve as a place to contact me or find out about work
that I have done.  I do not use social media in general, so this website is a
better place to access information about me or work that I have done.</p>
Blog</h2>
Previously I would have balked at the idea of writing a blog.  I did not think
that I had anything to write that somebody else did not already write better.
Nowadays, I realize that there are many subjects that I can cover that others
have overlooked or did not know about.  I intend to cover various overlooked
subjects or anything I think that I can provide new insight into on my own
schedule when time permits.</p>
Website design</h2>
All design is about tradeoffs.  I wanted a website that I did not need to spend
much time maintaining, focused on the content, but nonetheless is visually
distinctive.  In "good, fast, cheap, pick two", I picked "good" and "cheap".</p>
I tried out Zola</a> and Hugo</a> as
static site generators.  Both compile to a single binary, which was my only
requirement.  In the end, I used Zola because it seems to have a more gradual
learning curve.  It actually took me a surprising amount of time to compile
both since I had to manually install Rust and Go compilers due to both Zola and
Hugo requiring quite recent versions of Rust and Go respectively.</p>
I created my own theme in Zola since I wanted a visually distinctive website.
That said, I adopted a minimalist process to save time and reduce the work I
need to maintain the site.  I consider minimalism to be a process rather than
an outcome.  My goal was to reduce the number of design choices that I need to
make while still achieving my design goals.  Unlike other minimalist websites,
I wanted some visual elements beyond text and whitespace to create a visually
distinct website, so I included a few photos that I took over the years for
additional distinction.</p>


Contact
2024-08-10T00:00:00+00:00
See the image above for how to contact me.  This image may change from time to
time.</p>
I am on the web in a few other places too:</p>

Bluesky</a></li>
GitHub</a></li>
Hacker News</a></li>
Internet Archive</a></li>
Reddit</a></li>
StackExchange</a></li>
Twitter</a></li>
</ul>