By: dreev, implemented by Jake Coble
Spec level:
Repo: github.com/beeminder/transgreption
Changelog
2026-09-25: Migrate to hosting on Render.com 2024-05-15: Dreev & GPT4o get transgreption deployed at transgreption.replit.app 2020-09-24: Spec pretty much written, Jake Coble implemented it around this time
Problem: You have a bunch of google docs or wikis or articles and you want to search for something across the full text of all of them.
Transgreption is a webservice that takes a source document and outputs html that concatenates the contents of all the URLs mentioned in the source document.
Say you have a list of documents that are all accessible, with no login, via URLs. First, you, the user, make a metadoc that lists all the URLs of those documents. This document is such a metadoc!
If you want to search across your documents — everything linked to in the metadoc — you go to “https://transgreption.onrender.com?q=doc.bmndr.com/transgreption” (where the part after the “q=” is the URL of your metadoc). There you see a huge concatenation of the text of all your documents. We call that the megadoc. You can control-f (aka grep) it and figure out which document to go to when you can only remember some snippet like “bounded rationality” or whatever.
It has a nice dynamic table of contents pane on the left like, for example, api.beeminder.com has, so as you grep the massive concatenation you see immediately which document the search string is in and you can click in the left pane to go straight to it. In other words, the megadoc links back to all the original documents in your collection, just like the metadoc does. Think of the megadoc as a version of the metadoc with all links expanded in-place into their full contents. (Not counting the text of the metadoc though. If you want the metadoc’s text in the megadoc, just have the metadoc include its own URL in itself!)
It’s a completely general login-free webservice immediately useful to anyone with a collection of documents that all have public or quasi-public URLs!
transgreption.com?ur_url=example.com/metadoc”(If the ur-URL is omitted, it’s fine to prompt for it.)
Alright, let’s have some self-dogfood! Here are examples of URLs that are viewable in this metadoc’s megadoc:
raw=1 so Dropbox just goes straight to the file instead of showing you its chrome.
(Transgreption doesn’t
and shouldn’t know that those are the same document — the URLs are different! — so the megadoc includes the previous document twice.
In future work we may canonicalize URLs.)
https://www.statuscake.com/
By having the above list in this document, this (what you’re reading right here) is a metadoc. And of course the tool doesn’t try to be clever. It’s oblivious to the numbered list. Every other URL in this document, like the one in the previous sentence and the ones in the intro, will also be included. Transgreption just grabs everything in the metadoc that looks like a URL using a regex. This one works for all the examples I’ve thought of so far:
/https?:\/\/[^\s\)\>\"]*/g
(There’s no need to be fancy with the regex, like noticing that http://beemindercom/foo isn’t valid.
It generates an error when we attempt to fetch it and we just show that error in the megadoc.)
Again, no logins/credentials are involved in any of this, just GET requests on public or quasi-public URLs. And the whole tool has no UI other than the query string in its URL specifying the metadoc. The output is the static html of the megadoc concatenation. Input metadoc, output megadoc.
There’s as little of it as possible, just nice big obvious dividers between the constituent documents. And a big “TRANSGREPTION” header at the top. And the sidebar.
What if a GET request on one of the URLs fails with a 404 or some other error? We show that in a big red font at the top of the megadoc and again in the huge concatenation where the failed document was meant to be included. For example:
ERROR 500 fetching example.com/foo -- Can't connect to example.com:80 (Operation timed out)
Transgreption follows them. Whatever a human with a browser sees when they click on a URL, that’s what we see in the megadoc. Possibly an uglier version of it but with the same text.
What about a URL to an image or other binary data? See #9 and #10 in the list above.
What happens if this metadoc includes the link to the megadoc, i.e., the transgreption-ing of itself? Like “https://transgreption.onrender.com?q=doc.bmndr.com/transgreption” or wherever this thing lives. It only goes one level deep so the universe doesn’t explode or anything. But it makes sense to suppress it as a special case so that the megadoc doesn’t include two copies of everything in the huge concatenation.
I have a collection of URLs and I need to search the contents of all of them. It’s like having a personal search engine — but no indexed searching, just grepping — for a tiny, specific subset of the internet!
If the megadoc contains unlisted / quasi-public URLs, which it often will, maybe it’s bad to leak those. Imagine your metadoc links to both a secret, unguessable URL and also links to a site controlled by bad guys. If you click the link to the bad guys then your browser will tell them the URL you came from and they can now learn your secret URLs. Etherpad does a redirect thing to mask the http-referrer. Should we do the same? In the meantime it might be safer to not have the links from the megadoc to the individual docs be links at all. They can just be the plain text of the URLs that you have to copy/paste.
Assuming we’ve figured out the previous open question, could the links in the megadoc to the individual docs have little “this link will open in a new tab” icons next to them?
(And use target="_blank" to make that true?)
See the links in the References section of any Wikipedia article for an example of such an icon
(though in Wikipedia’s case it just means that it’s an external link, not that it will open in a new tab).