Sentry is a software service designed to facilitate automated error reporting from production systems, with rich information about the circumstances in which they occured. Sentry eventually expanded the product to also provide performance insights.
In 2021 I took part in a 3-month remote internship on Sentry’s Ingest team, which dealt with receiving the error and performance data from users’ deployments, doing the initial processing and filtering of the data.
Software Engineering Intern
I was primarily dealing with the Relay software, which is a Rust web server receiving the incoming data, running in one of two modes. At the time of internship there was a central instance of Relay feeding the data to the processing pipelines, but clients could also run Relay on their own infrastructure to perform the initial preprocessing (including but not limited to PII removal and more generally filtering out uninteresting data) before it reached Sentry’s infrastructure.
I made a few quality of life improvements to make the second use case of Relay less intimidating, more cloud deployment friendly, and more debuggable in complex situations.
But the biggest singular change I made was the move of payload compression out of async thread into the sync thread pool. Previously, this issue made compression into a drastic throughput bottleneck and therefore it was not advisable to keep it enabled. Deploying this alongside a configuration change made a slightly ridiculous dent in one resource monitoring graph (good!) while making neglible impact on other funny monitoring graphs (great!).
As the internship neared end, I was assigned a Sentry issue to debug for a change of pace. My investigation uncovered a very funny behaviour of the ClickHouse database causing one of the database abstractions to misoptimize a query containing a cursor-based pagination, such that instead of returning all items in the pagination, some showed up in multiple pages, while others never showed up at all.
I also took part in the company hackathon, teaming up with Markus and Betty. Inspired by what I’m told is a German word “Holzkurbelmodem”, apparently meaning a hand cranked modem, we built a wooden WiFi router which drastically caps internet speed if you’re not rotating a crank. It can be briefly seen in this office tour video. It was made by putting the insides of a Netgear router with OpenWRT in a cool looking treasure chest, bought along with a bunch of other cool looking components from random Amazon sellers. The crank is hotglued to something clicky we bought in OBI, that’s hotglued to the chest while also conveying the rotation to a rotary encoder. The rotary encoder is read by an ESP8266 board, which communicates the router via USB-serial, and on the router there’s a Go program that controls the Linux kernel’s traffic control subsystem to limit the speed of the upstream internet connection.