context window

I really enjoy the concept of “mechanical sympathy.” In my own words, it’s this “spidey sense” for how a system works under the hood that consciously or subconsciously helps you make better decisions or be more effective/efficient when working with that system. One example is having internalized the whole “CPU register access is faster than RAM access is faster than intra-datacenter network access is faster than inter-datacenter network access…” heuristic — if you don’t have a working mental model of this, you are prone to build things that aren’t performant and not understand why. I think this concept largely holds true all the way up the stack.

Something I’ve been thinking a goodly bit about lately his how to build mechanical sympathy myself for LLMs and other forms of “AI”. Similar to how mechanical sympathy for how computers work was (and is) incredibly useful, and then mechanical sympathy for “the cloud” became useful, mechanical sympathy for AI is or will be essential. So far, I’m mostly trying to establish this mechanical sympathy the same way I always have — wandering around the wide corpus of stuff to read and watch and letting it all just marinate. Maybe I should try to write about this.

Some stuff on mechanical sympathy:

• An old post from the AWS Well-Architected Framework
• A reddit post