🌆👀🤖 How AI “Sees” Images
Machines learn to see without ever looking
I was thinking about how AI “sees” images.
If you open a JPEG file, it looks like this for pages and pages:
°I cÖzûíáÂd†€® Ç á∑≠G —<÷fil`‘∏{ pƒ˝∂1 ¡i° UVW‹ iEH 0 T⁄J}XÄW™Ã˘…´¶"“È√ßCà™ õ¥˝[_ ë¿ ZÜÉ KSÚ:œu˘éꈿ Ü (4Ω¢µfl"Ñ©ïc©XØ ^;~ıLj¶ı@ÿ≠!@pŸöƒæèÖ {√CÈHíÊ∫9* o ˜J®’fiã±4¬É] ô“ıç ‘ªvr ™¯Î7Îr¸_Ò0 kiË∆ÒG »‚ n]ñ€íÏœ›{MÒUAÂ2’ûNå≈û\/3áÖ7“ÍfNπ,¨Í u Êú ÷ZÇs&OÃl≤tJ"Sz3◊=£œØïÕ( ÷ q (‡˜ã €1Ú% ˘∑≥˜,8YQ– ƒ [©lã µÏ<2¿¿Ùw *kÉÉ ÓQVÅoòÄ≠hf NjÈ E %q P(ÑrTej¨8ü‚jÌˇÄËaScÕ◊H4Ô3JÖi∫xˇgT c `0à es§kWx@ñ{|¶%`NÀd- æÚÃÎÜ- :R„ºPw ˬ ‡∏T7zÔòˆ ≤S íâx≠‡Ú¡ÖÊh`L¿, ⁄,i–—PûºÉßH[0JI! .ó˛{Ÿûo/ Tzfls7Eé8úá=eæ `^Å åIÉ≠Ãú‚ ‘ ‘¶∫!»Ô &j˙ ≤¸√fif ˙Ã3fùeÜ Å’‡¢ TÕ3º∂Mcî ∞¸¯\4≠#5“0j [ä∏ÈÈ·QÍ3 ‹¯üö øá≥«_ ⁄Ô8Åî`.)ÇùòãP°ÌB FŒfl∏Sͱ (TÍ8B¨ºê®lß:”»|«V7#B‚ÄÊ#=sÚèúG%øB(+°ø B…IÉ9Un"2¢ rôHZ ‘|Ê ›x· rãBXÂr/Ç3 <ˆ´Û)l· πç„∂åfiõ»;È µÿ x ¡ƒÛ[7ÌzÌ :£À oÌ/¥x ˙Î[À√µ∂¸ ã• ß—fi3Úñ∫” ,€Ï˜à˛™A“Éfipawî∑3 s3CÇo— °ˆÉÂJ LjxÏ˚÷jÔLÄgˆüÇEach pixel’s color and location are encoded in there (I won’t get into lossy compression today…).
A multi-modal LLM can “look” at thousands of pages of this 👆 and “know” that it encodes a dog 🐕 or a house 🏠 or a ninja 🥷. The model has been trained on *trillions* of pages of similar data representing images, and it has extracted the high-level patterns.
While it’s useful for us humans to think of these images in the way that our visual cortex renders them from the photons emitted by screens, for an LLM without a visual cortex 🧠 the image *is* the raw binary code from the decoded JPEG file.
Generative AI also works with this raw material to create new images.
There’s no point at which the model can “stand back and look at the image” like a human would 👁️. It has to create a new image that makes sense and looks good purely as thousands of pages of binary code. That code is then encoded as a JPEG and rendered on a screen by a device so that a human visual cortex can decode the photons and interpret them.
It’s a bit like a composer who has never heard a sound because they were born deaf, but despite this, became so proficient at reading notes on the page that they can write new music based on all the written music they have seen previously, and when musicians play it, it sounds good to their ears.
It’s like Neo looking directly into the falling characters of the Matrix…
🧭 This first appeared in Edition 496 of Liberty’s Highlights. New here? I made a page for that: Start Here.