Basics Catch-Up · One Character, Five Byte Strings: Charsets Number, Encodings Store
Part 23 said a tokenizer turns text into integers, but not that the first step turns text into bytes. A charset only says which number a character is; an encoding only says which bytes that number becomes — and once they are separated, UTF-8's four templates can be worked out by hand, half a Hanzi has a definite reason for showing as �, and the 256 entries at the bottom of a tokenizer's vocabulary stop being a convention and become a mechanical fact.