The Elegance of the ASCII Table

How to say you don't know what Unicode is for without saying it.

1, 2, 4, 5, 6, and, unfortunately, 8, all fall under "ability to encode written text from all human languages". And that includes historical. Some of the issues (5 & 6) are due semantic difference even if the resulting glyph looks the same. Unfortunately you can't expect programmers to understand pesky little thing like languages having different writing, so you end up with normalisation to handle the fact that one system sent "a + ogonek accent" and another (properly) sent "a with ogonek" (these print the same but are semantically different!), and now you need to figure out normalisation in order to be able to compare strings.

7. just like 8 are down to proposal of specific new forms of writing to add to Unicode. Elvish had one since 1997 but only now got a tentative "we will talk about it". Klingon, which is IIRC more complete language including native speakers (...weird things happened sometimes) does not have outside of private use area.

Emojis were added because they were used with incompatible encodings first, even before unicode happened, and without including something like SIXEL into unicode they were unrepresentable (and with SIXEL would lose semantic information)

As someone that whose native language isn't representable purely by ASCII, I celebrate it. Plus the first 128 codepoints are same as ASCII in UT-8.

Is Unicode kind of messy? Sure, but that's just natural consequences of writing systems being messy. Every point you made was for a sensible reason that is in a scope of Unicode mission (representing all text in all writing systems).

I can't wait for when the majority of Unicode codepoints/glyphs are emojis that are no longer fashionable! That'll be a really weird relic of history, later.

All languages are made up. For that matter, all glyphs are made up, too.

covered with ugly graffiti and ramshackle addons

Unfortunately there is plently of precendent for this ramshacklism. Like ACK/NAK - those are protocol signals, not characters! ENQ? What even is Shift In/Shift Out (SI/SO)? Then the database characters toward the end there FS, RS, GS, US.

backwards running text (hey, why not add spiral running text?)

You jest, but you do have cursor positioning ANSI sequences which are designed to let text draw anywhere on your screen. And make it blink! You also don't find it weird to have a destructive "clear-screen" sequence?

glyphs defined by multiple code points

I wonder when they started putting the slash across the 0 to differentiate from the O.

you vote for my made-up emoticon, and I'll vote for yours!

I mean you do have the private Unicode range where you can actually do that. But before that, SIXEL graphics.

2. backwards running text (hey, why not add spiral running text?)

Unicode encodes code points in logical order rather than visual order: the order in which text is supposed to be collated and spoken rather than the visual order.

One tricky issue is when both directions exist in the same text. Unicode can encode nesting of text in one direction within another. For example, text consisting of an English word and a Hebrew word can be encoded as either the English embedded in Hebrew or the Hebrew embedded in English: both would render the same but collate differently.

Is there a better way?

What would be the alternative? I think Unicode is pretty great.

You can pretty easily imagine a world where we had a bunch of different encodings with none being dominant.

Unicode is quite elegant in its encoding too. If you're going to criticize it for its content, maybe start with talking about how ASCII also has invisible characters and those that people rarely use.

Hey at least we got the astral planes. https://justine.lol/dox/unicode.txt

9. color variants

10. code points where the appropriate glyph depends on the language (CJK unification)

Language itself is a pile of ugly graffiti and ramshackle addons. It would be weird if Unicode didn't reflect this.

They're all made-up languages, some were just made-up a little bit more transparently.

Most of this is pretty useful for reproducing a wide gamut of human language. It gets completely fucked when it comes to fonts with png's embedded in svg's and other INSANE matryoshka doll nesting of bitmap/vector rendering technologies.

I also half hate emoji as it pollutes human writable text with bitmaps that are difficult to reproduce by hand on paper with a writing instrument - it's not text. I say half hate as it allows us a standard set of icons to use that can be easily rendered in-line with text or on their own.

For me it's how they inconsistently, backwards-incompatibly, make some existing characters outside of the emoji-plane (and especially when in technical/mathematical blocks) render colored by default, rather than keep everything colored related in the emoji plane (making copies if needed rather than affecting old character, the semantics are very different anyway), e.g. https://imgur.com/a/Ugi7K1i and https://imgur.com/a/UMppZHG

That's because the intended purpose is either useless (for machine control characters) or useless and logically impossible (for delimiters).

What do you do if you have a record that includes a record separator character? Given that you have this problem anyway, why do you want a character dedicated to achieving the same thing that a comma achieves?

I did some ETL work that used the ASCII delimiter characters. It was very enjoyable. I didn't have to worry about escaping or parsing escaped strings. The control codes were guaranteed to be illegal in input. It was refreshing.

Ah, Deborah␞ Records. Little Debbie Records, we call her.

That's my thought as well... I remember using them pre-xhr web in order to send data from the server to JS, which I could then split up pretty easily on the client side. I still don't know why we are so tethered to CSV.

As long as your data is not binary, so does not contain record separators itself, this would be a thousand times better than CSV (because text data _does_ often contain commas and double quotes).

The only thing you'd need is editors to support some way of entering and displaying the RS, and CTRL+^ is a bit of a kludge as it ends up CTRL+SHIFT+6.

Of course, if a record itself can contain RS for subrecords, things become more complicated. I guess you could use `\^`.

I recall working on a PICK D3 system, which was a "multivalue" database. Each field could have multiple values, those values could have sub values, and a third level beyond that.

Values were separated with char(254), subvalues were separated with char(253), and the third level were char(252) separated.

It was... unique, but worked. And to be fair, PICK originated in the 60's, so this method probably evolved in parallel to the ASCII table!

You also have to keep in mind the "interface" for 1962-1968. The printer teletype machine.

The "control codes" were to "control" the printhead. So "carriage return" meant move the "print carriage" back to the left margin. "New line" meant move the paper platen one line height of rotation to move the paper to the next line. In that context, "back space" was "move print head one space left" (rather more like a "reverse space"). The article does mention that there was some debate about whether space should be considered "printable", but if you consider a mechanical printer, as the head is moving to the right and banging out characters onto the paper, the spaces between words do, sort of, look like "printables" (of a sort, a "print nothing" character as it were).

Tab's being control characters then make a bit more sense, in that they cause the printhead to jump some fixed distance to the right.

The article stated why DEL is where it is (all ones) -- so that for punched paper tape, one could get a punch-out of every position, which was then interpreted as "nothing here" by the tape reading machine.

As for typewriters, no, none had a "black box" blot out key. Correction (for typewriters without built in correction tape) was one of: retype the page, apply an eraser (and hopefully not damage the paper surface too much) then retype character and continue, or apply correction fluid (white-out) and retype character and continue.

For those typewriters with built in correction tape options (at least some IBM Selectric models, possibly more) the typewriter would retype the character using the "white-out" ribbon, then retype the replacement character using the normal "typewriting" ribbon.

del is not a printing character. it's a control character. if you run a paper tape full of del characters through a teletype it does not print anything. it has to have that bit pattern, even though it greatly complicates the mechanics of the teletype (which has to do all the digital logic with cams and levers) because that way it can be punched over any character on the paper tape to delete it

a figure caption in this page says 'This is a historical throwback to paper tape, where the keyboard would punch some permutation of seven holes to represent the ones and zeros of each character. You can’t delete holes once they’ve been punched, so the only way to mark a character as invalid was to rewind the tape and punch out all the holes in that position: i.e. all 1s.' which is mostly correct, except that it wasn't a historical throwback; paper tape was perhaps the most important medium for ascii not just in 01963 and 01967 but probably in 01973, maybe even in 01977. teletype owners today are still using paper tape that was manufactured during the vietnam war, where it was used in unprecedented volume for routing teletype messages by hand

the dominant early pc operating system, cp/m (if it's not overly grandiose to call it an 'operating system') had system calls for reading and writing the console, the disk, and the paper tape punch and reader. when i hooked up a modem to my cp/m system to call bbses, i hooked it up as the punch and reader

Logically space maps to a character people use with pen and paper unlike tab

Space is what is represented in the output, i.e. in one cell of the terminal grid, whereas control characters like Tab and CR/LF don’t map onto such an output representation. If you want to represent the printed contents of each “grid cell” of a printout or a textmode screen buffer, you don’t need the control characters, only the printable characters. The printable characters are what you’d need in a screen font.

Space doesn't just move the cursor on a display; it will obliterate a character cell with a space glyph.

When a display terminal has nondestructive backspace (backspace character doesn't erase), it can be software emulated with BS-SPACE-BS.

At your Linux terminal, you can do "stty echoe" (echo erase) to turn this on (affecting the echoing of backspace characters that are input, not all backspace characters).

Dial-up BBSes had this as a configurable setting also.

While DEL didn't stamp a black square on typewriters, it sometimes did so (or something similar, like diagonal stripes) in various digital character sets. ISO 2047[0] established the graphical representations for the control characters of the 7-bit coded character set in 1975, maily for debugging reasons. This graphical representation for DEL was used by Apple IIGS, TRS-80 and even Amiga!

[0]: https://en.m.wikipedia.org/wiki/ISO_2047

strange: on MacOS 14.5 I get output for `man ascii` but `ascii` goes "command not found"

Damn, thanks!

Why the hell did I never try this? Maybe because typing ascii table into my favorite search engine and clicking one of the first links was fast enough

The reason I know this is because in 2004 I was squatting in an apartment with no TV and no internet. So each day after work I would go home and just read manpages for fun.

Ended up learning ipfw through the firewall manpage on FreeBSD, and using my skills to setup and manage an IPFW at work.

It's amazing how much you get done with no TV and no internet. Also played a lot of nethack.

Or even simpler use the ascii command, when installed: https://packages.debian.org/bookworm/ascii

Works on mac

not sure about other *nixes

Should be available on any UNIX, it was added to V7 UNIX back in the 1970s: https://github.com/dspinellis/unix-history-repo/blob/Researc...

Even before that, it existed as a standalone text file https://github.com/dspinellis/unix-history-repo/blob/8cf2a84... This still exists on many systems -- for instance as /usr/share/misc/ascii on MacOS

Similar in FreeBSD. It has octal, hex, decimal, and binary ASCII tables, along with the full names of the control characters.

For everyone else, UTF is a blessing.

Except people who want to use Japanese and not have it render weirdly, something that was easy in any internationalised software that used the traditional codepage system, but is practically impossible in Unicode-based software.

Which, given the people who designed this and the time they were designing for, was most of them (and most of their audience). Don't confuse "this old standard doesn't adequately cover all cases today" with "this old standard sucked at the time".

Here, take these: ae ue oe

No way!

No amount of extra characters was going to address what Unicode did.

ASCII was not a mistake at all. Adopting it unified what was surely going to be a real mess.

At the time it made sense, and the control functions were needed. Still are.

It's one of the greatest triumphs of early computing. Not only did it harmonize text representation and transmission in a backwards compatible manner; the fact that they deliberately kept it 7 bit for so long also helped for developing a sane set of other language character sets (ISO-8859), and paved the way for a smooth transition to Unicode (UTF-8) - which is now the dominant encoding worldwide.

Yeah you were not around when a kb of memory took up half your room. Looking back it doesn't make sense but at the time a byte was what-ever you wanted it to be. Considering number of characters in English language is 26, it is reasonable for a byte to be 5 bits, giving you a total of 32 possible states. Which leaves you with 6 values which could be used as control characters. how-ever lets not forget there are 7,164 other languages of the world, and they all have their own unique way of doing things.

Oh yeah, lets not forget that at the time you had other nationalistic countries/territories/people with their own superior technology all vying for the top position, all well trying to out do each other. Then you also had manipulative monopolies/trade embargo's and wars.

It isn't perfect but people aren't perfect.

Yes, that’s very intentional and just masking (or setting) the bit is the intended way to do case-insensitive comparison of the letter range in ASCII (eg. stricmp in C), or to transform text to lower or upper case (tolower, toupper).

But what’s more, ever wondered whence the control (Ctrl) key presses like Ctrl-H to backspace, or Ctrl-M for carriage return? Well, inspecting the ASCII chart it becomes evident: the Ctrl key simply masks bit 6 (0x40), turning a letter into its respective control character!

This can be made to work for ASCII and EBCDIC simultaneously for extra esoterica points:

  switch (my_char | 'A' ^ 'a') {
  case 'A' | 'a': /* ... */ break;
  /* ... */
  }

It was at some point looking at a chart like that where it also dawned on me where the control codes like ^D, ^H, ^[ etc came from

The third and fourth columns of the table are only a single bit apart from each other. If you mentally swap the first two columns, you get a Gray code ordering of the most significant bits, which is pretty close to what you're looking for.

you mean ?

    ascii

Unfortunately unfruitful.

Fortunately unfruitful, since if it had gained adoption, there'd be a mix of three different line endings (and combinations thereof) in widespread use, instead of two.

A separate control character was not needed to indicate where your Hollerith string ended: it ended at the end of the Hollerith string. If you wanted to render a Hollerith string onto print media, you'd often want to feed the line and then return the carriage before printing the next Hollerith string. Of course, that wasn't strictly necessary if you were using a line printer, which would just print the line and advance.

The filesystems I used had 5 kinds of file: random-access, sequential, ISAM, Fortran-carriage-control and carriage-return-carriage-control. The only people who used the latter were the eggheads that used that new-fangled C programming language brought over from Bell Lab's experimental Unix system.

You're probably just looking for the record separator (036). If you are storing multiple text records and a block of memory, that would be the ideal ASCII code to separate them.

It's really not.

In base-2 machines, the letters are mixed with punctuation, which is pretty horrible design which makes simple things complex, and does not actually bring anything new to the table.

In BCD machines it is slightly better, except letters aren't contiguous either - row 0 is bad, but it's the extra space between R and S which is really ugly. And it's unusable with BCD operations anyway, as high nibble values are used extensively.

Naive sorting simply does not work... lowercase before uppercase, punctuation in the middle of the alphabet, numbers after letters.

I see no elegance there, it's like the worst example of legacy code.

Its shit if you don't routinely speak or write English. On those grounds, I'll decry it as not only shit but purposely shit.

OK a bit over the top ... the designers of EBCSDIC had a rather tight set of constraints to deal with, none of which included: "be inclusive". Again, if I really had to be charitable (I looked after a System/36, back in the day), the hardware was rather shit too, sorry ... constrained. Yes constrained. Why should six inch fans fire up reliably after a few years of use and not need a poke after an IPL? No real dust snags and I carefully sprayed some WD40 on the one that I could get at. I have modern Dells and HPs in horrid environments that do better with shitty plastic fans.

EBCDIC is not elegant at all unless excluding non English characters in an encoding system is your idea of elegant.

According to this: https://en.wikipedia.org/wiki/EBCDIC it expended loads of effort with dealing with control eg: "SM/SW" instead of language.

ASCII and EBCDIC and that basically say: fuck you foreigners!

We now have hardware that is apparently capable of messianic feats. Let's do the entirety of humanity some justice and really do something elegant. It won't involve EBCDIC.

I never realized my first computer used ascii directly for the shifted number keys.

https://en.wikipedia.org/wiki/TRS-80_Color_Computer#/media/F...

Bespoke hardware for text handling isn't a thing these days but would have been in the 60's and 70's. A table layout that can be easily decoded in hardware simplifies the necessary circuitry for responding to control characters or converting binary numbers to/from decimal when the microprocessor hadn't been invented yet.

It was originally implemented in actual hardware (rods and bars). Just look inside a teletype, like a KSR-23 (pre ascii, but similar)

They might be talking about using escape sequences to map additional codepoints into ASCII. It was designed to be extensible. See: https://web.archive.org/web/20150810075144/http://bobbemer.c...

I thought the point of that was that a single bitflip makes an uppercase lower, or vice versa...

The background is white on my machine, are you using some kind of extension to force "dark mode"?

You mean 0D and 0A, or 13 and 10

He fixed it.

Thanks for posting that.

People tend to overlook that the technologies we use today have a much older history.