return to table of content

The Soul of Maintaining a New Machine

ggm
15 replies
1d14h

I'd forgotten how close to printing machines the old photocopiers were. You would basically either have a crap one you could operate yourself, or take your stuff to the printery to have professionals (a subset of librarians I think, or the logical join over librarians and computer operations staff) do it for you. Printing machines had a fleet of maintainers, craft unions who walked off the job if you touched a dial.

They were amazing at doing things which really mattered: shrinking an A0 architectural drawing down but maintaining aspect ratio. Adjusting offsets for the print for binding signatures, so the 1st and 16th page was not too far out because of wrapping around the other 8 pairs of pages. Even just working out how to rotate the pages for N-up printing. But the GUI sucked. I think they called ours "the bindery" because it's main gig was doing PHD from soup to nuts, binding included.

The repair techs had the most amazing flight cases, packed with tools which served one specific purpose.Like, A doohickey to adjust the corona wire, without dismantling the imaging and toner roller, with a tonne of equipment hovering over your head on a gas-lift. Screwdrivers with very very carefully chosen lengths. Torque wrenches. It was high tech meets motor racing meets.. IBM.

I am told they were paid better than many computer techs. The IBM guy was paid IBM scale to fix it on IBMs timescales. the xerox guy did more random shit, with more devices, more often.

They had a very corporate look. that amazing briefcase or six. Suit, tie. Very acceptable.

I know a guy who worked for a paper-folding-and-envelope-stuffing company and it was very similar culturally: can-do, fix anything, but working on giant multi-million dollar machines which were used twice a year to do tax mailouts, and election materials, and the rest of the time rented to the original spam merchants for 10c per thousand mailouts. The secondhand value of these machines were like photocopiers: Really significant. He was brought out of retirement to help take one apart into TEU equivalent chunks to be shipped to Singapore from Brisbane. His retirement gig at one point was repairing Espresso machines, he said it made him feel familiar and useful.

The era which was the end of the typing pool was fascinating. All kinds of arcane roles which only make sense in the absence of email and tiny printers everywhere. Some of those jobs had been there from the days of hand-copying, Dickens-era and before.

Taniwha
12 replies
1d13h

Back in my mainframe days (late 70s) we ran a large mainframe, the only one in the local uni - way slower than your phone, a couple of Mb of core, ran payroll and 30 terminals. We had a dedicated local engineer who had his own onsite office. In the south of NZ we were a really long long way away from his home office in the US, he was expected to be able to fix anything, and mostly he could.

But one time the machine started writing crap on random things - screens, printers, worst of all disks - could take a day or to to recover after it scribbled across the equivalent of the root file system (giant head-per-track coffee table sized platters). The poor engineers couldn't figure it out, it happened so in frequently eventually they flew a guy out from head office in the US - he came with a wooden stick - he ran it down a card cagore in the IO process, nothing happened, he tried the next row bang! it crashed, after we were back up he continued with his wooden stick doing a binary search for the source, eventually he pulled a card and 3 little solder balls fell into his hand, they'd been sitting there loose against IC pins since it was installed

As you point out sometimes it just takes having the correct tool and knowing how to use it

dmd
11 replies
1d6h

way slower than your phone

For the young'uns... I think you're underselling this a bit. A really powerful late 70s mainframe, generously would have been somewhere around 1 to 2 million instructions per second.

Your phone is probably on the order of 5-10 trillion instructions per second. That's a million times faster.

The_Colonel
4 replies
1d4h

Your phone is probably on the order of 5-10 trillion instructions per second. That's a million times faster.

Do you mean from iGPU? The fastest smartphone iGPU I found is MediaTek 9300 which has 2.4 TFLOPS which roughly corresponds to your claim. Cray-1 (1975) had 160 MFLOPS which would make today's high-end smartphone 15 000 times faster than Cray-1.

Jtsummers
2 replies
1d1h

Cray-1 wasn't a mainframe, it was a supercomputer. As such it was designed to push the performance envelope in a way mainframes (of the time, but even today) weren't even trying for. Mainframes were designed for transaction processing and reliability. They were substantially slower than contemporary supercomputers.

The_Colonel
1 replies
1d

That's true, but supercomputers are somewhat conceptually similar to GPUs, so it seems like a better comparison to illustrate technological progress. Comparing mainframes and GPUs is comparing apples and oranges.

zifpanachr23
0 replies
1d

This is a good analogy. If you look at a modern mainframe CPU it becomes pretty clear where the differences lie. Fewer, beefier cores with a lot of focus on cache.

Supercomputers tend to use more conventional cores, but way more of them, and connect them in a large fabric. There's a lot more focus on parallelization and horizontal scaling.

Mainframe overall compute is nowhere near a supercomputer, and you probably shouldn't be running a massive physics simulation on a mainframe, but you may get more consistency and reliability for well defined tasks.

vaxman
0 replies
3h11m

This table will help you, but it stops way back in 2021 at 346,350 MIPS https://en.wikipedia.org/wiki/Instructions_per_second

Looking forward to the Apple A18 Pro reported to be 4GHz x2 and 2GHz x2 +/- thermal conditions.

kragen
2 replies
21h35m

your phone is 5–10 billion instructions per second, a thousand times faster. it's probably a quad-core in-order arm running at 1–2 gigahertz, not 1–2 terahertz

dmd
1 replies
21h33m

Instructions and hertz have not been equivalent for almost a quarter of a century now, though.

kragen
0 replies
21h29m

not equivalent, but close enough in this case; even my laptop only manages less than 2 ipc per core, and the error in your earlier comment is a factor of 1000

vaxman
0 replies
3h26m

I think we need to put it in terms of smart watches now, soon smart rings.

Hope we remaining masters can stop the delusional tech bastards running around convinced they're a new and more powerful Steve Jobs (oblivious that his success came from leveraging microprocessors to put millions of formally educated tech people with families out of work in a disorderly manner by disrupting the market) from further inflating the public stock markets (again) and distorting energy capX over AI hype. AWernerS is clearly losing the battle against them. https://www.youtube.com/watch?v=ixe8Snxu3wo

brk
0 replies
1d5h

True, but 90% of the mainframe instructions weren't tracking, telematics, data exfiltration, and user profiling. So it kinda evens out.

Taniwha
0 replies
16h6m

Yes exactly, core cycle times were ~1uS so around 1Mhz clock for our old B6700

bruce511
1 replies
1d10h

We still have an old printing machine. It came as part of an acquisition and took a truck to move. We rent a small unit in an industrial complex to house it.

Nobody knows how old it is. It predated the folk who came with it (all of whom were close to retirement.)

It's still running (looked after by one of the retirees on a part time basis.) He does print runs a couple times a week. To seem him with an oil can in his hands is to step back in time.

It's more or less neutral profit-wise, but pays his wage and keeps some old customers happy. When he gives up the machine goes too (likely for scrap I guess).

actionfromafar
0 replies
1d8h

Beautiful little story. I love when such things are allowed to exist.

gumby
8 replies
1d3h

This was such important and transformational work and I remember at the time being quite dismissive of it.

I knew Orr’s and Suchman’s work (they worked in a physically adjacent area, but completely different group, though we were all under John Seely Brown and because they were nice people). Thankfully I was grown up enough to be polite, but really I was such a techno-determinist that I figured user problems came from ignorance.*

To be fair, I was not the only one: the insights described in this book draft surprised a lot of people, not just how they improved the copiers but how those two even approached the problem (starting with the sociology of the repair workers). It sure surprised Xerox management. But I’ve heard it said many times that this work led to restructuring the paper path in a way that justified (paid for) everything spent on PARC.

I did grow up of course and now do see my work (machines, chemistry, etc) as a small part of a large social system. A successful company has to base its product plans starting this way.

To choose an example of failure to appreciate the social scope (but not pick on it) the crypto folks spend their time on technology, based on a social model they want to exist rather than the one that currently does. I think it’s a big reason why it’s barely impacted the world in, what, 15 years? Xerox was the same, and it helped them sell a lot of copiers, but didn’t make them as ubiquitous as they could have been. Another example: everybody laughs at Google for launching “products” that go nowhere and are quickly forgotten. We all know it’s because of a screwed-up, internally-focused culture. But sometimes a product succeeds without marketing (e.g. gmail, at the time) because it happened to be matched to the actual, external need. It makes this kind of continuous failure even more damning.

* TBH, 40 years later I have not 100% shed this view — e.g. my attitude towards complaints about git. Maybe this means I’m still a jerk.

kragen
4 replies
22h14m

the crypto folks spend their time on technology, based on a social model they want to exist rather than the one that currently does. I think it’s a big reason why it’s barely impacted the world in, what, 15 years?

assuming you mean cryptocurrencies, i think they kept several million people alive throughout the venezuelan civil war, sustained wikileaks through the visa and mastercard blockade, and have made sci-hub and library genesis financially stable; maybe they haven't impacted the people in your direct vicinity, but they've made a huge impact in places like moldova which don't have functional currency

gumby
3 replies
22h3m

Yes, cryptocurrencies not cryptography. Bad of me to use that contraction for anything but the latter.

Have they really made a significant difference? A bit of searching around resulting in a few articles with anecdotes, and if someone's life really was saved this way of course it made a huge difference to them.

But for some macro perspective the best I could find was the "Crypto Adoption Index" (https://www.chainalysis.com/blog/2023-global-crypto-adoption...), where VE, or MD don't even register (AR barely does). In fact US is listed as having one of the highest levels of adoption, and it's basically invisible here.

Countries with nonfunctional currency systems usually do better by adopting an existing stable currency like USD or EU (or even better someplace economically coupled with them).

But I'm no expert and I'd be happy to be proven wrong. On a financial macro perspective cryptocurrencies don't even exist.

kragen
2 replies
21h42m

pretty much the entire currency black market here in argentina used to run on cryptocurrencies, which is more significant than it sounds given how large a fraction of argentina's employment is illegal (something like a third, last i heard). i think us banks have taken some of that business away from cryptocurrencies in recent years, but every illegal moneychanger still accepts them even if most of their business runs over zelle

as for how countries might do better, while it is interesting to discuss what policies policymakers ought to adopt, argentine (and turkish, venezuelan, etc.) policymakers do not want to adopt the policies they ought to adopt; they want to adopt the policies that serve their political interests. the rest of us are left to figure out how to cope with their terrible policies. and that's the sense in which bitcoin matters; it makes it possible for argentines and venezuelans to do things like save money, leave the country, and send money to their families back home when they're working abroad

on a financial macro perspective, there's currently about 1.1 trillion dollars stored in bitcoin, which is a few days of global gdp. other cryptocurrencies together are something like half a trillion. it's entirely plausible that 80%, 90%, 95%, or 99% of bitcoin's number is owned by keys that were lost in disk drive failures back in the cpu mining epoch, but that's less likely for the other cryptocurrencies. so i don't know that i'd agree that they don't exist on a financial macro perspective. certainly they aren't anywhere near the importance of asset classes like real estate, commercial paper, forex, commodities futures, or stocks, and it's questionable how much real liquidity exists in the market—if satoshi were to start selling off his coins now, how much would it tank the market before he was done?

gumby
1 replies
19h0m

on a financial macro perspective, there's currently about 1.1 trillion dollars stored in bitcoin, which is a few days of global gdp

You need to compare like to like. $2T of cryptocurrencies is about 2% of all the world’s stock exchanges (after the recent fall). Or compare it to total world fungible assets: https://www.visualcapitalist.com/all-of-the-worlds-money-and...

kragen
0 replies
16h54m

that seems like the right kind of comparison, yeah, although it's surprising the world's stock exchanges aren't worth more.† the infographic article is excellent, though i guess now cryptocurrencies would be more like 15 squares instead of three

2% is far from insignificant! argentina's gdp is less than 1% of the world's—as are each of switzerland, taiwan, belgium, israel, south africa, pakistan, new zealand, nigeria, greece, and hungary. if the advent of cryptocurrencies has added a new power to the world stage of the same order of magnitude as taiwan, pakistan, or nigeria, that's a momentous event—particularly since it's doing it by giving tens of millions of the world's poorest people relief from terrible governance

(funding universal public access to history's biggest library despite prohibitions from the world's greatest powers seems significant, too. you've been in india, you know what library access is like outside the rich world, and you're surely aware that sci-hub and libgen have had a huge positive impact even inside the rich world)

in the infographic, the m1 money supply is about four months of world gdp, 35 trillion dollars, and that seems like the most nearly equivalent thing in the list to compare bitcoin to. it's about 3%. if we assume that countries' money supply is close to proportional to their gdp, that would make bitcoin the 8th largest world currency, following the euro, the us dollar, the yuan, the yen, the indian rupee, the pound, and the brazilian real. it would beat the canadian dollar, the russian ruble, and the other 150 or so national currencies, going by https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nomi...

if we use the m2 money supply instead, bitcoin is about 1% of it and drops to about 15th

on the opposite end of the scale, supposedly there are two or three thousand billionaires in the world; bitcoin distribution is widely considered to be highly inequitable, and so it may have added a few dozen or a few hundred to that number. considering the historical impact of certain individual billionaires like andrew carnegie, george soros, john d. rockefeller, and leland stanford, that seems like it might also be important

______

† world gdp is 110 trillion dollars per year; if corporate profits are 10% of that, that'd be 11 trillion dollars per year of profits, and you'd expect share prices to average around 20 years of profits, which would be 220 trillion dollars, much higher than the 89.5 trillion dollars in the infographic. perhaps outside the usa corporate profits are much lower, or investing in the stock market is much riskier? it's certainly riskier here

com
1 replies
23h48m

Thank you for this very honest comment. It means more to to me than I was expecting it to when I started reading it.

Were there any moments on the journey of growing up that stick in your mind as being turning points in your path from techno-determinism to whatever you describe yourself now?

I’m really interested in how we can intervene earlier in people’s journey to provide bigger horizons, rather than just waiting for enough experience to build up…

gumby
0 replies
22h22m

Thanks. I think I mainly developed more empathy.

I married an artist. She knew no physics, no advanced mathematics, but plenty of philosphy and languages as well as the technology and philosophy of representational art. She was a lot smarter than me and we spent a lot of time expanding each others' horizons, not deliberately or didactically but just through living.

Also so many things seem clear when you are young and I was repeatedly humbled by how complicated the real world is compared to academia or research (where I worked on very abstruse subjects, e.g. the denotational semantics of reflexive languages).

I think you see this in a lot of people: libertarianism and even Ayn Rand is more popular with young people ("hey I'm smart and responsible; why are there all these annoying and pointless rules getting in my way?"). I was never a libertarian but back then I was sympathetic to Rousseau. But you have fewer grown up libertarians because many learn to realise that sure, things are imperfect, but we live in a world that evolved (sometimes well and sometimes not) to work with fallable systems and lots of opinions, some well thought out and some...not so.

jocoda
0 replies
10h35m

tangential

... without marketing (e.g. gmail

I would hard disagree that there was no marketing.

gmail had one of the first really successful guerilla marketing campaigns imo. Google made the initial phase by "invite only". By making it exclusive, a whole ecosystem developed around getting invites so that you could have that @gmail.com account. By the time of the official launch they had captured what could losely be described as the influencers of the time.

Invite only has been tried a few times since then, but never to my knowledge has it been done as well.

nonrandomstring
0 replies
1d6h

Good share. Haven't read but looking at the wiki entry it's an interesting book, Technology requires a culture to sustain it.

To use a fancy word there's a "noosphere" around that carries knowledge and lore along with it.

Plus the usual kind of Deming wisdom about real knowledge organisation:

    "in many ways opposite of traditional management...innovations are
    started at the grassroots level.", and "people will give their best
    when the work itself is challenging and rewarding."
That atmosphere is still around in engineering teams in smaller companies where there's a good range of advancement within engineering and some stable products so there's a stream of old-timers and apprentices building and exchanging chops.

[0] https://en.wikipedia.org/wiki/Noosphere

[1] https://en.wikipedia.org/wiki/W._Edwards_Deming

f0e4c2f7
0 replies
1d4h

This book is so good. It's about building early computers but it feels just like tech does today in the way it's described. Which make it feel like this bigger context you're reading into. That even before computers were mainstream, there were still those who tinkered.

Two things are especially memorable to me. One is a casual remark in the book that they found the best way to get things done is to pair someone very experienced and cynical with someone very inexperienced and naive. Combined they would get lots done together compared to either alone. I think this is still true today.

The other thing is the intro. It's about the head of the project getting a group together and renting a sailboat on vacation. On the sailboat the get tossed and at times feel like they barely survived and it ends with someone saying "if this was his vacation...what did this man do for fun!?"

avisser
0 replies
1d5h

"a great read" - Almost underselling it. It won the Pulitzer for non-fiction that year.

EncomLab
0 replies
1d6h

One of my top 5 favorite books of all time - the audiobook version on audible is excellent as well. A lost world - and we are all worse off for it.

neilv
2 replies
1d5h

The latter half has a lot of criticism of allegedly boneheaded management (from the perspective of the writer and researchers they quote).

rini17
0 replies
22h26m

Strictly siloing maintenance away from sales and development is boneheaded clearly, not allegedly.

momojo
0 replies
2h57m

The author paints a very vivid picture of mismanagement, but I wish he had fortified that picture with some quotes or anecdotes. The "management" people remain this fuzzy, dark force throughout the essay.

EdwardCoffin
1 replies
1d4h

This passage particularly struck me: He noticed when a technician on a call began by examining copies that had been thrown in the trash and deduced from them that the problem with the machine was different from what the customer had reported. “The trashcan is a filter between good copies and bad,” one technician explained “Just go to the trashcan to find the bad copies and then… interpret what connects them all.”

On a related note, I'd like to highly recommend Lucy Suchman's work, mentioned in this article as Plans and Situated Actions: The Problem of Human-Machine Communication, but the updated version now called Human-Machine Reconfigurations: Plans and Situated Actions. The new version has several extra chapters and some other revisions. I've read it several times, and had my mind blown each time.

indrora
0 replies
20h16m

For a long while, I've been helping people diagnose problems with 3D printers and one of the things that stands out to me is how often people throw away bad prints while they're troubleshooting.

One of the first things I ask someone now is to give me their test prints for the last few changes they've made and what they changed, if they remember. Nine times out of ten, just looking at other factors, I've been able to pinpoint what's wrong.

Interestingly, this has also led me to a few "is it turned on" questions, the first of which is "when is the last time you tightened all the bolts that should be tight and loosened the bolts that should be loose?" The simple act of making sure tight things are square and tight has resolved many an unrelated problem because it makes you check so many different parts of the machine.

This also correlates with one of the takeaways from the old Computerworld column called Shark Tank. The most common aspect of the solution was to simply go and observe the failure, because so often it was an unaccounted for externality -- such as a cleaning crew unplugging something at 2AM when a system would "mysteriously" go down.

vaxman
0 replies
3h47m

What the...

I open the article and it is not about "maintaining" the old Data General and DEC machines (the original "The Should of a New Machine" was about the race to create the first 32-bit minicomputer --between DG and DEC. DG was a former group of DECies --spoiler: THEY LOST! It was a hell of a lot more worthy a battle than Qualcomm vs Apple, which is almost the same situation.) The old XEROX machines were boring as they competed with IBM in the business space whereas DG and DEC were more focused on Science and Engineering. The XEROX Star was pretty cool, but not nearly as cool as History has recorded it to be (apparently because Steve Jobs, who later became an icon of sorts himself, once saw a demo and rushed to recreate it using microprocessors in his poorly implemented Lisa/Mac system). I developed on all of these machines (though I only used the Star to write documentation for my VMS internals code hahah) and these are MY opinions.

vajrabum
0 replies
1d1h

I spent nearly 20 years fixing computers and other electronics. Repair techs are my original work tribe and it was a fun if sometimes stressful way to make a living. I got away from it because the money went away. That said, I never wanted to fix copiers. They were always finicky, messy and dirty, but this is a really great piece and three things stand out for me.

The article claims that PARC paid for itself (1) through the anthropological sociological studies of copier repair technicians which revealed shortcomings in the engineering of the copiers and resulted in changes to the paper path and handling in newer designs and significantly reduced maintenance cost and difficulty. Two, enabling information sharing between repair technicians over radios and technician created and maintained documentation, saved the company 5-8% of service cost and these innovations were resisted by services management which was invested in the idea that copier repair technicians should be cheap, interchangable monkeys. Three, Xerox management likely left significant money on the table because they fundamentally and willfully misunderstood copier repair and copier repair technicians and the value they were creating for the company. Likely, mostly because repair was seen as a cost center which in an ideal world would be eliminated entirely.

1. It's really astonishing how much and in how many ways PARC paid for itself and yet business literature and likely Xeroxes management often focuses on the money left on the table for others to grab and asserts there was a failure.

stonethrowaway
0 replies
1d2h

Though they were doing messy blue-collar work, Xerox required the technicians to act and dress white-collar. They carried their tools in a briefcase.

We don’t carry tools in briefcases because it makes us appear white collar, but because the shell is hard and protective, there are many sizes, and the boxy interior can be formed to however you like if you use foam and cut it to fit your tools. Briefcases fit readily into many tight spots for transportation. The photo shows the usual layout of tools that techs use. Companies sell high end equipment in briefcase-like containers because it keeps them safe and waterproof in needed situations.

Not a big fan of the anthropology aspect. It’s a job. Techies improvise, it’s not a clandestine operation to fix a machine.

kmoser
0 replies
15h9m

Back in 1986 I worked for $BIG_BANK where we replaced humans, typewriters and carbon paper forms with Xerox laser printers (conveniently leapfrogging daisy wheel and dot-matrix technology). These beasts were the size of a large microwave oven, or maybe a small dorm fridge turned on its side, and I seem to remember them costing $6,000 but my memory is fuzzy and it may have been as much as $15,000.

Anyway, these printers were finicky, and I remember we were visited by the same two Xerox service techs on a regular basis. In addition to repairs, they were faced with the near impossible task of keeping the toner (which was dispensed by hand from a plastic container, similar to powdered clothing detergent) off their white shirts and ties. We saw those guys so often that even now, 38 years later, I still remember that one of them was named Randy.

katzenversteher
0 replies
1d6h

I while ago (around 2010) I worked at Océ Technologies R&D and at least to me the machines while incredibly complex where quite easy to operate and maintain. In fact every developer was allowed to print their private stuff (non commercial) on the machines under development. I believe this really helped because we basically had to become operators and maintainers and got a feel of their roles. If you print something for yourself, family or friends you also take a better look at the output or sometimes have a very specific use case in mind.

goffley3
0 replies
1d6h

This goes a long way in explaining the incredibly warm and sociable, yet undeniably peculiar Xerox tech that I would interact with on a regular basis. Despite him not actually working for the company, I would have to call him in constantly to get him to fix one or more of the machines we had in the office.

AstroJetson
0 replies
4h6m

For what it's worth, there are two other chapters up. One is about round the world sailing. It is also worth a read.

I've not been able to get into the third chapter, but it looks equally interesting.