Pandoc

This is one of the most useful programs that I use.

I use it for turning .md files into .html or .pdf.

I use it for creating slides with it.

I even use it for fixing the hard-wrapped text I write in vim before sending emails. When I write in vim, I prefer the text to be hard-wrapped, but for emails, I like it better when the text is not wrapped. I recommend arp242's essay explaining the problem with hard-wrapping [1], but basically the way I workaround this problem is using a local script which uses pandoc at some point [2].

Overall, pandoc is really good.

[1]: https://www.arp242.net/email-wrapping.html

[2]: https://github.com/kugurerdem/dotfiles/blob/2d68357273e1bc30...

I'll probably lose some nerd cred: I do most of my writing in MS word, I find it easier to cut and paste and add footnotes and section headers. And then I use Pandoc to convert the docx into the format I want, usually HTML*. I used to do markdown in vim but I found that for most of what I do I prefer word. I do write code in vim...

* I use this css file when converting: https://gist.github.com/killercup/5917178

I also use Word for all my writing too, so wanted to defend you. People (tech people at least) have lost sight of the fact that writing should happen in "word processing software", and Word is the best-in-class. To people who say "Word sux", I say "That just means you don't know how to use Word properly". Writing any markup or markdown syntax in an IDE is a disaster for the creative process. Jack Kerouac used to type using rolls of paper instead of sheets so he didn't have to stop is his process. He "got it".

As for pandoc, yes it's amazing, and I have been using it to convert my word documents to markdown so I can publish a technical textbook I'm working on using Quarto. I tried writing directly in Quarto for a while, but as per my point above, it really slowed me down and distracted me from actually writing, so I figured out the pandoc pipeline. My most favoritest feature so far is that it converts tables AND equations to markdown and latex perfectly. It's so seamless that I'd actually recommend Word->pandoc as the best way to write a complicated markdown table.

Most tech people who use Word know how to use it and that is exactly the problem. We don't have the benefit of ignorance. We send off drafts to someone for commenting or editing and get an inconsistently formatted mess back because others don't even know that styles exist and use manual / direct formatting instead.

We are the ones that have to suffer because people we are forced to collaborate with do not take 5 minutes out of their day to learn the basics of a tool they use professionally.

It is the equivalent of seeing somebody use right-click to copy-paste, except it tangibly makes my day worse.

So what you're saying is that the problem with Word is that it makes it easy to write unpolished and unprofessional documents? In your opinion, is that the fault of the word processor or the user? [0]

[0]: Not a rhetoric question that implies one answer to be the only "right" one.

IMO entirely user.

Word (and similar text processors) is a toolset. All it gives you are tools. Word lets you define formatting rulesets, lets you apply formatting rules and create rulesets from applied rules.

Some evangelists may say that "safe" text processor would only allow application of rulesets, because direct application of rules leads to "spaghetti formatting". However that is one of the powers of WYSIWYG text processors: you apply the rules and extract those to rulesets once you are satisfied with results, in an explorative way. Direct application of rules is a feature that makes Word what it is.

Now, if a user takes a document with predefined rulesets and still applies their own rules inconsistently that's simply misuse of the tool.

I find that question uninteresting, because no matter where the fault lies (it's probably somewhere in the middle) the end consequence is that there is no use case where it makes sense for me to use Word (unless I'm literally held at gunpoint).

I'm either working on a document for myself or with collaborators.

In the first case I'll use markdown for simple things or LaTeX for bigger things since I can work in a familiar environment where I work most efficiently (VSCode).

In the second case, I'll work with collaborators so I will never be able to trust that a document I have sent off for reading is still consistently formatted when I receive it back. This means that any benefits of the collaboration tools (eg. review history or suggesting changes) are wiped off the table. I will have to integrate any suggested changes into my own authoritative version of the document by hand anyways. At that point I may as well work where I fell comfortable and use markdown / latex, and send off Pandoc converted word files for comments by others when it is relevant.

This is of course for serious pieces of writing, not throwaway stuff like eg. meeting notes, but for those Google docs is plenty.

It is the equivalent of seeing somebody use right-click to copy-paste

What would you prefer, instead? CTRL-C, CTRL-V? SHIFT-INS, CTRL-INS? some vim incantation?

As a select-to-copy/middle-click-to-paste guy, seeing people use these inferior alternatives looks extremely annoying to me.

I think that depends if your workflow is oriented around the keyboard or the mouse. For me it is keyboard oriented and I frequently use (Ctrl)+Shift+Arrows for finegrained selection anyways so Ctrl+C/V is most convenient. I also use Vimium in my browser. If my workflow was mouse oriented I'd instead use Gesturefy.

I think they're equivalent and certainly both are better than using the right-click menu.

I don't always use the mouse, but when I do, I only use the mouse, no keyboard needed at all. Using CTRL+C/V requires an unholy synchronization of mouse and keyboard. But unix-style middle-click paste is entirely mouse-controlled and very elegant. Of course, if you are inside a text file, you can use vim keyboard tricks that are even faster because you don't need to select the text.

Is there a way to lock documents in some way, so that direct formatting is disallowed and only styles work? Not that we could truly lock a document, but at least having some sort of header that lists allowed features, such that one would get a warning whenever they veered off course?

Don't think such fine grained control exists. There is read only mode[1], which (I assume) limits the reader to only comment.

[1]: https://support.microsoft.com/en-us/office/make-a-document-r...

Good point, sigh...this is so true. I have created a 45 minute tutorial of 'how to use word properly' to address this exact pain point. I send a link to my students if their first draft commits any 'sins'.

BTW, when I said tech people, I was thinking mostly about the computer savvy academics who use Latex for everything.

People use things like LaTeX exactly because of this problem. Word processing software brought the problem of inconsistent formatting and layout within the grasp of everyone and boy! ...did they grab it with both hands. Systems based on plain text allow the author to concentrate on content only, without the need to format.

Personally I find LaTeX misses the mark. Too much markup is needed and it detracts. I'm a fan of asciidoc though. I just wish the templating was a little better though.

slowed me down and distracted me from actually writing

It's a bit of a matter of perspective.

The distinction between looking at a document in markdown versus Word is a bit analogous to the distinction between looking at a movie in its textual form as a screenplay versus looking at a movie as a piece of video: Text is capable of abstraction in a way that video is not.

In the screenplay, it might say "table", but when the director translates it to video, the director will have to decide: What kind of table? What design? What period? What texture? Is there anything on the table?

None of these decisions matter to the construction of the story, so, for a screenwriter, it would be very distracting if they had to make all of those decisions just to be able to get "table" committed to the medium.

In Markdown you worry about text and nothing but text. But Word shoves a particular font in your face as soon as you're laying down the first letter, so, if you don't like Word's choice of fonts, you can either let it annoy you throughout the project, or you can start worrying about fonts right then and there, which will be a distraction. If you write Markdown in a code editor, then, presumably, you've already set up the code editor in a way that doesn't annoy you. And then your future self (or someone else entirely) can worry about the font.

I know you're just font as an example, but doesn't your markdown editor also shove a font in your face? You can adjust default behavior in Word just as well as in another editor.

Anyway, to work within your analogy, I would say that Word lets the write do a bit of a 'mockup' of the set with nearly 0 effort. Like "I want a table here", so 2-3 clicks and you have it. Then you can "let the director" take your mock up and flesh it out properly later. As a writing, it helps me to see the mock up of the product as a go, but I want that mock up to be effortless. And as I said above, I do some work up front to make sure that Word's mock up looks good (or good enough).

...and that's the problem. You can easily get distracted by the look of the words on the page. Is the heading big enough, centered, got enough white space around it. By the time you've faffed about with that, I'm onto my second or third paragraph of content.

We collaboratively write screenplays in markdown (actually fountain but its markdown plus some screenwriting stuff) and save them into Dropbox.

pandoc and some other tools turn those scenes into a full screenplay.

It's worth noting here (even though I also use vim/markdown/typst, and Word is a thing of the past) that markdown does say something more than "just the text".

It puts the header notation and style (like italic, etc) in-line. So does Typst or LaTeX, and I can't think of any typical stand-off examples for headers and such, but it does muddy the text in-line in that sense. It typically doesn't really slow down the writing, but if you're writing \# as comments for code, then don't have those wrapped in \`\`\`, you can get some problems.

Part of the reason I stopped using Word was that I absolutely abhorred 2007-and-beyond’s equation editor. I was using previous versions of Word with Mathtype for my math homework, but I found the new equation editor really hard to use. Around the same time I had coincidentally to Linux, and OpenOffice has an even worse equation editor.

Pandoc was a game changer for me. I picked up LaTeX equation editor pretty quick, and being able to write markdown was so much more pleasant in my mind.

It’s not perfect; tables are a pain still, but I have no desire to go back to Word.

You are probably the only person I've heard that liked the old mathtype editor :-) The new editor is terrific because you can use just type the equation in latex format (e.g. A_c = \pi R^2 ). Then you hit the space bar and it converts to wysiwyg style. Most of my equations are on the simpler side I guess.

I didn't know about being able to type the LaTeX stuff; that's pretty neat.

I think part of it was that I had basically memorized all the keystrokes for the MathType editor, and most of them didn't work in the MS Equation editor, which annoyed me. Also, I had issues with parentheses formatting correctly but I suspect that's been fixed in the last 15 years.

Still, I really do prefer to work with Markdown in general. The Markdown -> Pandoc -> LaTeX rendering just ends up looking prettier in my opinion, and at this point I'm pretty useless in any editor that doesn't have Vim keystrokes. Pandoc irons out the parts of LaTeX that I really hate (the `` vs " being the thing that's given me the most headaches), while letting me drop into raw LaTeX when I need it; not even getting into the fact that there's just math stuff that (as far as I know) doesn't work in Word or MathType's equation editor (e.g. bussproof trees).

I do get pretty annoyed when people try and tell me that regular LaTeX is "just as easy" as Word, because even as someone who has a reasonably good handle on LaTeX I can say that is just not true. TeX is arcane and weird and annoying and inconsistent, and I don't blame people for using Word compared to it.

People (tech people at least) have lost sight of the fact that writing should happen in "word processing software"

The fact? Should? In "word processing software"? Why shouldn't it happen in "text editing software"? Writing produces text, after all.

This. Text editors automatically give you that exact Jack Kerouac "rolls of paper" experience. It's just text. Formatting comes later, word or tex or some other system you like.

> To people who say "Word sux", I say "That just means you don't know how to use Word properly".

Or maybe those people just have eyes which can spot the differences in spacing between documents created with different Word versions, the mediocre kerning, and a multitude of other typographical annoyances.

Even a website is easier to replicate exactly in another browser than a Word document to be replicated from scratch in a new version of Word. And if by some miracle you manage that, the end result will always look meh.

Yes, anyone who's ever learned LaTeX and why it was invented immediately sees how absolutely awful Word is at its rendering. Funny that a trillion-dollar company can't seem to figure out a better algorithm than Donald Knuth mocked up in Pascal decades ago.

I think of this as another version of enshittification -- the acceptance of poor performance as the "standard".

And don't get me started on math equations in Word ...

For me personally it's the reverse. After 15 years of writing almost exclusively with vim (or vim-like) input scheme using anything else would really break my flow. Can you even do something like ct, (delete everything between current cursor and next comma, then start insert mode) in word? Even if you could, it would be some 5-key monstrosity which makes you move from home row.

Now granted, if you're just hammering out words without any editing this doesn't matter, but I think almost every piece of good writing has had 3+ revisions.

Of course this is a personal preference, but my friends who do a lot of writing (and never used vim at all) still seem to prefer a distraction-free editor with a lot less features and a much less noisy UI for the actual writing.

Rant ahead:

So, as it must be me not knowing, can you tell me why Word won't let me change table column size ~50% of the time (for the same table from the same source). Half the time autofit works (yay!), half the time it won't let me resize columns, neither by typing the number, nor dragging the invisible off-page divider (Microsoft should just make the transparency of off-page content 50%??), nor changing to draft view and dragging the actual dividers ... wtf is going on there? If I paste it into OneNote first then it fits ... sometimes, if I drag a column wider first (which it already rendered 5x the width OneNote did) then it will let me narrow it afterwards, what's that feature called AutoNoNarrowColumnRenderedIncorrectlyFiveTimes ExpectedWidth?? I'll just search the settings to turn that off ... oh wait!

These are the times I long for markdown, or 'reveal codes' ... MS Word has a lot of problems they could probably have fixed if they hadn't put so much effort into preventing interoperability. These sorts of issues were around 20 years ago when I stopped using Word, and 5 years ago when I restarted. Same asinine poorly implemented numbering and styles that are unintuitive, opaque, and ungainly ... and don't get me started on search! Multi-highlights? Sorry best I can do is "find next" with no find previous, no regex, ... you can do find in AutoText though, right, right? ... and all the AutoText and AutoCorrect gets saved in a single sensible format that's easily modified? ... Word changes the format of all windows when you open a new one too, just in case you thought the suck was restricted to within the window chrome ... and doesn't have always-on-top, and doesn't open windows in their last position, and ...

Doesn't suck ...???!

Whilst you're here, any ideas why OneNote eliminates footnotes so you can't cut-paste between Word and OneNote? I'm sure it's not flawed and I'm just holding it wrong, right ...?

I agree with your premise of "don't be distracted" when writing, but for me Word often is the distraction. I use a live-preview markdown editor (e.g., Typora, MarkText) to let me get my thoughts onto paper (screen) with low friction. It's easier to just hit "#" rather than drag the mouse to the style bar and select heading. Or more importantly for me, it's so much easier to hit "$" and seamlessly go into LaTeX for math than it is to open the equation editor and start selecting all the template objects.

When it's time to collaborate, I use Pandoc to turn it into docx and then I send it around and the final formatting happens in Word because that's the easiest for everyone to work with, but the "get the ideas down" phase works best for me in a more "minimal" editor with little formatting.

I love the idea of Quarto, and if I had that when I was in grad school, it would have made my life so much easier. The workflow I see for Quarto is that you can write your paper while you're doing the experimentation because the code is embedded with your thoughts. But in that case, you're mostly slowed down by the research process so it can be a little more clunky to get the writing done because you have time and you're iterating over ideas more than words in that phase. I'd use it now for work in the R&D phase, but I know I won't have a critical mass of collaborators to make it worth while.

My most favoritest feature so far is that it converts tables AND equations to markdown and latex perfectly.

You might be interested in Texmacs [1]. It is has a wysiwyg interface, and it handles nicely tables and mathematical equations. Also you can export documents to the latex format.

[1]: https://www.texmacs.org

There are other reasons to not like Word other than 'it six's. One is availability. If you're on Linux all day, your options don't really contain word - it's not free to download and install due to needing a license. You have to go and find some other computer that runs windows, and start it up - just for that one document. Then you have a single use computer, which obviously isn't a great experience. Closest thing that I use is Google docs, sometimes. I (and many others like me) know how to use word (and excel) pretty well due to some life in that world. I still remember most of many of the hotkey combinations to step through the menus. The reason I use vim for everything now is - I use vim for everything. I live in vim/tmux/ssh/tui, so most problems go there first if they can. There are obvious benefits to the vim/git/Typst setup, if plots need to go in a document for example, but it's also strengthened by word not being easily available.

People (tech people at least) have lost sight of the fact that writing should happen in "word processing software", and Word is the best-in-class.

It depends on the person. The best way for me to write is on paper, on a desk with lots of empty space, with paper versions of all reference material.

Writing any markup or markdown syntax in an IDE is a disaster for the creative process.

I would like to quote this on a MonsterWriter landing page

If you want to use Word, then that's great, but when you start saying what other people 'should' be doing then they're going to speak up.

writing should happen in "word processing software"

Writing should happen where you are comfortable editing text. I am comfortable in same editor where I write code.

To people who say "Word sux", I say "That just means you don't know how to use Word properly". Writing any markup or markdown syntax in an IDE is a disaster for the creative process.

To all the people who say "Writing markup in an IDE sux", I say "That just means you don't know how to use it properly". I can write in a flow and apply/change formatting easily. I can jump around and rearrange documents with ease. And it is in a format that can be opened and read by native software on almost any computer.

If you think that Jack Kerouac would prefer MS Word over a much simpler plain text editor, I don't agree.

We do the opposite of this - we write in markdown but sometimes need to get feedback in Google Docs. Pandoc doesn't convert to google docs so well, but it does to docx, so our pipeline is

markdown -> pandoc -> docx -> upload to google doc -> share

My kingdom for proper markdown support in google docs. Just let me toggle between wysiwyg vs markdown. Collaboration at a tech company using google docs is comically painful at times.

+1 strong agree. I like Google Docs’ realtime collaborative editing, but the wysiwyg formatting (even if one has memorized the keyboard shortcuts) makes it hard to be fast the way I am in a Markdown-aware editor or vim. Plus I miss my vim motions!

You know what, that's a sufficiently cursed workflow that it wraps back around to adding nerd cred

Like using a hex editor to build up a Word doc?

For documents that require a longer process, I prefer formats that I can use comments and TODO in, with a text editor that feels familiar.

Other than that, I strongly agree. Word is also de facto standard when you want others to open and edit your document.

Word has comments. They work well as todo markers if they don't need to be retained (just make a comment to mark something and then delete it when it is resolved).

you managed to find a lean way to go docx to pdf using pandoc ? last time i tried it required a whole latex stack behind it.

Process over Tools.

You might find it interesting that I never read any docx sent to me and instead run it through pandoc to convert it to markdown first.

While pandoc is certainly a very useful tool, don't you think it's overkill to call a 143MB (on my Linux system) pandoc executable to do word (un)wrapping?

Emacs has `fill-paragraph` built in and `unfill-paragraph` is a short function definition [1]. Both work across multiple paragraphs.

[1] https://www.emacswiki.org/emacs/UnfillParagraph

Nice point! I agree that it's a bloated method, however, this was the best solution I could come up with at that point of time. I am certain that there are better solutions, it's just me who could not find it. I went with this solution anyways since it was still an improvement compared to my previous method in vim which involved setting the text width to a large number like 9999, highlighting the text I wanted to unwrap and then typing 'gqq' for formatting, and then yanking it. And of course, I had to re-wrap the text if I wanted to maintain the original format of the text file.

You are absolutely right and I understand what you mean though. I am open to trying other alternatives and I should try to come up with a better method to workaround this problem.

Never tried Emacs, just went with Vim so far. Did not know that Emacs had already an elegant solution for this problem. Nice! :)

My solution would be to go to the start of each paragraph and hold J (shift + j) until the entire paragraph was joined onto one line, and then go down to the next one. I guess it depends on how long your emails are but this is pretty quick.

My problem with this method is that it's not convenient especially when there are lots of paragraphs. When you highlight all the paragraphs and press shift J, it unwraps everything into a single line. This results different paragraphs ending up on the same line. You also need to undo the changes you made to avoid disturbing the original file content.

I haven't needed it myself, but it might be easier to just keep the document in line-break-per-paragraph form and just turn on visual-line-mode...

I have almost never thought about the file size of a program (even for big games) on a daily use machine.

I am certainly not gonna be learning and switching to emac just to perform this task for unwrapping a .docx file

It is one of the most useful programs I use and the only useful program I ever used that was written in Haskell.

only useful program I ever used that was written in Haskell

Never used Shellcheck?

xmonad team checking in.

Wasp team reporting to duty: https://github.com/wasp-lang/wasp

Whoa, I use Shellcheck a lot and did not even know it was written in Haskell.

Interesting. And how do you send and read emails? Mutt?

I used to use Mutt, but now I don't because my email has its own special domain, and the email hosting service I use doesn't let me export IMAP and POP3 details unless I pay them.

Currently, I just open vim in the terminal, write what I need, then copy the text. I open dmenu using a shortcut, type something like "unwr", which is sufficient for selecting the "unwrap-clipboard" script of mine, press enter, which unwraps the text on the clipboard. Finally, I paste it in the email client.

I know it might seem a bit tricky, but it's better than what I did before. I used to set the textwidth to a really large number like 9999, highlight the text I wanted to unwrap, and then type 'gqq' for formatting. And don't forget, you also have to wrap the text back if you don't want to change the original format of the text file you wrote.

You can use soft-wrapping in vim, but I don't prefer it. Soft-wrapping is not as convenient to me as hard-wrapped text when using vim shortcuts. For instance, if you are using soft-wrapping in vim and you press 'o,' the insertion mode will start at the end of the paragraph because vim will consider the entire paragraph as a single line. However, there are many situations where I only want to insert text in the middle of a paragraph. You will also most likely set j to act like gj and k to act like gk, to make the cursor move between lines that are visually separated but actually form a single line. I don't like this either.

I had a script that piped to curl to send, Mail.app for downloading via POP and Vim to read/write, but I gave up and ended up copy/pasting as well.

Someday I may try that again, if email is still relevant by then. Vim and email would be my perfect set up.

I had a script that piped to curl to send, Mail.app for downloading via POP and Vim to read/write

Wow, this is very interesting, and I might even try it at some point. It might have been a bit challenging to sync and read the emails though, but the sending part seems nice.

Was the reason for why you gave up related to syninc emails?

I liked that Mail.app stored every message as a single separate file, instead of the mbox that Thunderbird and others did. But at some version they started saving a hash with a multiple folder structure and that broke the setup. I had a few issues with MIME encoding as well, which were probably my fault. It was too much work and I eventually gave up, but if I find the will to do it again, I'd choose a simple POP downloader + some filter (if not in address book or previous recipients, move message to “Unknown”).

qlmanage worked great for HTML messages and attachments, though.

Quite a coincidence: I also stopped hard-wrapping my emails after reading that same essay, and I also use Pandoc to prepare my emails (or a program, invoked by a shortcut in vim, that sends the selected text to Pandoc).

A very interesting coincidence indeed. Nice to see that others have also thought about the same problem and have found solutions similar to mine. :)

I probably use it every day, without noticing, to view markdown docs in a terminal, via a `.lessfilter` invoking [my fork of] https://github.com/Orange-OpenSource/pandoc-terminal-writer

I also very much like Pandoc-markdown's ‘simple table’ syntax, because it's actually human-readable and human-writable without confusion and pain.

A question to experienced Pandoc users:

I want to write a small book that I want to generate in 3 formats: HTML pages, EPUB and PDF. What is the best input format (source format) for the book? Pandoc Markdown? CommonMark? GFM?

I'm a little hesitant to committing myself to Pandoc Markdown or any Markdown because they all have tiny differences with each other. Each is like its own standard.

I considered Org-mode for some time but there are so many edge cases in which Pandoc does not parse Org-mode properly. I mean sometimes simple things like internal links are not rendered properly by Pandoc in the generated output.

So what's the best format to write the input in? Any ideas? Opinions?

I'd use leanpub markdown. They generate those formats plus you get a page for the book, can charge for it, and advertise it. (Some of those tasks cost money).

Won't work if you want to keep everything local, though.

Thanks! Didn't know Leanpub has its own markdown too. Yes I do want to keep everything local.

Ah, then probably not a fit.

I came here to say the same thing. Laying content out for a book comes with way more issues than "just" converting between document formats. Leanpub does it well out of the box.

If you want to look down the path of implementing book layout yourself, here are two breadcrumbs from my bookmarks:

https://journal.stuffwithstuff.com/2014/11/03/bringing-my-we...

https://iangmcdowell.com/blog/posts/laying-out-a-book-with-c...

For leanpub, I started using it for the layout. And kept using it for the marketing.

That's one thing I learned, having written a couple of books. Even though writing is tough, it is easier in many wyas than the marketing.

Personally, whatever helps with the specific writing part of it all the most is what's best. If you find writing in a given dialect of Markdown or LaTeX or Org-mode is easiest, do that. For me, that's Markdown with embedded LaTeX, for others it's Org-mode, or RST, and so on.

Pandoc handles these fairly seamlessly, and with many options for PDF engines, though I'd say it has a preference for LaTeX and HTML in the backend and Markdown in the frontend, based on my experiences with the edge cases (sometimes entirely solvable with a little Haskell or Lua).

Since LaTeX is the default for PDFs, it pays to keep that in mind and help LaTeX help you (you can use it inline with Markdown or included as preamble in configuration), but sometimes I've just had better luck converting via HTML to PDF ("-t html output.pdf" or directly chaining on from output.html) for what I'm writing in the moment, though other times I'm not stressing LaTeX as much and can just go straight from Markdown to PDF (for example, just writing up something with inline maths). I prefer to avoid LaTeX or HTML's escaped character encoding and often need far more than a single Latin font can provide, so I've ended up dealing with LaTeX's limitations here (even in lualatex and xelatex) more than what I'd suspect is typical. Meanwhile, the standard HTML to PDF backend uses Qt, and I've found it works for everything else I've needed when LaTeX isn't the right backend (and it does come up). On one occasion, I did have to switch that to weasyprint, and that was everything sorted. Alternative backends is an unsung power that few have, while pandoc not only has many built-in (or it is at least internally aware of) but will also integrate with any CLI needed.

Output to all three with HTML, EPUB, and PDF can just need a bit of fiddling before it comes out right, depending on how much you're willing to mess with specific metadata for each versus accepting the limits of what Pandoc can handle universally in its AST. Invariably, some compromise is required, but the core semantics of Markdown (including extensions) almost always translate without an issue. The dialect problem of Markdown is really just in the confluence of said semantics with things that have not been separately included, such as the lack of an actual header in Markdown (Pandoc here allows YAML for some, or you just fall back to HTML).

So, tldr; there's no "best" input format, except the one that you find most comfortable to just write the book in, but I find Pandoc is usually best approached from Markdown with the LaTeX or HTML backends. It's powerful and oh so very handy, but it's not going to do all the thinking for you, just a lot of the grunt work, same as any other tool. When in doubt, the user manual is quite readable, and I've found it answered almost every question I had. When it doesn't, other people do, and when they don't, it means I'm either going about it the wrong way or I get to solve an actual problem (but usually the former). But, as always, the most important thing is actually writing it, distribution comes later, so focus your efforts on that and the tools you need to do that effectively.

If you find writing in a given dialect of Markdown or LaTeX or Org-mode is easiest, do that.

I find Org-mode the easiest but like I said in my comment, the conversion quality is not great. Pandoc breaks a lot of stuff in Org-mode in edge cases. One example I shared in my comment was Pandoc breaking internal links.

So by selecting something I find the easiest I have burned many hours of troubleshooting figuring out why the output does not look right.

That's why I want to draw upon the wisdom of the community here to find out which input format works best and by best I mean flawlessly. No edge case issues. No rendering flaws. If I get the specific recommendations, I'll try them out for sometime and then commit myself to it instead of burning more time trialling all of the different input formats.

Unfortunately, the perfect is very much the enemy of the good here. Aside from HTML, I'm afraid that PDF and EPUB are very much driven by purpose-built tools designed to show interactively what it will look like as output. This means that they've both delved into a depth of subtle semantic differences that makes flawless output an extremely difficult task. Of course, practically, pandoc can resolve the vast majority of what people actually use, but everything will still be hit by edge cases from time to time, leading to subtle issues or incompatibilities between EPUB, PDF, and HTML. Each edge case can, of course, be solved in isolation, so finding something that's solved the ones you are encountering already is the ideal, providing a seamless experience for your work. Sadly, each of those is built to solve someone else's specific work, and so sometimes we just have to accept that we either need to compromise on something, we need to paper over the gaps by combining the right tools, or we have to write something ourselves. Fortunately, it isn't the 80s anymore, so many of the tools we have are the "right" ones, and pandoc is very good at combining them.

Again, I find that Markdown (with inline LaTeX or HTML) seems to be Pandoc's preferred starting point, and that the HTML backends are quite useful (particularly when not needing full LaTeX), so perhaps there's some luck to be had there, since HTML may preserve Org's linking and such a bit better, though I don't use Org myself so can't attest to it. And if there's really a problem, then perhaps Pandoc needs some help sorting Org-mode out!

Great comment! Thanks for engaging in this discussion and offering some good perspective about my Pandoc issues. Really appreciate it!

Is there any reason for why you’re not considering AsciiDoc?

No such reason. I'm willing to try out AsciiDoc.

I mean I did not try AsciiDoc until now because there are so many choices of input formats and the ones I've tried so far have been disappointing one way or the other.

I talked about Org-mode rendering broken in edge cases. Same with Latex too. I see Pandoc has first-class support for its own Pandoc Markdown format. But the support for all other input formats seem patchy.

If you think Pandoc has good support for AsciiDoc without any edge case issues, I'll be most certainly trying it out.

Seconding this. Asciidoc was created to be a simple mapping to DocBook - which was specifically designed for writing books. It should be able to handle everything you need, but if there is some esoteric requirement, you can write your own processor.

Then again, content is king. Write it on napkins if you must. When complete, you can spend two days to transcribe to whatever format the publisher requires.

What snet0 suggested might be your best bet. You can use pandoc markdown but then slip into LaTex if you need to do something more complicated.

If anyone else was looking for what snet0 suggested and where, here's their comment: https://news.ycombinator.com/item?id=39227851

I've translated HTML markup to Markdown, and from Markdown to LaTeX before fine-tuning the LaTeX, to produce PDFs for printing hard-cover books.

Does Markdown have any way to specify eg "begin a chapter on a new page" ? I don't think this is really a thing in Markdown or HTML but I'm admittedly a casual Pandoc user.

If you are writing a book, presumably you would have each chapter in a separate markdown file.

So, then you convert each chapter file to pdf, and then join the pdfs.

Might depend on what kind of book it is - do you have a lot of images, tables, cross-references, ... or is it mostly plain text?

Mostly plain text but some images, tables and cross-references too.

Like others above, I write long form in Word/docx and convert with pandoc. Word supports styles that help experiment document-wide with look, it has an outline mode that helps with re-organizing material, and it handles inline media gracefully. Any of that is painful in markdown. There's no way I would write 20-500 page document without Word.

I use markdown for the 90% of my writing that is blurbs (and parse it into something like a graph/knowledge kb, and often render via pandoc to pdf, word, Anki, and html).

In both cases, I restrict myself to using features that can be parsed.

If you are committed to using Pandoc to generate the three formats then I don't see why you can't commit to Pandoc's flavor of markdown.

There is not a good answer. Markdown is a poor format and there are a number of almost compatible variations. However Markdown in all formats is somewhat limited and so there will be some things you cannot do that if you work on a complex project you will want to work on.

However markdown - if you stick with the subset that everyone supports is the most widely supported alternative. If you go with a specific markdown you lose support for something else you might want. If you go for non-markdown you will lose support for most of the world.

I personally selected restructured text which is really powerful for the complex documentation I'm trying to create. However I keep running into nothing else supports it problems (I can extract doxygen from C++ - but only with tools that don't support the latest. I haven't figured out what to do about Rust documentation)

I tried to use it to make an invoice system: wanted to convert plain-text CSV (description,amount,cost) --> to Markdown tables --> to PDF.

But I was unable to align the following 2nd table with taxes: cells are all over the place and it does whatever it wants. And there is no information online to be found about it.

(I eventually gave up long time ago and still to this day manually do them in LibreOffice Writer adding taxes with a calculator)

Except this, it's a really neat piece of FOSS software!

I tried something like that but ended up going with markdown -> html -> puppeteer to generate an A4 pdf -> ghostscript to compress it.

It’s an ugly script that’s been working quite well for more than a decade, but I wouldn’t recommend it to anyone other than myself.

I have a very similar homegrown mess. I wonder how many of us there are doing the same thing for this use case.

By looking at this thread, quite a few. The problem is creating a solution that would fit all of our idiosyncrasies.

For example, in my code, if a table has the class “total” it sums all <td>s which contains a dollar sign, and so on.

Maybe you could use html as an intermediate point instead of markdown. Might give you more control over the layout.

Might have to use a headless chromium wrapper (maybe pandoc has this anyway) to then get to pdf but that may not be too bad

I was thinking the same thing: I would use HTML as an intermediate, targeting PDF through weasyprint.

In fact I quite often go .md -> .html with pandoc, but write the .md in such a way that, when translated, it is the kind of html that weasyprint will be able to turn into the PDF that I want.

I just recently put together something for invoices that wound up being Jinja2 + data -> HTML-> weasyprint -> pdf. Was quite straight forward, all in all.

I use pandoc quite often - but I wish the intermediate, internal pandoc format was a little more expressive, exactly for things like this. I also tried making an invoice.

Have you looked at hledger, which generates everything from plain-text accounts files?

https://hledger.org/

https://hledger.org/invoicing.html

https://plaintextaccounting.org/

One previous discussion: https://news.ycombinator.com/item?id=20012499

Pandoc is one of my favourite all time tools. As the website says:

If you need to convert files from one markup format into another, pandoc is your swiss-army knife

It also sits at the heart of Quarto[0], which adds Jupyter-like code execution (in R, Python and others) into document production. Combined with RStudio as an IDE, it's my new favourite way to write anything - from static documents to full on code notebooks.

No affiliation with Posit, the company behind Quarto & RStudio. Just a happy user.

[0]: https://quarto.org/

[1]: https://posit.co/products/open-source/rstudio/

Codebraid[0] is another option for integrating code execution with Pandoc. I find myself using Quarto for building sites and Codebraid more for single documents. Both great tools building on Pandoc.

[0]: https://codebraid.org/

Why do all these tools pretend that knitr / R Markdown never existed, and that they have invented some novel concept? I looked through the docs of Quarto, Jupytext, and now Codebraid, and none of them mention prior art.

There's a long legacy here, it does nobody any good to disregard it. Maybe Knuth is well-acknowledged for his invention, but I think for instance Yihui Xie is a little under-recognized.

Is there some way we could advertise this better? The quarto homepage already says “Quarto is a multi-language, next generation version of R Markdown from Posit, with many new new features and capabilities.”

When talking about Quarto within the R community we usually frame it this way, but obviously it’s not a very useful description if you’ve never heard of RMarkdown.

From what I can tell, Quarto is essentially an installer for Knitr/R, that also comes with a bunch of goodies, like when working in VSC (or Rstudio) it auto-suggests cross-references to content in the document, like figures/chapters/equations/etc. It also has a github action that builds and deploys the site in like 1 one line. Just removing the friction and lowering the bar is very helpful sometimes.

Quarto is amazing for creating reports and programmatic slide decks!

Every couple of years I need pandoc for some project. And teach time I relearn the same idiosyncrasies. Some odd defaults, the sometimes annoying depths you have to do to customize HTML templates, the weird filter infrastructure. What a neat strange program it is.

I have this same issue, and the same with `jq` and `GNU Parallel`.

When you need them you need them, and nothing else quite works, but I have to re-learn them every time.

Set a huge number for your shell history and dedup. You’ll effectively save every command you ever typed in chronological order. You can even append comments to the end of the command for your future self.

Then, fzf your history.

I do, and use fzf, and that helps, but with most tools I'm a bit OCD about understanding what I'm doing.

I also use Anki SRS flashcards and put a lot of tool usage exemplars in there; it not only helps me remember a bit of what I need to do, but if nothing else I remember THAT I put it there so I can use that to go look it up again. And this is coming from someone who grew up with and is comfortable with `--help` screens and man pages.

Which is actually quite a strong point for guis, amidst all their problems

As is generally the case, there are official docker images readily available and it's a fantastically light, low-coupling way of adding conversion to a stack.

Recently rewrote a content stack to use Markdown (among other formats) for the source, the file system as the database, generating outputs (including HTML with embedded Mathjax LATEX) via pandoc, and it works absolutely brilliantly. Fully recommend.

Why do you use the docker image instead of just the normal executable?

I'd reckon it's partly habit, but it is a very nice way of "installing" something easily and being able to also very easily _uninstall_ it without leaving any cruft behind, as well as having completely disparate multiple versions of things, if you need that.

For single-file binaries this is less of an issue, but even those sometimes require dependencies that you already have something else that needs a different version of that same dependency causing conflict.

What distro doesn't have pandoc in its package manager?

A better question is why wouldn't I use the docker image? The docker image is an official work output of the project, handles all dependencies without messing up my target machines (I use the pandoc/extra which includes pretty much every ancillary need such as Latex), and is trivial to keep up to date and current holistically. It is by default isolated and controlled to a degree, and allows me to trivially tape together as necessary.

The other comment nailed it pretty well, though they hedged it by citing habit (presumably to counter the weird anti-docker trend that has arisen). Dockerizing (or simply containerizing) most vendored products is a choice that is often beneficial, and the marginal overhead is a rounding error.

Pandoc is a great cli tool in terms of its UI and code quality. An interesting fact is that its creator is John MacFarlane, a Philosopher [0]

https://en.m.wikipedia.org/wiki/John_MacFarlane_(philosopher...

code quality

It has 995 open issues in its Github repository. Haskell program was supposed to work right if it compiles, wasn't it?

It has 995 open issues in its Github repository.

This is not a sensible metric for code quality. For one thing, only about 20% of the currently open issues are tagged as bugs - more than that are suggested improvements.

Haskell program was supposed to work right if it compiles, wasn't it?

No. Especially for tasks like string manipulation and format munging, you cannot capture the complexity of the domain into types.

20 % of 1000 is 200 . Pandoc has been stuck at this number of issues for years.

> Haskell program was supposed to work right if it compiles,

> wasn't it?

No.

I seem to remember differently.

And a damn fine bluegrass fiddler [0]!

[0]: https://www.whiskeybrothers.net/wb_bios.html

I wrote pretty much all my university work in Markdown with inline Latex, using Pandoc to generate a pdf. I'm sure there are things you can do in "pure" latex that you can't do this way, but for most normal cases, it's so much easier. You just use the simple Markdown syntax for basic text formatting, and then can use the power of Latex when you need to display mathematics, graphs, tables or similar.

I'm doing that too... did you render against a template file in the end?

I also went ahead and set up my root "notes" folder to be served with mkdocs so I can easily browse them, and just render a PDF when I must submit some file to a third party

I used a vim template plugin (I believe called vim-templates, believe it or not) to get some YAML and formatting boilerplate, and then I think I rendered against the default templates. I do remember having to adjust the defaults for some reports, though.

Me too :-) It was a steep learning curve but worth it. I used it 'within' VSC.

The only issue I have with pandoc is the dependency hell it requires. It requires a lot of haskell dependencies and just eats up your storage space.

The pandoc binary I have is certainly large at 206MiB (more than I expected!), but it doesn't have any weird dependencies I can see. Just GMP, ncurses, and such. All the Haskell parts are statically linked, which is probably the reason it is so large.

Arch Linux is linking dynamically, IIRC, and there it is 'only' 64 MiB: https://archlinux.org/packages/extra/x86_64/haskell-pandoc/

The author is great - smart and responsive.

He's inspired a small number of long-time, serious contributors.

Together they maintain a super-high-traffic utility that simplifies the very arbitrary complexity of document formats for untold millions of users.

It's really a stunning example of social good.

Besides that, he's a tremendous philosophy professor. I took a class with him years ago. Besides being brilliant, I found him funny, approachable, and never condescending, which is rare for philosophy professors of his renown.

Has anyone had success with running pandoc in a browser/compiling it to webassembly?

I haven't had time to look into it a lot but I think that would be amazing.

Well there's this: https://github.com/y-taka-23/wasm-pandoc I tried it some time ago and it worked quite well

Worth noting that the author has also created a markup language, djot.

https://github.com/jgm/djot

Prof. MacFarlane is also one of the commonmark maintainers.

I have develop a commercial data wrangling tool (Easy Data Transform) that outputs data tables to CSV, Excel, markdown, XML, JSON etc. It would be neat to support output to some of the other formats supported by Pandoc, such as DOCX, PPTX, PDF, Latex etc.

Has anyone tried integrating the Pandoc command line tool into a desktop product on Windows or Mac?

How big is it?

What are the licensing implications of shipping it with my (closed source) software?

(not a lawyer, not legal advice) With Pandoc being GPLv3 you can't just link it with your software. Distributing pandoc standalone (different, unmodified binary) and calling it from your software should be okay as they're technically different programs. Probably requires further investigation and maybe a lawyer though.

Pandoc is amazing. I think of it kind of like ffmpeg for text files...

great analogy

Pandoc is the FFmpeg of document conversion!

While I am very fond of both tools, Pandoc's command-line interface seamlessly integrates with my understanding and intuition, unlike FFmpeg's byzantine command line options and concepts. With FFmpeg, I frequently find myself documenting specific incantations and recipes in my notes, lest I should forget how to solve certain conversion problems. Pandoc, on the other hand, has spared me from such cognitive overload.

This should not be taken as criticism of FFmpeg though. FFmpeg is solving very complex problems too. Both Pandoc and FFmpeg are excellent tools. Both tools have saved me hundreds of hours of research, experimentation, trial and error, etc.

Absolutely love Pandoc. I used it through my undergrad to take notes for all of my courses. Markdown with inlined LaTeX just made sense.

It made university more accessible, as I get frequent hand cramps while writing notes. So I started to take them with Pandoc and added custom macros.

The best feeling was when professors would ask for a copy of my notes at the end of the term because they were formatted so well!

Yes, I know.

I see a Haskell project, I upvote.

I recently switched to Pandoc to parse my Markdown documents for my blog. Up until then I used the original Perl script. It’s not slower and I get syntax highlighting for my code snippets. My only regret is not switching sooner.

Maybe one day I’ll write a blazing fast Markdown parser that does exactly what I want, full control and maximum simplicity and all that. In all probability though, I won’t. I really like what little I have seen from Pandoc.

This is what I used to typeset my novel. My editor and publisher tried multiple methods and they kept being impressed by how clean and "just right" the versions I sent them were... In the end, we ended up using the PDFs generated by Pandoc instead of InDesign or whatever proprietary stuff they used.

I mainly use Typora for markdown editing and saving as PDF

Just check pandoc installation and get started guide. Overwhelmingly detailed, like I never use command line before. Maybe it assumes many users of pandoc are writers or Information Developer, without much tech skills.

I love Pandoc. I wish the python library were a little more accessible.

I've been (slowly) writing a book for about a year and a half now probably, and Pandoc has made it so easy. I write everything in Markdown and use a seven-ish line Makefile and that's it. It generates the PDF and EPUB both extremely well and the customizability of everything in the generation process is fantastic.

I end up using it for random doc generation too in my day to day. It's just a damn fine piece of software that always works and has every feature I could dream up for my use cases.

I used Pandoc recently to convert a large Word document into markdown. It took a lot of babysitting and manual tweaking to get to the final result. Overall it probably still saved me time compared to doing the conversion manually, but only just. YMMV.

Love this, been using it for years to write markdown as a base and then transform to html or pdf using latex for maths.

I love Pandoc!

I recently learned you can use LUA to write custom plugins and change some of the converting behavior. I'm using it for example to create slides similar to the "sent" program.

It helps me bootstrap new presentations and talks very quickly: https://github.com/KarimJedda/justslides

Recently I learned pandoc can act as a web server, and that the type system of Haskell itself provides a strong security guarantee that nothing will ever be written to disk, because nobody understands the IO Monad. Security through obscurity!

Pandoc saved my ass so many times when I worked in research. I would write a beautiful typeset paper in latex and then have to send a colleague a word doc.

You can turn any file into anything. PDF to rtf, latex to .doc, etc. It does a great job. Written in Haskell, too!

One of the best pieces of software you’ll ever use. And if you do find a bug, the maintainer will fix it within hours of your Github report.

The latest website I was asked to build is just Pandoc + Make. It works super well, it's very fast and decently flexible. I even do blog-like processing by just calling Pandoc several times.

I really like using pandoc as a build system [1] for my personal website to convert .md to .html. I can use templates, automatically generate a table of content and run some lua scripts to get the desired result, such as clickable headers.

[1]: https://github.com/furiousteabag/asmirnov.xyz/blob/master/bu...

the svg on the side of their page is kind of hilarious

though I've used this program too in the past to seamlessly convert html to text to easily import into a database...

I had a one-off requirement for a slide deck for something, and I really don't think in that form - so I ended up writing a detailed outline in markdown, using pandoc to turn that into pptx, and then checking that in libreoffice. Worked surprisingly well; I didn't have a style to conform to (so I don't know how well libreoffice handled those) but the defaults worked out.

I've been using it to generate Docx/PDF from my markdown. Just started to explore means to customize (with tex template). Recent updated obsidian plugin requires explicitly configuring pandoc with latex which started the rabbithole customizing experiment :)

This is sick!

I use pandoc with some Make scripts to generate the epub of my novels and short stories. Having a reproducible way to iterate from source to final docs during edits and correction passes is amazing. I can't imagine doing it any other way.

Huge fan of Pandoc. I don't use it for my personal website anymore, but I created a very crude "site generator" that piggybacks off Pandoc called pblog[0].

[0]: https://pblog.btxx.org/

I use it to generate PDF/EPUB versions from GitHub style markdown for my ebooks. The default output was good enough, but I wanted to customize a few things [0]. I didn't know LaTeX, but I was able to use solutions from stackoverflow sites. Later I found that some users had created templates I could've borrowed.

I use mdBook [1] for web versions though. I found the default setup much easier to use. And it came with themes (light/dark/etc) that readers can choose.

[0] https://learnbyexample.github.io/customizing-pandoc/

[1] https://github.com/rust-lang/mdBook

Such a beautifully effective tool.

The static site generator for my personal site is just it with a thin bash script wrapper: https://git.sr.ht/~kb/open-notes/tree/main/item/.build.sh.

So much simpler than pulling a universe of node modules.

Pandora is awesome. Add a self published author, this is a key tool in my tickets to have a single source of truth and (relatively) easily create beautiful PDFs and EPUBs.

I previously used restructured text and had to write custom tooling, but now I can write markdown on Jupyter.

A few weeks ago I wrote a MediaWiki extension which uses it to convert documents to wiki articles

https://m.mediawiki.org/wiki/Extension:PandocUltimateConvert...

Pandoc is amazing. I'm an architect so I use a lot of InDesign. But of course it's not good for intensively editing the text itself, it's a presentation tool. Everyone I know just uses Word and copies and pastes. The poor souls.

I learned LaTeX late in life for a few reasons: - I publish in journals that all have different layout requirements, so reformatting to submit to more than one is a big pain. - Using a non-WSYIWYG editor enforces clarity. It's a lot easier to see each sentence as a whole. If a sentence is longer than a single line in VSCode it should probably be more than one sentence. - The features of an IDE allow you to see each line (sentence) in one place, and move things around more flexibly with the IDE shortcuts. You can easily rearrange the flow of a text. And of course you can make inline comments without having to clean them up before sending the doc somewhere. - I don't have to care about things like image placement and anchoring in the text until output. This is an area that WYSIWYG editors like Word are particularly painful.

There are other reasons like equations and notations, but that's enough for now. All that said here's my workflow:

I write LaTeX in VSCode (soon switching to vim). Then, I can use Pandoc to convert to Word if I need to (it's still where most of the templates come from in the discipline). This is also helpful in working with collaborators in my area since they typically won't know LaTeX.

Here's where it gets fun - I write LaTex, then use Pandoc to output directly to .icml (InDesign Markup). These link directly in my InDesign document, and so I can edit text where it's better and more clear to edit (IDE), then seamlessly get it into the environment where I have maximum layout control. I haven't gotten to do this so many times that I need to write a script to automate conversion as part of my tex build, but I probably will soon for fun.

Pandoc just works so well, allowing me to concentrate on making good content and not having to sweat all the annoying file conversion details. Thank you to the developers and maintainers.

While looking for Pandoc + Make for Website templates, I stumbled on the Website of Jilles van Gurp[1] (hi @jillesvangurp[2]) and I have seen it evolved over time. It is beautiful[3].

1. https://www.jillesvangurp.com

2. https://news.ycombinator.com/user?id=jillesvangurp

3. https://github.com/jillesvangurp/www.jillesvangurp.com