How to Write a Book in Emacs, Hacker News

ByMickey Petersen

Writing a book is a time consuming process, and if you’re going to dedicate a year, or more, of your life to it, it makes sense to pick the best tool for the job. If you’re technically inclined – and if you use Emacs – then the choice is obvious.

As I recentlyfinished writing a book on Mastering EmacsI figured I would share my thoughts, and ideas, on how to write a book – specifically my own book – in Emacs.

The traditional way

The traditional model of publishing involves aPublisher; aWriter(that’s you); anEditor(a real human being); and, more often than not, Microsoft Word. There are many good reasons why a non-technical person wouldwantto write a book in Word:

It’s ubiquitous

So sharing documents is easy and so is finding a computer with any version of Word installed on it.

Publishers and Editors expect it **** It’s What Everyone Uses. So even though you really,reallyprefer your dinky old copy of WordSTAR – well, tough luck. They want your manuscript in Word anyway.

It has a battery of features

And it’s not all bad, either. Spell- and grammar checking is excellent; the track changes and annotation functionality is handy; it’s WYSIWYG, so it’s easy for someone to format; and if you’re fancy, you can evendiffdocuments.

The irony of course is that, once your manuscript’s written, a crack team of Santa’s Helpers spend days extricating your text from Word so they can typeset it in Adobe InDesign. It’s even funnier when you consider that a lot of techie-types will write the book in Markdown, docbook, or reStructuredText only to paste it into Word.

But that’s all beside the point.

When you write a book on Emacs it would behypocriticalnot to use Emacs. Well, it so happens Emacs is the Right Choice for this anyway.

The Emacs Way Picking a source format

When I decided to write the book I wanted it in both ePub and PDF format. My personal preference for technical books is PDF or print. But not everyone shares my enthusiasm for PDF files, so I also had to pick a toolchain that would allow me to output, at the very least, ePub.

The most obvious choice is LaTeX. Indeed, the final PDF output is typeset with LaTeX. But LaTeX is a rather baroque, and turing-complete, format; If you stray too far off the beaten path (or if you get too clever) the LaTeX-to-ePub generation would suffer. I’d have to reach in and manually fix things. Knowing how cumbersome mapping one source format to another can be, I decided not to write the book in LaTeX.

Ultimately I settled on reStructuredText. As a Python developer I knew it already, and as I have an aversion to Markdown – nobody can seemingly agree on a singular specification for that format – that ruled it out, even though it is far more ubiquitous.

So the idea was I’d write the book in reStructuredText and convert it to ePub and LaTeX usingPandoc.

Another option is, of course, Org mode. Unfortunately converting Org mode from one format to another made it difficult to work with. I could’ve exported Org mode as HTML, but like LaTeX there are many ways of doing the same thing. Plus, reStructuredText has a clearly-defined specification, and thedocutilstool to fall back on if Pandoc failed me. I didn’t want the hassle of switching formats mid-way either, so I went with reStructuredText.

============= First Steps=============.. epigraph ::    I use Emacs, which might be thought of as a thermonuclear word processor.    - Neal Stephenson, * In the Beginning ... was the Command Line *.  ...  In Emacs, there are several modifier keys you can use, each with its own character:=================Modifier Full Name=================`` C-Control '' `` M-'' Meta ("Alt" on most keyboards) `` S-'' Shift=================
Here's a snapshot of what the reStructuredText looks like. Very clean, and very readable, especially with syntax highlighting.
Proofreading and Tracking Changes
The next thing I had to worry about was proofreading. I wanted the book professionally edited, so I had to hire an editor. But I simultaneously did not want to deal with Microsoft Word.
Instead, I'd send the manuscript - the raw reStructuredText file - to the editor, and she would edit it as though it were her own manuscript. No track changes. If she had comments, she'd insert it above the paragraph, like so:
SUE: You should consider rewriting this paragraph.  When you use the command `` Mx toggle-truncate-lines '' [...]
She still wanted to use Microsoft Word, so I inserted the raw text into a Word document, switched it to a fixed-width font, and told her to disable Word's “helpful” AutoCorrect, as it'd mangle the reStructuredText formatting.
********** When she finished with the manuscript I'd use Emacs'smx ediff-buffers  to ediff the corrected and original source manuscript, in effect usingdiff(track changes. Now you might be wondering: what about the comment paragraphs? Wouldn’t they screw up the diffing? Why yes, yes they would. But Emacs'sediffis so advanced that you can tell it tohide regions match patternsby typing  (# h) ************************ in the  Ediff Control
and enter the following regexp:
^ SUE:
And Emacs would disregard lines beginning with that pattern. Extremely useful, as I could read the comment and modify my manuscript according to the instructions given, while I also merged her changes. It's hard to make out from the screenshot above that it works, and there were a couple of times it generated a too-complicated diff, but that took up maybe 728 minutes of my time, out of 4 hours or so of reviewing and merging her changes.
There were a few niggles such as the unicode character forNON-BREAKING SPACEnot reading right, but I chalk that up to me using UTF-8 and her Windows machine usingiso -  - 1. Again, with a regexp replace, fixing it took maybe 2 minutes. No problem.
So, all in all, I got to have my cake and eat it: I wrote the book in reStructuredText, had it proofread in reStructuredText, and corrected it in Emacs using ediff.
Spell Checking
This is one where I wish I had Microsoft Word. I usedMx flyspell-modeto track misspelled words as I type, and I tweaked(aspell) ************************ ((the commandline spell checker Emacs uses in the background) so it'd use American English instead of my native UK English  (LOCALE) :
************************** .. - * - mode: rst; encoding: utf-8; ispell-dictionary: "american" - * -
****************
But
is an adequate, not great, spell checker. It’s fine for source code comments and minor README files, but for a book I found that it missed too many things, and used awkward hyphenation rules. Still, though, in the grand scheme of things it worked well. And being able to typeCMion a misspelled word and have Emacs auto correct it was certainly useful.


Jumping around

Jumping between chapters was very easy thanks to Helm'sMx imenusupport.mx rst-modebuilds an imenu index of all the chapters and sections, making cross-chapter navigation easy.
Pandoc
Pandoc's
great - it really is. But I still had no end of trouble with it. First of all, there is no documentation. There’s a man file and a smattering of examples and a few scattered blog posts explaining how to do basic things, but that’s it. You might be thinking: how much documentation does one need? It’s a converter. Well, when you’re trying to work out why certain formatting rules in reStructuredText aren’t applied in the LaTeX output, you do care. You care a great deal, in fact.
Ultimately I had to delve into the source to figure out why some things worked and other things did not. As it turns out, reStructuredText is notfully supportedlike its markup cousin,Markdown; or the red-headed stepchild of Markdown,Pandoc Markdown, Pandoc's own take on Markdown.
I found this part exceptionally frustrating. That one format or the other is not equally supported is totally fine, but I wish they’d spelled out what it could and could not do. I may have picked a different source format had I known - or maybe not, who knows?

	

			
	

			


		
			
			
					
			


For instance, in reStructuredText you can annotate text withroles. There's a handful in the specification, but I wanted one for small caps, like so:
: small-caps: `GNU` (**************** I discovered Pandoc converts everything into a kind of Abstract Syntax Tree in JSON format. So I was thinking: great, I’ll use Pandoc’s ability to call out to Python with each token from the AST and then hand-generate the LaTeX (and HTML for ePub) that I need. Nope. Not gonna happen.

It turns out that Pandoc “helpfully” strips roles and directives that it doesn’t recognize. So I figured: let’s hack the damn thing so it does what I need. But then I learn it’s written in Haskell. All right, I thought, the parser’s clean and easy to understand, let’s get a Haskell dev environment set up. Cue 8859 MB of library cruft I had to download to evenrunthe damn thing from source. I got flashbacks of bower, and npm, and the entire JS ecosystem and decided, in the end, to bard the reStructuredText source file with custom markers that I then map into LaTeX and HTML with (sed)
Still, Pandoc's a great tool, and I am glad it is there and that I got to use it. But although they support a wide variety of formats you best steer clear of all but the most maintained ones. Were I do to this all over again I would usedocutilsto spit out HTML (the LaTeX it generates is OK but not great) and then convert that to my target formats.
(Outputting LaTeX)
Or should I say, XeTeX. I needed XeTeX because the amount of METAFONT fonts available is vanishingly small. XeTeX also supports unicode, which is rather important when you want a PDF format that is as close to plain text as possible, for accessibility reasons.
XeTeX works great, and LaTeX Is not a difficult thing to write. Whatisdifficult is customizing LaTeX. Enabling old-style numerals (lowercase numbers) is easy enough, right up until you want toonlyenable them in some parts your book. Fact is, fiddling with LaTeX is time consuming but, to me, very important, so I spent the time needed to make it look as good as I could.
LaTeX is a great typesetting system, and I could make the book lookeven betterif I had focused solely on PDF. As it stands, the lowest common denominator (ePub on Kindles, but more on that in a second) held it back. I wanted fancier tables and numbered, vectorized arrows that shows how the point moves through text instead of the simpler narrative format I adopted in the book. But I couldn’t do that in reStructuredText and Pandoc. A shame. Having said that, I am very happy with the PDF output. On a High DPI screen the book looks fantastic and I have received nothing but positive feedback so far.
(Outputting ePub An ePub file is a zip file with a CSS file, an image of the book cover, a metadata file and a bunch of HTML files. That’s it. And yet, I found it extremely hard to make it render properly on Kindles. To render HTML and CSS you need a browser engine, and unsurprisingly the engines on a lot of Kindles out there are pretty outdated and not very CSS compliant. Things like spacing between paragraphs instead of indented opening paragraphs vary. I cannotstandindented paragraphs and I went out of my way to disable it in the PDF version but it sadly lingers in some, but not all, Kindle readers. So converting to ePub is not a turnkey affair, even though it certainly seems that way from the outset.
I'm glad I added ePub support but it was a lot of work to make it look okay on old versions and good on new ones. So far, I’ve had no complaints. Touch wood.
Compiling) the books
I use a simple Makefile to generate the output formats and in the case of LaTeX the intermediate.texfile so I can apply somesedscripts to it. From inside Emacs I usedMx compile

to run themake command and display the output. The Makefile itself was obviously authored in Emacs also.

Combined withinotifywaitin a Makefile rule I could seamlessly update the final deliverables as I typed. In the end I realized that I did not need that quick a turn-around on the output files as I am a habitual file saver and did that far more frequently than I reviewed the output. I could also do it with a save hook (or even aninotifywaithook) in Emacs.
With Emacs's ability to browse PDF files I could review the book inside Emacs, next to the reStructuredText file, which came in handy occasionally.
General Editing



Throughout the course of writing the book I spent the entire time editing and moving and commands that work on sentences (Me (***********************,
(Ma) ************************, (Mkand paragraphs (******** (M - {, (M -}came in handy often. Part of what made my book editing less tedious was the elimination of superfluous stuff like fiddling with fonts and styling - a task relegated to simple markup in reStructuredText and settings defined by me for the actual typesetting itself.

I often found myself abusing certain turns of phrases and Emacs's custom highlightingMs hp would highlight, in garish black on yellow, such occurrences. The book was obviously stored in source control with Git and Magit

.
On the whole, the wholeety of the book, from conception as a series of notes and TODOs in Org mode, to the editing and merging of changes made by the proofreader took place in Emacs. The only tool I did not have in Emacs was Amazon's Kindle Previewer tool.
Conclusion

Emacs is a great tool for book editing and writing. I used reStructuredText text but Markdown, ASCIIDoc, Docbook or any number of formats would work equally well. Emacs’s ediff is a powerful diffing tool that made it possible for me to track the changes made by the proofreader and selectively apply, but also edit, the changes she made before purchasing them to the original manuscript. I think that feature alone made it worth writing it in Emacs.









              (**************************************** (****************************************
****************************** (Read More) ********************************************************