BUCKYVERSE

Digital Translation to Ebook

Notes on how to transform a physical book to its best digital self, so it makes the trip to Mars and lives on forever.

Tools

Hardware

My first choice was to build an Archivist Quill. Due to COVID-19's impact on supply chains for electronics, collecting components will take more time.

My second choice was a new product line from Hong Kong that been expanding through crowdsourcing. A CZUR Aura, their mid-range scanner (~$300 USD), was in-stock on Amazon and arrived the next day.

Product Page: https://shop.czur.com/products/aura?variant=28598339436592

Using the Aura is easy and quick. You place the book on the mat, open to the page, and tap the foot switch to capture. Three red laser lines appear across the pages during the scan. When they disappear, you turn to the next page. Our Aura is connected to a MacBook Pro running the CZUR software. The software stores the captured images and then processes them into digital documents, PDF, DOCX, etc.

Update: Have upgraded to a CZUR ET18 Pro

Update: Upgraded to CZUR ET24 Pro

Software

WebStorm (IntelliJ)

WebStorm is an Integrated Development Environment and my current text editor of choice. There are many other options, choose wisely. (Crosses fingers—please don't spark another emacs vs. vi like-war.)

  1. For working with TeX files, I installed the TeXiFy IDEA plugin.
  2. Prettier is a code formatting library and with a plugin, it will format LaTeX documents.
  3. Prettier plugin for LaTeX, lets Prettier format LaTeX documents.
  4. LatexIndent library brew install latexindent
  5. In preferences, I added \*.tex to Soft-wrap these files in Editor > General > Soft-Wrap section. LaTeX prefers on newline per sentence, so soft-wrap make files easier to read.
  6. WebStorm integrates with GitHub, so all changes to the book are tracked with git.
  7. Text is easier to edit using monospaced/fixed-width fonts since visual differences and corrections are easier to spot. I use PragmataPro, but there are many popular fonts for coding like Inconsolata, Fira Code, MonoLisa, Space Mono, Hack, or Source Code Pro.
  8. Setting > Editor > Inspections > LaTeX > turned off warnings for sentences starting on a new line to work better with Prettier formatting.
  9. Setting > Languages & Frameworks > Javascript > Prettier > Selected Automatic Prettier Configuration. Added tex to the list of file name extensions in the "Run for Files" field.

WebStorm

Pandoc

Think of this as the Babel Fish of text documents. One of
the big productivity boosts I stumbled onto was pandoc. Its creator, John McFarlane, is a long-time LaTeX user so support for it is great. Scanning a book into DOCX retains document formatting, that pandoc translates directly to LaTeX. Compared to bringing in plain text and hand decorating lines into LaTeX, pandoc saves hours of tedious, error-prone work.

Pandoc

TeX Live (MacTeX)

To produce TeX «tech» documents, you need to install a TeX environment. On a Mac, I use MacTex. It is a large collection of software, so they offer a smaller download as BasicTex.

TeX Live Utility helps keep all the packages updated.

TeXShop provides a basic text editor and typesets your file into a PDF.

https://www.tug.org/texlive/

Kindle Previewer

Kindle Previewer is how you prep a book for the Kindle Publishing platform. Once completed, books can be sold on Amazon for Kindle or as physical books with their print on demand service. I use it to convert to the latest version of the Kindle format, .kpx, to handle advanced typesetting—custom fonts and rendering math. Since I do not publish to Amazon, the Previewer file goes through another conversion step and is side-loaded to my Kindle—means I connect Kindle over USB and transfer the file from my laptop. .kpf or .kfx file types can't be uploaded via email or over WhisperNet.

Kindle Previewer

To update to latest version: kindlepreviewer -update

Calibre

Calibre «cal-i-ber» is an eBook collection manager with tools for format conversions. It will convert text to html, ePub, Mobi and many other ebook types.

Calibre

Skim

Unlike Adobe Acrobat Pro, Skim will refresh the PDF when it changes on disk. When generating new ebook versions with our script, the PDF is opened/reloaded for review.

The Process

  1. Scan the book
  2. Crop out images
  3. Convert images scans to DOCX (preferable to preserve some document formatting)
  4. Covert DOCX into .tex with pandoc
  5. Add LaTeX macros, convert content, and proofread .tex file in WebStorm (repeat, repeat, repeat)
  6. Typeset .tex file into PDF
  7. Convert .tex to html using tex4html
  8. Convert .html to ePub using Calibre or tex4ebook
  9. Test ePub with ePubCheck and fix errors
  10. Convert .ePub to .kpf using Kindle Previewer
  11. Convert .kpf to .kpx using Calibre
  12. Convert .epub to markdown (.md)
  13. Alter .md into a wiki.js page.

Levels of Completeness (Galley - Editing - Typesetting - Proofreading - Distribution)

  1. Images of text (jpg, tiff)
  2. Readable text
  3. Images included
  4. Footnotes
  5. References
  6. Index
  7. Glossary
  8. List of Figures
  9. List of Tables
  10. List of Equations

Convert Content

  1. Chapters
  2. Images, Figures
  3. Appendixes
  4. Epilogue
  5. Index
  6. References
  7. Glossary

Proofread

  1. Define Paragraphs --- put extra empty line between sentences
  2. New line at end of each sentence --- WebStorm "Join Lines"
  3. Change double quotes " " to open quote `` and close quote ''
  4. Convert . . . to \textellipsis
  5. Space after abbreviations (Dr.\ , Mr.\ , Mrs.\ , Ms.\ , Fig.\ , St.\ , U.S.\ ,
    U.S.S.R.\ , vs., Inc., N.\Y.\ )
  6. Fix scannos, OCR mistakes (
    1. l <-> 1
    2. m <-> rn (bom -> born)
    3. . <-> -
    4. . <-> ,
      • -> ---
    6. die -> the
    7. C -> G
    8. `/ -> Y
    9. id -> ul
    10. 11 -> II
  7. Hyphens (-), en dashes(--), em dashes(--- -> \textemdash)
  8. Symbols --- degrees, pi, math, dots
  9. Lists --- environment for itemize or enumerate
  10. Define small fractions - 1/2 -> \sfrac{1}{2}
  11. Italicized/Emphasized words
  12. Acronym definitions
  13. Glossary definitions
  14. Equations
  15. Tables
  16. Footnotes
  17. Figures
  18. References --- beware of # in .bib entries
  19. Index items and links

Produce eBooks

Determine Scan Image Sizes

magick identify * | cat > images.txt

docx from scanned pages -> LaTeX with images

pandoc --extract-media ./images -s FILENAME.docx -o FILENAME.tex

HTML

make4ht -l FILENAME.tex "htm,pic-align,notoc*" -d FILENAME-html

ePub3

tex4ebook -l -c FILENAME.cfg -t -f epub3 FILENAME.tex

Check ePub file

java -jar epubcheck.jar FILENAME.epub

KPF with Kindle Previewer 3

kindlepreviewer FILENAME.epub -convert

Kindle Version

  1. tex4ebook -l -c FILENAME.cfg -t -f azw3 FILENAME.tex
  2. Open .epub in Kindle Previewer
  3. Export file as a .kpf
  4. Import .kpf into Calibre
  5. Export .kpf as a .kpx
  6. Side-load .kpx to a Kindle e-Reader

Install MacTex on Mac OS

  1. Download from https://tug.org/mactex/mactex-download.html
  2. Check md5 hash to confirm complete file downloaded
  3. Install MacTex.pkg
  4. Update LaTeX packages
    1. sudo tlmgr update --self
    2. sudo tlmgr update --all
  5. Install Ghostscript https://www.tug.org/mactex/morepackages.html
  6. Install a modern version of Tidy to clean up output HTML brew install tidy-html5
  7. Install diff-pdf for visual comparison of pdfs brew install diff-pdf
  8. Install ePubCheck curl -SL https://github.com/w3c/epubcheck/releases/download/v4.2.6/epubcheck-4.2.6.zip | tar -xf -