Study Bible or ePub Books in Roam? My Roller Coaster Ride with Roam JSON

I don't give up easy. After some mixed results the first time around (see my previous post: My Adventres with Roam.JSON) I have now successfully created a full featured Study Bible in Roam by importing the Bible and cross references. I have also created an ePub importer and loaded War and Peace into Roam as a pilot. This may be helpful to those of you who want to integrate your reading experience fully into Roam.

This post will be a mix of technical details and a demonstration of the Study Bible I have created. I'll start with the introduction of the end result - you can bail out after that if you are not interested in the gory details.

I decided to publish this post even though I am still not 100% satisfied with my solution. I have lots of further ideas to explore, which I will share in Part 3 on Roam JSON. There are still many open questions and I have lots of additional ideas to explore. I'll provide a list of some of these at the end. 

• My Python scripts for generating roam.json for
• the Bible is here: Link to Colab Notebook 
• ePub books is here: Link to Colab Notebook

Since the writing of this post I have released the Roam archive Bible.EDN that you can load into your database. I still recommend reading this post, as this explains much of the logic and use-cases. Once you are done here, I recommend heading over to my next post Importing the Bible to Roam - Final Solution

Study Bible key requirements

I am gifting Roam to my brother this Christmas. I thought he'd enjoy using it for Bible study. 

My key requirements were to import two translations and to include a full cross reference. He prefers King James and the Hungarian New Translation.

Additional requirements include a responsive user experience including on a high-end phone or a decent tablet.

The primary use-cases for the Roam based Study Bible include note taking during daily meditations, conferences, bible studies and preparing sermons. In all cases I imagine he will reference a set of verses from the Scripture and add his notes to them. Doing this in Roam will show its benefits over time, when Roam, through its efficient block reference structure will surface past notes regarding the Bible verse he is studying. Using Roam he will also be able to add additional layers of references such as topical indices and multimedia such as pictures, presentations, etc.

ePUB converter key requirements

The benefits of reading a book in Roam comes from integrating the book into your research or daily notes. Based on my experience of importing the Bible, I am currently unsure as to the limits of Roam. I have the feeling that even if you only read one book a month, by uploading all those books to the same graph, Roam will not be able to keep up with performance. You'll probably hit a barrier before the end of your first year. 

An alternative to loading each book to your main graph is to load them into separate local graphs. This may give you an opportunity to experiment with this new reading approach, but will lead to a fragmented landscape of Roam databases.

Therefore I make the assumption that you will very selectively load books into your graph. Maybe an important reference material, maybe a single translation of the Bible, maybe some other religious text.

The script I developed for converting ePubs to Roam JSON provides you with an example of loading a fairly large and complex book: War and Peace by Leo Tolstoy. I did not set out to create a universal ePub converter script, but rather to provide you with a toolkit, one that with limited understanding of Python you can mold to your needs.

The script can generate a table of contents for the book, has the basic capability to identify chapters and sections in the book, but this all very heavily depends on the ePub in question.

How big is the Bible?

The Bible consists of 66 Books, approximately 1200 chapters and 32000 verses. Each verse has a set of cross references. I have decided to create a dedicated page for each verse and place the cross references there (see video further below to understand). My alternative was to place the cross references nested under each verse, but this approach was less efficient when multiple translations of the text were in the database. Cross references on the verse pages add an additional ~32000 pages with ~32000 blocks of text.  

The traditional way to reference Bible verses is through the [[Book chapter#:verse#]] structure, e.g. [[Genesis 1:1]]. Each verse starts with its own verse reference taking you to the verse page that has the cross references. For convenience I added an additional block to the verse page with the ((block reference)) to the verse. This helps in the process of navigating cross references, by reducing the number of clicks required to reach the actual text of the verse. Having these block references on the verse page add an additional ~32000 blocks.

Adding this all up, the Bible takes up ~33000 pages and ~96000 blocks in Roam. Adding a second translation increases this with roughly 1200 pages and 64000 blocks for the verses and the verse block references on the verse pages. Cross references don't need to be duplicated for the second Bible.

How big is an ePub?

The simple answer is that it depends. War and Peace is only 11689 blocks long. This is good news, because it is one of the thicker books and it is still a fraction of the Bible (especially if you consider loading multiple translations of the Bible). Additional benefit of books in general vs. religious texts or other highly referenced literature is the lack of extensive semantic cross referencing. Not having references saves lots of space.  

How does Roam Bible look?

What about performance?

It takes about five to seven hours to import the files for one Bible including the cross references. Since there is not much to do during import, just to click on the screen in about every 30 minutes, this is not much of an issue. You can read, cook, do household chores, etc. while your PC does its job. You only need to do this once! After the Bible is loaded from JSON and exported to Roam's native EDN format, restoring the graph to a new database is a matter of just minutes.

I measured json import speeds and experienced a gradual slow down as the size of the graph increased. The chart below is the result of a single set of measurements and can't be considered scientific by any stretch of imagination. To me this chart seems to suggest that there is more to this than just throttling of upload speed as Conor mentioned on Twitter. In my read there is something structural to the underlying algorithms. I can't really look under the hood, but I can attest that the chart matches how the process felt like.

Interestingly it feels faster to load small chunks of JSON's (i.e. 100 pages or ~3000 blocks per file), then to load few bigger ones (i.e 200 or 300 pages in size). I did not use a stopwatch to validate this perception, maybe it's just the more frequent fiddling with the PC that tricks me into feeling things are moving quicker when slices are thinner.

Loading has four phases. In phase 1 you select the files to be uploaded. After couple of seconds a list comes up with the files that will be imported. Phase 2 starts when you click import.  A spinner is running on the load screen. Phase 2 can take between 2-3 minutes up to 15-30 minutes. In my experience its best not to touch the mouse after hitting import, until the spinner screen is gone. Moving the mouse will trigger a warning from the browser (I tried it with chromium browsers) that the page is not responding, and a watchdog will eventually kill the process. In phase 3 Roam seems dead for a long while, the indicator for pending local changes is red, but the number of pending changes counter does not move. This phase takes another 10-30 minutes, but in case of larger datasets this third phase can take up to 2 to 3 hours. After about 3 hours I got an error that the page is not loading... Finally in phase 4 the pending local changes counter starts spinning. Once your graph gets large you can visibly distinguish each number as the counter decreases. 

Overall my advice is: Slow and steady wins the race! I tried multiple time uploading larger sets during the evening hoping to make more progress by the morning, only to find my browser with a timeout message when I woke up. 

The performance picture remains mixed once the json was imported. With one Bible translation and the cross references in the Graph the performance of Roam is just borderline ok on a relatively strong desktop PC. The Roam Bible does not really work in any meaningful way when accessing it from my Samsung Galaxy Note 8 (my Brother has a Note 10, maybe that will be strong enough) or my Lenovo Chromebook Duet. I takes few minutes to open the graph for the first time, but also during use it feels undeniably sluggish. 

Design considerations

Creating the Study Bible I faced a number of design dilemmas. I've tried at least 10-15 variations till now, but haven't yet found the ideal one. I was amazed at the available number of meaningful permutations of placing chapters, verses, references, translations, etc.

Should I use page links or block references for cross referencing? 

I opted for using page links to individual verses following the format explained earlier: [[Genesis 12:4]] or [[Luke 2:7]], or etc. I decided for this style simply to comply with tradition. This is how verses are typically referenced in religious literature. I also considered that I would reserve the ((block references)) for use in daily mediation and in other situations when the verse is referenced verbatim in text.

Where should I put the cross references? Should I create a separate table of cross references, or nest the cross references under the verses? If I have multiple Bible translations loaded do I need to duplicate references?

My first option was to nest references under each verse. This approach proved to be cumbersome when working with multiple translations. My current best solution is to place cross references on the verse's page (see above). This way even when working with multiple translations the cross references are easily accessible. I also added the X-Refs:: attribute to the beginning of each block containing the cross references. This may be helpful if I want to filter out references from a view. 

Cross references eat up many blocks in Roam and thus heavily impact performance. There are two alternatives I am considering. One is to load the cross references into roam not as blocks, but using javascript and creating a simple popup window that is activated with a key combination to show references. The other alternative also involves a simple {{[[roam/js]]}} script that takes the title of the verse page and queries the internet for cross references using one of the many available such websites.

Should I add a block reference to the verse on the page for the verse?

Verses appear on the page for the chapter. i.e. The verse for Matthew 1:3 is on the page Matthew 1. The verse page is empty - except for the linked and unlinked references.  (note also the picture shows an earlier version when cross references were nested under the verse)

Having a block reference to the actual verse text on the verse page would be very convenient. This is because when I open the page for the verse in the right sidebar (by shift clicking on a cross reference to see what it says), I would see the verse immediately. Without the block reference 3 extra clicks are required. In addition, having the block references to the various translations of the verse listed one under the other on the verse page would provide an excellent way to see multiple translations together.

These are compelling reasons for including the block reference to the verse on the verse page, but in practice, this adds an additional 32000 blocks per Bible translation to the Graph - which is already slow. 

In the current version I opted to include the block references nonetheless as I thought the convenience outweighs the performance considerations. 

Including the block references also proved to be a small challenge on its own. During import Roam reassigns block IDs (I believe that this may be a contributor to the slow JSON load speeds), thus the block references I create in Python using the UIDs I generate do not work once loaded into Roam. My end solution involves loading the Bible JSON to Roam, then exporting it from Roam, loading it into Python to read the block references and creating the verse pages, and then exporting these from Python and loading into Roam...

Should I include a verse reference at the beginning of each verse?

Having the verse reference at the beginning (or end) of each verse is practical, since when querying for example [[Job 23:4]], the verse would come up. However including these long references in each verse hinders readability. I opted for using an alias. i.e. Instead of [[Job 23:4]] I write [4.]([[Job 23:4]]) which looks as just 4. when reading, however hovering the mouse over the full verse reference is visible.

This however creates a large number of pages. Without individual verse reference there would be only 1300 pages (one for each chapter) and 32000 blocks (one for each verse). Adding the verse references adds to the database an additional 32000 pages. This again could be bridged by a simple {{[[roam.js]]}} script which takes the number at the beginning of the verse and takes the title of the page and navigates to the right location using these. One alternative to explore will be to use Roam42 SmartBlock buttons. More on this in my next post.

How should I name pages for the chapters?

I first opted to place chapters in the namespace of the Bible translation. i.e. "[[King James Bible]]/Exodus 3". In the current solution I still maintain the namespace approach but created a namespace without the page link i.e. "KJB/Exodus 3". I am at two minds about which one is the better approach. The first is more versatile, but looks more complicated on screen.

The benefit of the namespace approach shines when loading multiple translations. Then I need to be able to distinguish between the chapters of different translations, else verses would be mixed up between translations. Imagine that "ASV/Exodus 3 'would be the other translation.

The drawback of the namespace approach is I can't simply reference [[Exodus 3]]. Doing this could be another way out of the verse cross reference dilemma of getting to the text quickly.

Other approaches to include multiple Bible translations in the Roam Graph

There are alternative approaches to putting each Bible into its own namespace. For example I could play with the following tactics:

• Create a Heading for each translation on each chapter's page similar to a table of contents, and nest verses for each translation under each heading. This way two translations could live next to each other on a single page separated by their nesting under the different translations
• Genesis 1:1
• King James Bible
• verse 1
• verse 2
• ...
• American Standard Version
• verse 1
• verse 2
• ...
• Nest the secondary translations under the primary translation of the verses.
• Genesis 1:1
• verse 1 in KJB
• verse 1 in ASV
• verse 1 references
• verse 2 in KJB
• verse 2 in ASV
• verse 2 references
• ...

Quo Vadis - Where do we go from here?

My next line of experimentation will be with Roam42 smart buttons. The basic idea is to include the cross references in script. To replace the verse numbering with smart buttons that bring up the references. Also instead of loading multiple translations I will only include a single primary translation and use a service such as bible.com and the smart button to pop up additional translations on demand. This should decrease the size of the Bible to 1300 pages and 32000 blocks making it much more accessible given Roams constraints.

So if you are interested, for now, I would sit tight and wait. Of course, if you are enthusiastic and want to join in the fun, I am happy to collaborate on a solution. Equally I am including my scripts at the beginning of this post so you can play with them at your leisure. 

More about my journey with Roam here

• Importing the Bible to Roam - Final Solution
• My Adventures with Roam.JSON
• My GTD - How I Organize Meetings and TODOs in Roam
• My Book Summaries in Roam Using Tiago's Progressive Summarization
• De Bono's Algorithms of Thought  for Lateral Thinking and Creativity Part 1
• ...and my other Roam related posts

Like this post? Show your support.