In this post I will present my final solution for getting the Bible imported into Roam Research. I have finally found a way to drastically decrease the size of the imported Roam graph, but still have a reasonable solution for accessing multiple translations and cross references. I will share my thoughts on performance and potential next steps for making the Bible work even better in Roam. If you are interested in the topic, I have two other posts on the subject. Links are at the end.
- You can download the Roam .EDN archive for the Bible from here: ASV.edn
- For a real time demonstration of the full import process within 3 minutes check out this video.
- WARNING: by restoring the ASV.edn into your graph, all other data in your graph will be lost!
- The source code for the converter is here: Roam.json Bible importer with BUTTON for verse numbers v1.ipynb
- Use this if the primary translation you'd like to use in Roam is not the American Standard Version. Also you can tweak how the verses are formatted and whatever else you might want to change on the import.
If you haven't watched my other video about the Bible in Roam, I recommend watching that also. There I briefly show how you can add your own notes into your Roam Bible, and how it can become a powerful Study-Bible where you'll be able to find your notes and reflections when you revisit a verse in the future. Follow this link: Roam Bible Tour - YouTube. Also be sure to read my other post as it explains some of the logic for how I intend this Bible to be used: Study Bible or ePub Books in Roam? My Roller Coaster Ride with Roam JSON
Roam is not ready for handling a large body of text
In its current state Roam does not seem to be ready for handling a large text-corpus. In my previous attempts with importing the Bible I tried loading 33 000 pages and 96 000 blocks into Roam. The loading process took days... and as I was reaching 70 000 blocks in the graph, the import speed dropped to well below one block per second. More concerningly the graph took minutes to open and was very sluggish to work with even on a reasonably powerful desktop PC. Access from a tablet or phone were simply out of the question.
Based on this experiment, my conclusion is not to load full-text books into my main graph. I also advise keeping the Bible in a separate Study-Bible Roam graph, ring fenced from your other notes.
Additionally, the local Roam graph is distinctively slower than online Roam. While this version of the Bible works well in an online Roam database, working in a local graph I ran into performance problems.
Scaling down the size of the Bible graph
I did two things to drastically reduce the size of the Bible graph.
1. No pages for verses
I decided against creating a page for each verse. e.g. [[Genesis 1:1]]. This approach seemed convenient for linking multiple translations of the same verse and to store cross references, but also accounted for ~32000 pages and ~32000 blocks in the graph. Since I am only importing a single translation this is not need.
2. No cross references
Cross references created enormous number of backlinks in the database and also accounted for an additional ~32000 blocks. Cross references are also not crucial to the Bible-study workflow in Roam. Also, there are plenty of powerful search tools on the internet with extensive Bible cross references. Where Roam shines is not the cross referencing of the text-corpus but the cross referencing of your own notes built around that text.
Lean approach to accessing other translations, searchability and more...
I believe the ability to search for a verse is important. You want to be able to enter "MAT 1:1" and find Matthew chapter 1 verse 1. Easy access to different translations of a verse would still be preferred as well as access to cross references.
I solved this challenge by adding the verse reference for each verse as a custom Roam button instead of a page link to the verse: {{1:BIBLE:MAT 1:1}}. This approach provides almost unlimited flexibility by putting what ever logic you may want into a {{[[roam/js]]}} script. Also the text in the button is indexed by Roam, thus entering "MAT 1:1" in the search bar yields Matthew 1:1.
For now, I created a simple event handler that opens up bible.com at the verse specified in the button. This script can be easily further customized to pull up cross references either from a file on the internet using roam/js, or to call an internet service for cross references. Your imagination and coding skills are really the only limit.
Here's my script. Note Roam42 is a prerequisite. Also the script is largely based on code I took from Roam42. Thanks @RoamHacker! You need to put this code into a code-block nested under {{[[roam/js]]}}.
window.bible = { async event(e) { if(e.target.tagName=='BUTTON') { var block = e.target.closest('.roam-block'); if (!block) { return; } var blockInfo = (await roam42.common.getBlockInfoByUID(block.id.substring( block.id.length -9)))[0][0].string; if(blockInfo.includes(e.target.textContent + ':BIBLE:' )) { const regex = '{{\\d+:BIBLE:(.{3})\\s(\\d+).(\\d+)}}'; var v = blockInfo.match(regex); if(v.length == 4) { window.open('https://www.bible.com/bible/547/'+v[1]+'.'+v[2]+'.'+v[3]+'.KJVAE','_blank'); } } } } } document.addEventListener('click', window.bible.event);
Hi am interested in getting the NIV version of the bible into my Roam rather then the ASV one. Is this something you could assist me with. Amazing stuff by the way. The work you have done to get the entire bible into Roam looks immense. I am not that confident with script etc so hopefully you can put me on the right path. Thanks
ReplyDeleteHi Jesse,
Deletewould be happy to help, however as much as I can see the NIV version is not in the public domain. I tried Googleing it, but I always get back to the 10 translations available in the biblehub file.
If one of the 10 works for you, that is easy to generate - I can also load it for you... though it takes a few hours.
0. King James Bible
1. American Standard Version
2. Douay-Rheims Bible
3. Darby Bible Translation
4. English Revised Version
5. Webster Bible Translation
6. World English Bible
7. Young's Literal Translation
8. American King James Version
9. Weymouth New Testament
If you have an epub version of the NIV you can try the epub converter (https://www.zsolt.blog/2020/12/read-books-in-roam-detailed-how-to.html) Because of the size of the Bible, the ePUB converter will need a slight modification to cut the .json files into smaller pieces so Roam can handle the import - I am happy to help with the change if you've got the .epub.
Amazing thanks so much for getting back to me. I think I have found an ePub version. I will try and track down your email to send that to you.
ReplyDeletewhat is your email?
ReplyDeleteYou can DM me on Twitter @zsviczian https://twitter.com/zsviczian
DeleteAlso the info@zsolt.blog should work as well. It filters emails with .zip and other potentially harmful attachments. Best is if you send a link to your ePub on either of icloud, google drive, dropbox, onedrive, etc.
Hi Zsolt - I use Roam Research for my research - but hadn't though of using for bible scriptures - what a great project. Unfortunately, I am rather naïve when it comes to the techy - programme stuff part of your approach - in particular the creation of edn format. I see you offer help with this. I would like to use the ESV - which is on BibleHub. Are you able to help.? Russell
ReplyDelete