Creating Notes Site

For my Solutions Architect studies, I have prepared notes for the studies. The notes site is hosted here. How I deployed the notes site on S3 using cloud domains is already described in previous post. In this post we will see how this site is made from simple markdown notes to fully searchable HTML site.

I like writing notes and blogpost in Markdown. Markdown is simpler to write hyper text compared to writing HTML, and powerful to write compared to simple text files. This site, as well the notes are written using Markdown. The notes and the code are open source and hosted on Github.

When I started my studies, the notes are simply there in pure markdown format for some time. When the topics were stand-alone, markdown files were fine being stand-alone as well. I had 5 commits which only contained Markdown files. But very soon, things started to change and I need to connect one section of notes to another section of another file. Also I realized that to make sense of these notes to be read from everywhere, I need to convert them into HTML files and a static site.

Setup

There are many Markdown to HTML converters, including static site builders like Jekyll, Hugo to many scripts and libraries in various languages. I chose not to go with Jekyll or Hugo because at start I thought it’s an overkill, and later on I decided to have a chance to understand how static site generators are built. I chose to write a node script and selected showdown as base library to convert Markdown to HTML, the very first commit of code that does the trick looks like this.The code in convertHtml.js lists all the files with md extension, read contents of files, pass them to showdown.Converter instances and write the output (which is html converted from markdown) to same file name as markdown, but with .html extension in /html/ directory, which is added in .gitignore.

Soon I realized I needed a comprehensive list of all files as the entrypoint of static website. So in next commit, I have automatically created index.html. And to populate the content, I listed all .md file and created a link to .html file, pushed that code to an array, and written the array in index.html. You can see the code in convertHtml.js of this commit.This is also when I started styling the pages, starting with Sans Serif font face.

Github Actions

Github Actions are CI/CD and automation tool offered by Github. Using simple scripts we can do many things. I decided to use Github Action to automate deployment process. My ideal flow would be write the markdown file and push to the repository. On push, Github Actions should run convertHtml.js and deploy the resultant HTML files to S3. We write our Github actions in .github/workflows/ directory in our repository. The Github actions (or Any CI/CD pipelines hosted on Git servers like BibBucket or GitLab), are yaml files that contains various informations, conditions and steps regarding to the flow. We create that file, push our repository, and based on the contents of yaml file, Github Actions perform the steps.

The problem with this setup is that, we don’t have any local emulation where we can test our workflow. We have to try this in CI/CD pipeline offered by Github (Or BitBucket or GitLab), check for any error, fix that error, push the code and repeat this process until we achieve our final goal with Actions. From first commit on actions(Commit), to first workable setup, uploads to S3 (Commit), it took me 8 tries to make things right. At this point I had my Github Action ready and every time I push, I got new version in S3 website in mater of minutes.

Metadata

Markdown has a feature called Front Matter, which helps us to attach parseable metadata that can be useful with the post. For Example, summarry of every post, tags and series information, title etc are stored as Front Matter for every post in this blog. For notes site, I need to store title with the posts, so that different sections can be identified properly. ShwodownJs has an option called metadata which can parse metadata (AKA Front Mater) from Markdown files, using .getMetadata() method, it gives whole front matter as JSON object.

As you can see in this commit, I have added a front matter called title in each post, and using converter.getMetadata() I was able to get Title, using which 1. I was able to create better name to links in Index page and 2. Give Proper title to HTML pages.

Adding more styles and Fixes

ShowdownJs had one problem, when I interlink two pages using Markdown, Shwodown does not convert links to HTMLs. For example if from b.md I link something of a.md using [Link from A](./a.md), a proper Markdown to HTML converter should convert this into <a href="a.html">Link From A</a>. But Showdown converted above link as <a href="a.md">Link From A</a>, which leads to broken links between the pages. I needed to replace .md in all the pages with .html. And in Javascript, you don’t have replaceAll method like other languages like Java or Kotlin, the working replacement of replaceAll(), is to split the string for existing token you wnat to replace, and Join the resultant array using new token you want to replace, like as shown below.

let htmlData = converter.makeHtml(fileData)        
htmlData = htmlData.split(".md").join(".html")

Also I have added some navigation links to go to index page, and to original Repository in same commit.

I also tried to add some additional information, I tried to have Created On and Modified on in Next Commit, these values are stored in fs.stat apis in NodeJS, where birthtime denotes created on and ctime denotes last modified on. This works properly on my machine, showing different times for both, but in CI/CD, all files are created and modified on same time. So I removed it later as they’re useless in main website.

For later parts of course, there is lot of interlinks. One point in one document notes to whole section in other documents, and I needed something which creates proper div ids and hashes in the pages where I start a section. And also an event which can give me a link to that div id which I can refer when required. In markdown, we start our section with one or multple hashtags #, which converts into corresponding header tag <h> in HTML. Like # First Header converts to <h1> First Header </h1>, ## First Header converts to <h2> First Header </h2> and ### First Header converts to <h3> First Header </h3> and so on. All I needed is to add additional information to header tags and convert them into links, which gives url with hashtag when clicked.

And for that ShowdownJS has a good plugin system where using combinations we can change converted HTML before it gives final output. I have created the ShowdownJS plugin using Regex which adds a button, when clicked copies the id to clipboard (you can see the code in this commit). I also added roboto fonts in that commit to make site look better. In later commit, I have converted this button in link. Also in this commit I have made some page optimization and common header.

Search

• The commit that contains whole code of my search integration is here

For quick memorization, I needed to add search where based on some keywords and I should have a list where this keyword appears in Search. For site search solution, I have looked to Google Site search and Algolia. I decided not to go with Google Site Search because of privacy reasons, and I have tried to go to Algolia for couple of days, but their documentation is not as straight forward and I still inclined towards fully local search, where no content of site should go to other service and search happens locally.

Fortunately node has two libraries, lunr.js and elasticlunr.js which can provide fully local search. Both of these libraries are not prepared by node devs so their documentation is quite vague and confusing to start with. I chose for less confusing of these two and went ahead with elasticlunr.

In Elasticlunr you can pre-populate search index data and link from your webpage, also you can provide additional metadata. You need to provide id to differentiate one document from other, title to show the title of search result and body where all search is indexed, additionally I added a field called filename to prepare better linking. I wrote this index json in search.js and linked this file in index.html. I also added elasticlunr.min.js so I can inject elasticlunr instance for search.¹

In functions.js, which serves as main Javascript file, I have added search() function which works as search engine for the site. First of All I make sure I have searchIndex ready before I perform search. When searching, I get the text of search text field and pass it to my searchIndex for search. The result comes in array when one or more indexes match with search query, and the index are referenced from corresponding data from index which we already prepared when preparing the whole site. And in index we have file name and title which became helpful in preparling the links. And I have injected the search results in appropriate div using innerHTML property. This was the last tweak I did in my notes site.

The notes site is fully searchable, interlinked properly and is not that hard on eyes. At this point there is no more bugs or improvents are needed for me, also between that and today, I got more occupied with Exams, current work etc. If you think there is room for improvements do file an issue here.

This is the end on series of AWS Solutions Architect. Till we meet next time.

May the force be with you