Academic Notes ‘database’
When it comes to University classes, being able to organize the information provided is an essential aspect. I, for example, can only work in an efficient manner once I've found the right sources of information to study from. However, this process is sometimes more time consuming than it's worth. As of the time of writing there are 4 note banks directly connected to my course and they are, without taking into account note banks from other Universities. This generates a problem, and it isn't lack of information, but rather how that information is presented and if it's readily available or somewhere in a forgotten cloud account.
This is something me and my colleagues have noticed and discussed. Something I've wanted to change for about a year now.
When I started to seriously consider making a good source of information about science and technology I easily became overwhelmed by the effort such a project would require. I mean, if I wanted to do everything right I would need to create a website to host the resources, create a way to enable simultaneous collaboration, a database to have everything organized, a distribution system to make sure that updates wouldn't require going to the website, a version control system and some way to curate the content and keep track of everything. This without even taking into account the content that could be missing and would need to be created. This does sound overwhelming.
So I started my search for finished solutions. From complete and tested software like OpenEdX and Canvas to simpler note-taking tools like Trillium, however everything involved compromising some feature, getting a capable server, much tinkering and hoping people would want to learn to use this software in case they want to contribute.
So, evidently, that wasn't the way to go. But what would make a good solution. Well, from my point of view, it would have to:
- Consist of a variety of materials
- Allow to easily write new materials
- Allow simultaneous changes
- Have a good interface and easy to maintain
- An option to view materials online
- An option to update materials locally with the latest changes from the server
- Some kind of version control in case there is a need to go back to a previous version
- Some system to curate, organize and keep track of all materials
- A system to keep track of bugs or features that need to be implemented
- Not prohibitively expensive, time-consuming or complex
Keep It Simple, Stupid
Since I was introduced to the vast world of programming and informatics, I've learnt valuable lessons. Most of them the hard way (RIP all information in my, now gone, Windows partition). Nevertheless, some of those lessons haven't sync in yet. One of those lessons is the KISS Principle, which states Keep It Simple, Stupid, a nice reminder that, more often than not, the best solution isn't the more complex that takes everything into account. Sometimes we just have to stop, think and not reinvent the wheel.
This is one of those times.
After way to much time spent thinking about a problem that wasn't really there I stumbled upon Sphinx, a project that facilitates the process of making documentation, and the feature that really caught my eye: its ability to output ePub, PDF and HTML from restructuredText files.
RestructuredText is a markup language, much like the famous Markdown, that includes features like the ability to integrate Math expressions (something Markdown lacks) and is relatively easy to write, specially when comparing to LaTeX. This makes restructuredText the right tool for the job. Actually, this very document was written in restructuredText!
I have the typesetting system and I know that it's possible to produce the desired outputs. So now what? Well now I have to share the documents. Luckly for me, they can be share just as this very post is!
This post, as well as everything on this website, is made with a Static Site Generator, GoHugo, which allows me to create a template only once and add the content from easily written files using Markdown and restructuredText instead of producing a new HTML file everytime I post. This means that not only would I be able to choose one of the templates available, I could also do everything without a server since I already use Gitlab's tools for Continuous Integration and Deployment. I would just need to modify one template, so it would resemble a documentation page/note bank instead of a blog or personal website.
This was the point where I started to think: why did I really need a database? If the file hierarchy is good enough and with the search tools available, why bother with something complex and that requires resources I don't have? I decided that everything would be static and updated everytime there are changes, just like my website.
Having everything in Gitlab would also mean that I would use Git as the version control, it might be a bit intimidating for beginners, but with GUI helpers like GitKraken this is easy to overcome. And Gitlab brings other nice surprises like the ability to review changes and solve conflict editions, to have development versions, a bug tracker, milestones, lists and some more niceties.
At this point the project is totally different from what I imagined and there is only one feature missing: the ability to have several computers fetching the latest changes from the website. They could always access it and download the notes, but that's boring and I want to open a folder and have the files already there. This feature seems hard to implement in a static website. However, I haven't yet given up on this feature and after some Googling I find RSync, a program that keeps directories in a computer in sync with a centralized source, without the ability to change it. This is exactly what I needed, and the centralized source can be a Gitlab repository!
Old goal, New project
From a project where I would have to write everything from the ground up and have a 24/7 server to a project where I just have to integrate different, already established, tools in a centralized platform that carries everything out automatically, without losing even one of the initial features. I would say that one year of procrastination totally paid of.
Bringing it together
When it comes to the actual project, that seems a bit out of the scope of this post.
I'll publish its development on the project page once I have a PoC (Proof of Concept).