Welcome

This document is the official documentation for stellar, a courseware software. Stellar is a very modular framework and can be used in various ways. The first part of this book will explain the fundamentals of this framework, and then we will take a look at an actual implemented a of LaTeX-driven environment for stellar, which is available in the paolobettelini/notes repository.

Installation

The stellar-cli contains everything needed to manage the stellar ecosystem. It contains the tools to manage the contents of the courses as well as the server.

You can either download the executable from the releases or compile it yourself. Regardless, you still need to install the python scripts as described below.

Compile the CLI

The webserver uses leptos.

git clone https://github.com/paolobettelini/stellar
cd stellar
cargo leptos build --release
mv target/release/stellar-cli /usr/local/bin

Development

cargo leptos watch -- web ...

Do not use --port or --address in development, set those values in the Cargo.toml instead.

MongoDB

paru -S mongodb-bin
systemctl enable mongodb
systemctl start mongodb

Necessary scripts

The following steps are absolutely crucial if you plan on generating pages and snippets from PDFs. The plan for the future is to write the following scripts directly in rust and avoid extra steps like these.

chmod +x scripts/*
mv scripts/* /usr/local/bin

# Install pdfminer and pymupdf
# Arch:
pacman -S python-pdfminer python-pymupdf

Issues with cropping PDFs: Unfortunately, there are some cropping problems with the mupdf package. The problems arise from version 1.24 onwards.

wget https://archive.archlinux.org/packages/p/python-pymupdf/python-pymupdf-1.23.21-1-x86_64.pkg.tar.zst
sudo pacman -U python-pymupdf-1.23.21-1-x86_64.pkg.tar.zst

wget https://archive.archlinux.org/packages/p/python-mupdf/python-mupdf-1.23.9-4-x86_64.pkg.tar.zst
sudo pacman -U python-mupdf-1.23.9-4-x86_64.pkg.tar.zst

wget https://archive.archlinux.org/packages/l/libmupdf/libmupdf-1.23.9-4-x86_64.pkg.tar.zst
sudo pacman -U libmupdf-1.23.9-4-x86_64.pkg.tar.zst

paru -S python311 # use this version

Lock the packages version in /etc/pacman.conf

IgnorePkg = libmupdf python-mupdf python-pymupdf

In order to use the correct python version, you can replace the shebang in pdfcrop.py with #!/bin/python3.11. The other script does not need it.

Data Structure

The fundamental types of stellar are four: snippets, pages, courses and universes. These are the building blocks of all the information within a stellar instance.

NOTE: each of these objects have a unique ID, which is written in lower-kebab-case format.

All the data is stored in a folder named data.

data/
├── snippets/
├── pages/
├── courses/
└── universes/

MongoDB database

A MongoDB database is also required in order to index the data and store some information about the contents of the website.

Snippets

Definition snippet: a snippet is the most primitive type of information. It might be an image, a PDF or simply some HTML data. Every snippet has its own folder named as its ID, which may contain multiple files. The primary file of the snippet is also named as the ID, but with the correct extension (E.g. snippet-id/snippet-id.pdf). Files other than the primary file are called secondary files. Secondary files are auxiliary and can be imported by the primary file (or, sometimes, from somewhere else).

The route /snippet/snippet-id will render the snippet primary file of the snippet with ID snippet-id, alongside some additional data.

The route /snippet/snippet-id/file1.csv will query for a secondary file within the snippet with ID snippet-id.

Parameters

A primary file of the HTML form may optionally require parameters. A parameters can be expressed by the syntax #{paramName|default}. This entire string will be replaced by the given parameter, or by the default value if no value was provided.

As an example, the following snippet will render an image. The width, height, source and alt of the images are all parameters.

<div style="text-align: center;">
    <img
        width="#{width|auto}"
        height="#{height|auto}"
        src="#{src|Missing SRC}"
        alt="#{alt|Could not display image}"
    >
</div>

Stellar will inject the parameters only in the primary HTML file of a snippet. If you need to setup some JavaScript variables from the parameters, you will need to do it in the primary file.

File `meta.json`

A special secondary file is the meta.json which contains additional metadata (more information will follow in the book).

JSON Structure

Field	Type	Description
`generalizations`	Array of snippet IDs	Mathematical generalizations.
`requires`	Array of snippet IDs	Necessary libraries.
`default-params`	Params string	Default parameters for snippet.

Example

{
    "default-params": "width=70%|src=https://youtu.be/dQw4w9WgXcQ&",
    "requires": ["some-lib-snippet"]
}

Pages

Definition page: a page is an HTML page which can contain arbitrary HTML. A page may include other snippets that exist in the enviroment. For now, every page is a single HTML file named <ID>.html in their own folder, but in the future it will have the same primary file/secondary file mechanism as snippets.

Snippets tag

There is a special tag which can be used to include snippets in the page

<stellar-snippet>ID</stellar-snippet>

Parameters can be passed as follows:

<stellar-snippet params="param1=v1|param2=v2">ID</stellar-snippet>

Here is an example of a page content:

<h1>Welcome to this page<h1>

Here is some normal text which will be displayed

<stellar-snippet params="src=https://...">yt-embed</stellar-snippet>

And here is a snippet

Courses

Definition course: a course is a collection of ordered pages. Every course is a single JSON file named ID.json containing the list of pages and headings.

Every course has a title and a list of pages. Every page is an array of 2 or 3 elements. The first element is a number (either 1, 2, or 3) which indicates the level of the heading. The second value is the title of the heading of the page (string). The third argument is optional and contains the ID of the page (string). If no third argument is specified, the title is not clickable.

{
    "title": "Stellar documentation",
    "pages": [
        [
            1,
            "Welcome",
            "welcome-page-id"
        ],
        [
            1,
            "Stellar",
        ],
        [
            1,
            "Introduction",
            "stellar-introduction-page-id"
        ],
        [
            2,
            "Snippets",
            "stellar-snippets-page-id"
        ]
    ]
}

Universes

Definition universe: a universes are a collection of courses connected by functional dependencies in a graph. Every universe is a single JSON file named ID.json.

Every universe contains a list of all the courses to display along with their coordinates in the (x,y) plane. It also contains a list of pairwise dependencies between courses IDs.

{
  "title": "Rendering",
  "courses": [
    {
      "name": "Linear Algebra",
      "id": "linearalgebra",
      "x": 280,
      "y": 150
    },
    {
      "name": "Calculus",
      "id": "calculus",
      "x": 280,
      "y": 280
    },
    {
      "name": "Physical Rendering",
      "id": "physicalrendering",
      "x": 520,
      "y": 220
    },
    {
      "name": "Computer Rendering",
      "id": "computerrendering",
      "x": 520,
      "y": 360
    },
    {
      "name": "Vulkan",
      "id": "vulkan",
      "x": 800,
      "y": 280
    }
  ],
  "dependencies": [
    {
      "from": "linearalgebra",
      "to": "physicalrendering"
    },
    {
      "from": "calculus",
      "to": "physicalrendering"
    },
    {
      "from": "computerrendering",
      "to": "vulkan"
    },
    {
      "from": "physicalrendering",
      "to": "vulkan"
    }
  ]
}

PDF Stellar Format

Stellar is able to interpret special PDFs with certain text commands written on the page. This is referred to as PDF Stellar Format. The file contains a full stellar page and snippets or just a collection of snippets. Stellar will crop the PDF into multiple pieces to generate every snippet individually, and is also able to generate the corresponding page containing them.

Format Commands

There following commands are to be written as text in the PDF.

Command	Description
`!id <ID>`	Set the ID of this page.
`!snippet <ID>`	Start a PDF snippet here.
`!endsnippet <JSON meta>`	End the current snippet here. The JSON metadata is optional.
`!gen-page <bool>`	Set to true to generate the HTML page for this PDF.
`!include <ID> <Params>`	Include a snippet here given its ID. The parameters string is optional.
`!plain`	Write plain HTML to the page here.
`!section`	Add a level 1 heading `<h1>`.
`!subsection`	Add a level 2 heading `<h2>`.
`!subsubsection`	Add a level 3 heading `<h3>`.

Furthermore, it is possible to reference another snippet by adding an annotation with a link of the form /snippet/snippet-id or /snippet/snippet-id|Label.

Stellar will crop the PDF between !snippet and !endsnippet commands (margins can be adjusted in the CLI parameters).

LaTeX Package

The file stellar.sty provides the LaTeX package stellar, which implements PDF Stellar Format. The file is available here.

Usage

To use the package, use the following document class:

\documentclass[preview]{standalone}

and import the stellar package:

\usepackage{stellar}

The command \title needs to be inserted within the document. The commands \section, \subsection, and \subsubsection can be used normally as their behavior is overwritten by the package.

Available commands: - \id - \genpage - \plain

You can reference another snippet using:

\snippetref[snippet-identifier][Some text].

To include a snippet:

\includesnpt{snippet-identifier}
\includesnpt[param1=value1|param2=value2]{snippet-identifier}

Create snippets using environments:

\begin{snippet}{snippet-identifier}
    % snippet content
\end{snippet}

Other types of snippets:

\begin{snippetdefinition}{snippet-definition1}{Definition 1}
    % snippet content
\end{snippetdefinition}

\begin{snippettheorem}{snippet-theorem1}{Theorem 1}
    % snippet content
\end{snippettheorem}

\begin{snippetproof}{snippet-proof1}{snippet-theorem1}{Proof 1}
    % snippet content
\end{snippetproof}

The plain command

The plain command is used as follows:

\plain{This is some <b>HTML</b> text.}

The plain command embeds its content into the HTML code of the page. Note that the passed argument is printed verbatim, LaTeX commands will not work. If you want to write a percentage, use \HTMLPercentage.

Example

\documentclass[preview]{standalone}

\usepackage{stellar}

\begin{document}

\id{page-identifier}
\genpage

\section{Section 1}

\subsection{Subsection 1}

\begin{snippetdefinition}{snippet1-definition}{Some definition}
    This is some definition.
\end{snippetdefinition}

\subsection{Subsection 2}

\section{Section 2}

\begin{snippetdefinition}{snippet2-definition}{Some definition}
    This is some definition, the sequel.
\end{snippetdefinition}

\includesnpt{snippet2-definition}

\begin{snippet}{text-snippet}[\{"meta": "json"\}] % will add a meta.json file
    Some text as a snippet!
\end{snippet}

\end{document}

Visual Studio Code

You can Configure User Snippets on VS Code (not to be confused with stellar snippets) in order to access some macro for the stellar.sty LaTeX package.

The snippets file is available here.

The file provides some autocompletions such as sdocument which will provide a basic template

\documentclass[preview]{standalone}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{stellar}
\usepackage{definitions}

\begin{document}

\id{id}
\genpage

\end{document}

and also various macros to define snippets such as snippet

\begin{snippet}{id}
    body
\end{snippet}

sdefinition

\begin{snippetdefinition}{id-definition}{name}
    body
\end{snippetdefinition}

sproof, stheorem and such.

Stellar CLI

The stellar cli has various subcommands.

Stellar CLI

Usage: stellar <COMMAND>

Commands:
  generate  Help message for generator
  import    Help message for import
  web       Help message for web
  check     Help message for check
  help      Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print versio

stellar web

The web subcommand is used to start the web server.

Help message for web

Usage: stellar web [OPTIONS] --data <DATA> --connection-url <CONNECTION_URL>

Options:
  -a, --address <ADDRESS>                Listening address [default: 0.0.0.0]
  -p, --port <PORT>                      Listening port [default: 8080]
  -d, --data <DATA>                      Data folder
  -c, --connection-url <CONNECTION_URL>  MongoDB Connection URL
  -h, --help                             Print help

stellar import

The import subcommand is used to process data and import it to the MongoDB database.

Help message for import

Usage: stellar import [OPTIONS] --connection-url <CONNECTION_URL>

Options:
  -c, --connection-url <CONNECTION_URL>
          MongoDB connection URL
  -i, --import <IMPORT>...
          Import data/snippets/pages/courses folder or single snippet/page/course
  -h, --help
          Print help

Here is a bunch of examples:

stellar import -c "..." --import ./data
stellar import -c "..." --import ./data/snippets
stellar import -c "..." --import ./data/pages
stellar import -c "..." --import ./data/courses
stellar import -c "..." --import ./data/snippets/snippet1
stellar import -c "..." --import ./data/pages/page1.html
stellar import -c "..." --import ./data/courses/course1.json
stellar import -c "..." --import course1.json course2.json

stellar generate

The generate subcommand is used to parse a PDF in the PDF Stellar Format and generate a page and snippets.

Help message for generator

Usage: stellar generate [OPTIONS] --input <INPUT> --data-output <DATA_OUTPUT>

Options:
  -i, --input <INPUT>                    Generate data from a PDF file
  -o, --data-output <DATA_OUTPUT>        Output for generated data
      --import                           Import the generated snippets
  -c, --connection-url <CONNECTION_URL>  MongoDB connection URL
      --top-offset <TOP_OFFSET>          Snippet cut top offset [default: -20]
      --bottom-offset <BOTTOM_OFFSET>    Snippet cut bottom offset [default: 2.5]
      --left-margin <LEFT_MARGIN>        Snippet cut left margin
      --right-margin <RIGHT_MARGIN>      Snippet cut right margin
  -h, --help

To compile from the PDF generated by the previous LaTeX example:

stellar-cli generate snippets \
    --input stellar.pdf \
    --data-output ./data \
    --bottom-offset 9.5 \
    --top-offset=-20

stellar check

The check subcommand is used to check the integrity of the existing data.

There are various integrity checks:

Existence of refereces in the snippets, pages, courses and universes
Autoreferentiality in snippets
Reference linearity

Help message for check

Usage: stellar check [OPTIONS] --connection-url <CONNECTION_URL>

Options:
  -c, --connection-url <CONNECTION_URL>
          MongoDB connection URL
  -e, --existences
          Check only existance of references E.g. existance for courses checks whether the pages exist
  -a, --autoreferentiality
          Check only snippets self-reference
  -l, --linearity
          Check only snippets linearity
  -h, --help
          Print help

Installation

Install stellar and clone the repository

git clone https://github.com/paolobettelini/notes

Set the necessary enviromental variables:

NOTESPATH # pointing to the notes/ directory
MONGO_CONNECTION_URL # connection URL to mongodb

Install the required libraries (some snippets require npm and wasm-pack to compile)

pacman -S tectonic npm wasm-pack git

Compile the compiler or download it from the releases

cd notes/compiler
cargo build --release
mv target/release/compiler /usr/local/bin/notes

Compiling everything and starting the server

cd source
notes # compiles everything
cd ..

It is advisable to set the CARGO_TARGET_DIR variable so that the rust projects share the same target folder.

Then, start the web server

stellar web --data data/ --connection-url $MONGO_CONNECTION_URL

Go to localhost:8080/search.

Structure

data/
source/
├── latex/
├── courses/
├── pages/
├── snippets/
└── universes/

The latex/ folder contains all the .tex files for the PDF Stellar Format files. Even though pages and snippets can already be generated from PDFs, the pages/ and snippets/ are a place to put manually crafted snippets and pages.

Compiler usage

The notes executable is a command used to compile data according to the structure of this repository.

Stellar notes compiler CLI

Usage: notes [OPTIONS] [INPUT]

Arguments:
  [INPUT]  Compile query

Options:
  -r, --regex        Use a regex
  -i, --ignore-case  Ignore case
      --snippets     Check only snippets folder
  -p, --pull         Compile current git status files last pull
      --latex        Check only latex folder
      --pages        Check only pages folder
      --courses      Check only courses folder
      --universes    Check only universes folder
  -h, --help         Print help
  -V, --version      Print version

The compiler will automatically handle everything. It will import the data to the database. You can call notes from any directory.

notes # compiles everything
notes Something # compiles every file name that contains "Something"
notes -r "S|T|N|G" # compiles every file name that matches the regex
notes --latex # compiles all the latex files
notes --snippets # compiles all the universes
notes --pages # compiles all the universes
notes --courses # compiles all the universes
notes --universes # compiles all the universes
notes --universes -r "..." # compiles all the universes that match the regex
notes course1 File1 File2 # compile multiple inputs
notes -r reg1 reg2 # compiles files matching reg1 or reg2
notes "Something" --containing "\command" # compiles every file name that contains "Something" whose content contains "\command"
notes -r "Something.+" --containing "\command" # like the one above but with regex matching
notes File1 File2 --containing "hello"

Note: the --containing option is not affected by --regex and --ignore-case.

Snippet Build Scripts

By default, snippets in the snippets/ folder are just copied to the data/ folder by the compiler. However, it is possible to add a custom build script build.py to the snippet directory to implement custom behavior. The build script will receive a single parameter representing the target folder where the files need to be written to.

Example Script

import sys
import subprocess
import os
import shutil

# This build script compiles a nannou project into a snippet

def main():
    target_folder = sys.argv[1]
    
    try:
        # Execute wasm-pack build --release
        subprocess.run(["wasm-pack", "build", "--release"], check=True)
        
        # Change directory to website
        os.chdir("website")
        
        # Execute npm install
        subprocess.run(["npm", "install"], check=True)
        
        # Execute npm run build
        subprocess.run(["npm", "run", "build"], check=True)
        
        # Generate snippet folder
        dist_folder = "dist"
        target_files = os.listdir(dist_folder)
        
        # Remove "dist/index.js"
        index_file_path = os.path.join(dist_folder, "index.js")
        if os.path.exists(index_file_path):
            os.remove(index_file_path)
        
        # Move all files from "dist" to target_folder
        for file_name in target_files:
            file_path = os.path.join(dist_folder, file_name)
            if os.path.isfile(file_path):
                shutil.move(file_path, target_folder)
        
        # Remove the "dist" directory
        shutil.rmtree(dist_folder, ignore_errors=True)
        
        print(f"Successfully moved files to {target_folder}")
    
    except subprocess.CalledProcessError as e:
        print(f"An error occurred while executing: {e.cmd}")
        print(f"Return code: {e.returncode}")
        sys.exit(1)
    except Exception as e:
        print(f"An unexpected error occurred: {str(e)}")
        sys.exit(1)

    if __name__ == "__main__":
        main()

Common snippets

yt-embed: Embeds a YouTube video. Params: src, width, height
centered-img: Displays an image. You can put the images in snippets/static. Params: src, width, height
iframe: Embeds an iframe. Params: src, width, height
file-url: Embeds a file download. Params: href, display
mathjax: Loads the MathJax library. Use \usepackage{mathjax} instead to achieve the same.
desmos: Loads the Desmos API.

MathJax support

You can enable MathJax and write LaTeX in the plain command by importing the mathjax package.

\usepackage{mathjax}
...
\plain{Some math: \(e^x\).}

Fish configuration

If you're writing notes using stellar you might find this setup useful (~/.config/fish/config.fish):

set --export MONGO_CONNECTION_URL "mongodb://127.0.0.1" # modify accordingly
set --export NOTES_PATH "/home/user/notes" # modify accordingly

function startstellar
  sudo -v
  sudo stellar web --data "$NOTES_PATH/data" --connection-url $MONGO_CONNECTION_URL --port 80
end

function stopstellar
  sudo -v
  set pid (sudo lsof -t -i :80)
  if test -z "$pid"
    echo "Server was not open"
  else
    sudo kill -9 $pid
    echo "Server was killed"
  end
end

function updatenotes
  sudo -v
  set latest_url (curl -s https://api.github.com/repos/paolobettelini/notes/releases/latest | grep browser_download_url | cut -d '"' -f 4)
  set temp_file /tmp/notes_latest
  curl -L -o $temp_file $latest_url
  chmod +x $temp_file
  sudo mv $temp_file /usr/local/bin/notes
end

function updatestellar
  sudo -v
  set latest_url (curl -s https://api.github.com/repos/paolobettelini/stellar/releases/latest | grep browser_download_url | cut -d '"' -f 4)
  set temp_file /tmp/stellar_latest
  curl -L -o $temp_file $latest_url
  chmod +x $temp_file
  sudo mv $temp_file /usr/local/bin/stellar
end

VS Code configuration

You can install the Action Buttons extension and add the following command:

{
    "name": "$(triangle-right) Run Notes",
    "color": "green",
    "singleInstance": true,
    "command": "notes ${fileBasename}",
}

This will provide a button to compile the current file.