New Borg, Epkg, Closql and Emir releases

I am excited to announce these releases: borg v2.0, epkg v3.0, closql v0.4 and emir v2.0.

Introduction

A short summary of what these packages do might be in order:

• borg is a bare-bones package manager that installs packages using Git submodules. It is most useful to package authors and users who also contribute to the packages that they use.

• epkg provides access to a database containing information about ~6600 Emacs packages. It provides tools for querying the SQL database and an user interface to list packages and to view detailed information about individual packages.

• closql stores objects in SQLite databases. It relies on emacsql by Christopher Wellons and was written to be used by epkg.

• emir is used to maintain the Emacsmirror, including the package database.

Earlier announcements and discussions about the Emacsmirror and these packages:

• Re-introducing the Emacsmirror (on r/emacs)
• Assimilate Emacs packages using Git submodules
• On r/emacs: Borg: Manage packages as Git submodules

Until a few months ago borg did not have more than a handful of users, but recently it gained some popularity. Given its limited feature set and the up-front warnings about rough edges found in the earlier announcements, this slow start is not really surprising, and I did actually hope and expect that it would fly under the radar for a while.

Still I put a lot of work into the Emacsmirror and these packages, going back long before the initial borg and epkg releases about one and a half years ago, so I do hope that they will grow in popularity eventually and am happy that this has began now. Someone even created an extension to borg!

I am quite proud of these packages. The guiding principle is “achieve more by doing less”. Which — somewhat to my own surprise — does not mean that I haven’t added any new features, just that I usually put up a big fight first ;-) So far I haven’t outright rejected any suggested features, and some of the features that I have ruled out in the initial announcement are actually back on the table. Never-the-less, moving slow and keeping the code base and feature set small are still very important to me — but that’s not a problem because one of borg’s biggest strengths is that it makes dealing with stubborn maintainers easy.

Getting started

Originally I planned to give the user manuals a makeover before creating new releases, but have now decided that that can wait until later.

The manuals are up-to-date, but they would benefit from some reorganization and there isn’t any documentation on how to convert an existing configuration to use borg (but some unpolished notes are available).

If you are not already using borg, then I recommend that you wait a few more weeks until I have improved the documentation, as I probably won’t have the time to help new users individually.

On the other hand there’s no reason to delay taking epkg for a spin.

Updating from older versions

Normally updates are seamless, but this time around you might end up in an inconsistent state because the database schemata changed, making it necessary to update epkg and the database at the same time.

If you read this before attempting to update, then proceed like so:

1. Update the drones mentioned here, using Magit or git — as usual.
2. Quit Emacs.
3. In ~/.emacs.d/epkgs/ run git pull.
4. In ~/.emacs.d/ run make.

If you update epkg without updating the database at the same time, or vice-versa, then epkg will complain about that when you first attempt to access the database and advice you on how to recover.

New features

These are just the highlights, if you already use borg, then you should also read the latest entries in the borg, epkg and closql changelog files.

Review packages before installation

The new command borg-clone makes it easier to review a package before assimilating (installing) it. Assimilating a package gives that package a change to run arbitrary code, so the review should happen first. I strongly recommend that you use this command and only use borg-assimilate once you have inspected the cloned package.

Some other commands that previously only operated on assimilated packages can now also deal with packages that have only been cloned but which are not being tracked as submodules (yet).

Store GNU Elpa and Melpa recipes as first class objects

The database has always contained information about packages from GNU Elpa and Melpa. Now the upstream recipes can be accessed as objects, and queried using reasonable SQL queries just like the core “Emacsmirror metadata” as seen by epkg.

This required major changes to closql and some boilerplate in epkg and emir. It also meant that some table schemata had to be updated.

For now this mostly benefits me as the maintainer of the Emacsmirror and as a Melpa contributor, but I do hope that eventually other people will use the database more actively than just to browse the available packages. But again, that probably needs some additional documentation.

Planned features

Cooperate with package.el

While borg was initially intended as an alternative to package.el, I understand that most users prefer the simplicity of the latter.

Still I think that package maintainers should run their own packages from the Git repository instead of using that only for development and installing the package using package.el. And some of the upcoming Magit features will probably live on feature branches for a while and I would like to make it easier to use those so that I get more feedback before merging into master.

This entails:

• Teach package.el about packages installed by borg, i.e. assume that they are installed.

• Support installing packages without also assimilating them, i.e. support automatically activating packages that are not being tracked as submodules. That would remove the sharpest of the sharp edges and might make borg more attractive to casual users.

• Add borg to Melpa.

Further improve security

Package archive maintainers cannot possibly review all packages for security issues and malicious code, so users have to share some of the burden. We should however make it easier to identify packages that might deserve more scrutiny than others and prevent mistakes.

• It should be possible to hide insecure packages from package lists (as is already possible for packages that are no longer maintained).

• When such a package is displayed anyway for some reason, then it should be marked as insecure.

• When the user proceeds to install an unsafe package, then s/he should have to confirm explicitly before any potentially malicious code gets executed.

• Verify GPG signatures and allow users to insist on packages being signed. While this is a good idea in principle, I fear that it won’t do much good as most maintainers don’t sign their tags or even every commit. Still basic support at least should be available.

Comments on Reddit.