Developers’ Weblog

Sponsored by
HostEurope Logo

Developers’ Weblog

⚠ This page contains old, outdated, obsolete, … historic or WIP content! No warranties e.g. for correctness!

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

Nik wishes you to know that the Movim packaging sprint (sponsored by the DPL, thank you!) is handled under the umbrella of the Debian Edu sprint (similarily sponsored) since this package is handled by the Teckids Debian Task Force, personnel from Teckids e.V.

After arriving, I’ve started collecting knowledge first. I reviewed upstream’s composer.json file and Wiki page about dependencies and, after it quickly became apparent that we need much more information (e.g. which versions are in sid, what the package names are, and, most importantly, recursive dependencies), a Wiki page of our own grew. Then I made a hunt for information about how to package stuff that uses PHP Composer upstream, and found the, ahem, wonderfully abundant, structured, plentiful and clear documentation from the Debian PHP/PEAR Packaging team. (Some time and reverse-engineering later I figured out that we just ignore composer and read its control file in pkg-php-tools converting dependency information to Debian package relationships. Much time later I also figured out it mangles package names in a specific way and had to rename one of the packages I created in the meantime… thankfully before having uploaded it.) Quickly, the Wiki page grew listing the package names we’re supposed to use. I created a package which I could use as template for all others later.

The upstream Movim developer arrived as well — we have quite an amount of upstream developers of various projects attending MiniDebConf, to the joy of the attendees actually directly involved in Debian, and this makes things much easier, as he immediately started removing dependencies (to make our job easier) and fixing bugs and helping us understand how some of those dependencies work. (I also contributed code upstream that replaces some Unicode codepoints or sequences thereof, such as 3⃣ or ‼ or 👱🏻‍♀️, with <img…/> tags pointing to the SVG images shipped with Movim, with a description (generated from their Unicode names) in the alt attribute.)

Now, Saturday, all dependencies are packaged so far, although we’re still waiting for maintainer feedback for those two we’d need to NMU (or have them upload or us take the packages over); most are in NEW of course, but that’s no problem. Now we can tackle packaging Movim itself — I guess we’ll see whether those other packages actually work then ☺

We also had a chance to fix bugs in other packages, like guacamole-client and musescore.

In the meantime we’ve also had the chance to socialise, discuss, meet, etc. other Debian Developers and associates and enjoy the wonderful food and superb coffee of the “Cantina” at the venue; let me hereby express heartfelt thanks to the MiniDebConf organisation for this good location pick!

Update, later this night: we took over the remaining two packages with permission from their previous team and uploader, and have already started with actually packaging Movim, discovering untold gruesome things in the upstream of the two webfonts it bundles.

I’m currently working on an mksh(1) and bc(1) script that takes a pitch standard (e.g. “A₄ = 440 Hz” or “C₄ = 256 Hz”) and a config file describing a temperament (e.g. the usual equal temperament, or Pythagorean untempered pure fifths (with the wolf), or “just” intonation, Werckmeister Ⅲ, Vallotti or Bach/Lehman 1722 (to name a few; these are all temperaments that handle enharmonics the same or, for Pythagorean in out case, ignore the fact they’re unplayable). Temperaments are rule-based, like in ttuner. Well, I’m not quite there yet, but I’m already able to display the value for MuseScore to adjust its pitch standard (it can only take A₄-based values), a frequency table, and a list and table of cent deltas (useful for using or comparing with other tuners). Of course, right now, the cent deltas are all 0 because, well, they are equal temperament against equal temperament (as baseline), but I can calculate that with arbitrary and very high precision!

For outputting, I wanted to make the tables align nicely; column(1), which I normally use, was out because it always left-aligns, so I used string padding in Korn Shell — except I’m also a Unicode BMP fan, so I had F♯ and B♭ in my table headings, which were for some reason correctly right-aligned (for when the table values were integers) but not padded right when aligning with the decimal dot. So I worked around it, but also investigated.

Turns out that the desired length was used as second snprintf(3) argument, instead of, as in the right-align case, the buffer size. This worked only until multibyte characters happened. A fun bug, which only took about three minutes to find, and is covered by a new check in the testsuite even. Thought I’d share.

Feedback on and improvements for the tuner, once it’ll be done, are, of course, also welcome. I plan to port the algorithm (once I’ve got it down in a programming language I know well) to QML for inclusion in the tuner MuseScore plugin, even. Check here, for now, for my work in progress… it’s quite big already despite doing basically nothing. Foundation laid (or so…).

I’m currently working on consolidating mirsolutions.de (as my former business is long defunct) and “The MirOS Project” (that as of 2018 is also back to being my own one-man show / hobby) into www.mirbsd.org as my hobby / personal sorta homepage to not need any vhosts and simplify EU-DSGVO conformity. (I’ve also reduced logging.) Please excuse upheavals, as well as the continued presence of old, obsoleted or outdated content that may even be, as of now, completely wrong; I’ll fix it as time permits.

Happy Birthday, GPS Stash Hunt!

2018-05-03 by tg@
Tags: debian fun geocache personal

GPS Stash Hunt, also commercially known as “Geocaching”, “Terracaching”, or non-commercially (but also nōn-free) as “Opencaching”, is 18 years old today! Time for celebration or something!

mirabilos’ Waypoints

Excluding Munzees, I visited 2365 waypoints to date, ever since 2006 (I knew about GPS Stash Hunt in 2004 already but thought one’d need one of those expensive GPS receivers to play, which is untrue but I only got that when Pfeffer (from BOSng) took his GPS and me out for a bike tour in 2006, so, here we are…

(Some older on-platform logs have a number that still counts Munzees; the current format uses “number﹟”.)

Today I’ve released another new CVS snapshot of the FixedMisc [MirOS] font; as usual, the tarball contains the font in BDF form, with no conflict with the system Fixed [Misc] font; sources for use (compilation, editing) with bdfctool(1) are in CVS.

mksh on Jehanne, a guest post by Shamar

2018-04-15 by tg@
Tags: archaeology debian fun guest mksh pcli

Giacomo Tesio referenced mksh(1) in his annual Jehanne report and provided a guest post (dated 2018-01-09, sorry for posting it this late only) for us on his journey on porting mksh to Jehanne, his Plan 9 derivative operating system. Read on for his story!


MirBSD's Korn Shell on Plan9 Jehanne

Let start by saying that I'm not really a C programmer.
My last public contribution to a POSIX C program was a little improvement to the Snort's react module back in 2008.

So while I know the C language well enough, I do not know anything about the subtliness of the standard library and I have little experience with POSIX semantics.

This is not a big issue with Plan 9, since the C library and compiler are not standard anyway, but with Jehanne (a Plan 9 derivative of my own) I want to build a simple, loosely coupled, system that can actually run useful free software ported from UNIX.

So I ported RedHat's newlib to Jehanne on top of a new system library I wrote, LibPOSIX, that provides the necessary emulations. I wrote several test, checking they run the same on Linux and Jehanne, and then I begun looking for a real-world, battle tested, application to port first.

I approached MirBSD's Korn Shell for several reason:

  • it is simple, powerful and well written
  • it has been ported to several different operating systems
  • it has few dependencies
  • it's the default shell in Android, so it's really battle tested

I was very confident. I had read the POSIX standard after all! And I had a test suite!
I remember, I thought "Given newlib, how hard can it be?"

The porting begun on September 1, 2017. It was completed by tg on January 5, 2018. 125 nights later.

Turn out, my POSIX emulation was badly broken. Not just because of the usual bugs that any piece of C can have: I didn't understood most POSIX semantics at all!

 

First, Cinap had to patiently explain me on #cat-v that UNIX signals are reentrant.
It took him a while: I wasn't able to understand.
Even now, I keep asking: "Why?!? Why they did this! why..."

 

Fixed that, I saw that mksh was unable to execute ls: in Plan 9 common environment variables are lower case.
The $PATH variable is called $path, the $CDPATH variable is called $cdpath and so on.
Also, when appropriate, they are NULL separated char arrays, since they are exposed as files from the env device, and rc can get their size with a simple seek.

I reflected on the issue for a while, tried several solutions to preserve both conventions (some of which even worked).
But finally, I surrended to the simplest solution: I adopted the POSIX convention for Jehanne.

Aesthetics amuse, but simplicity helps.

However it was not enough: I needed to hook mksh startup to read the variables from the filesystem (just like rc does). How to do that cleanly? I asked on #!/bin/mksh and tg did not simply explained a poor noob how to do that. He did it himself!

I was enchanted by his kindness. So far Jehanne is just a toy. Still he spent his own time for me.

 

But the journey was still ongoing.

I realized that to run a command, mksh requires SIGCHLD support. I added it. The first implementation worked.

Once.

It was able to run exactly one command in mksh. The shell stopped reading input after the second one.

So I wrote it again from scratch. And it worked! Yuppy! :-)

Till I tried echo test | grep test

Grep didn't get EOF, as mksh for some strange reason was keeping the pipe open.
I extended devdup to ensure fcntl's emulation was working as expected. I rewrote fcntl emulation. Still broken.

Out of despair I turned to annoy tg again over IRC. Talking with him I realized that the problem was the signal dispatching. So I rewrote it again, introducing a new 9P2000 file server that handles signal IPC among POSIX processes, taking care of masks, ignored signals, waited ones.

Finally echo test | grep test worked.

 

But...

mksh was blaming me with two annoying warnings:

mksh: No controlling tty: open /dev/tty: No such file or directory
mksh: warning: won't have full job control

I asked tg and he tried to explain me what /dev/tty is, providing links about /dev/tty, /dev/ttyN and /dev/console.

So I modified vt (and later hmi/pipeconsole) to provide /dev/tty as an alias to /dev/cons.

The first warning was gone... but only the first one!

It was not just a matter of warnings: I was unable to interrupt a script (what you do with Ctrl+C on unix).

 

Down the rabbit hole, again.

I had to study the complex semantics of tty job control (asking boring questions to tg, again).
I had to fix setsid, getsid, setpgid, getpgid, getpgrp and to add support for termios' tcgetpgrp and tcsetpgrp.
Worse: I had to mostly rewrite the file server I had just written. Sob!

It took a while.
In the process I realized that a sys/posixly instance actually represents a single terminal session..

 

... did I say it took a while?

 

Then, suddenly... I saw this, and it was like an epiphany:

mksh running in Jehanne with full job control

 

MirBSD's Korn Shell was working on Jehanne! What a happy new year! :-D

Groeten vanuit Brussels, tijdens FOSDEM

2018-02-03 by tg@
Tags: event personal

I’m in Bruxelles again, as every year since 2001, for FOSDEM. (I only missed OSDEM in 2000, mostly due to the curse of late birth.)

To revive a tradition, I’ve attempted (and successfully at that!) to find a place where we can eat Couscous Merguez, and we met up with bsiegert@ and had some nice conversation and, besides the overly LOUD!!! belly dance, delicious food. It was nice to catch up with each other again.

Other than that, see you over the next few days at ULB! Don’t miss the MuseScore booth and the two Teckids talks.

Colophon: complexity sucks.

FOSDEM

2018-01-10 by tg@
Tags: event fun

I’ve been going to FOSDEM for about half of my lifetime, give or take a year I think. So, of course, I will be there again this year. Thanks to my employer for sponsoring travel and accommodation again.

It’s a bit annoying that the future of alternative OSes is a bit misty right now, depending on the hardware, but we’re continuing development, in subprojects (like mksh(1) and jupp(1), for example) and other projects (like Debian and MuseScore, whom I’ll meet at FOSDEM again) while researching possible fixes for the security theatre.

We had joy, we had fun… ♫ @ #34c3

2017-12-28 by tg@
Tags: event fun snapshot

MirBSD/sparc poster mod

MirBSD on the Sun

FrOSCon

2017-08-16 by tg@
Tags: event fun pcli personal

I’ll not respond, much, until next Monday. We have FrOSCon.

[PSA] Fixing CVE-2017-12836 (Debian #871810) in GNU cvs

2017-08-11 by tg@
Tags: archaeology bug debian pcli security

Considering I’ve become the de-facto upstream of cvs(GNU) even if not yet formally the de-iure upstream maintainer, fixing this bug obviously falls to me — not quite the way I had planned passing this evening after coming home from work and a decent and, worse, very filling meal at the local Croatian restaurant. But, so’s life.

The problem here is basically that CVS invokes ssh(1) (well, rsh originally…) but doesn’t add the argument separator “--” before the (user-provided) hostname, which when starting with a hyphen-minus will be interpreted by ssh as an argument. (Apparently the other VCSes also had additional vulnerabilities such as not properly escaping semicoloi or pipes from the shell or unescaping percent-escaped fun characters, but that doesn’t affect us.)

The obvious fix and the one I implemented first is to simply add the dashes. This will also be backported to Debian {,{,old}old}stable-security.

Then I looked at other VCSes out of which only one did this, but they all added extra paranoia hostname checks (some of them passing invalid hostnames, such as those with underscores in them). OK, I thought, then also let’s add extra checks to CVS’ repository reference handling. This will end up in Debian sid and MirBSD, pending passing the regression tests of course… hah, while writing this article I had to fixup because a test failed. Anyway, it’s not strictly necessary AFAICT to fix the issue.

Update, about 2⅕ hours past midnight (the testsuite runs for several hours): of course, the “sanity” testsuite (which itself is rather insane…) also needs adjustments, plus a bonus fix (for something that got broken when the recent allow-root-regex patch was merged and got fixed in the same go to…night).

tl;dr: a fix will end up in Debian *stable-security and can be taken out of my mail to the bugreport; another few changes for robustness are being tested and then added to both MirBSD and Debian sid. The impact is likely small, as it’s hard to get a user (if you find one, in the first place) to use a crafted CVSROOT string, which is easy to spot as well.

Update, Monday: apparently someone took care of the DSA and DLA yesterday after ACCEPTing the uploads — thanks, I was outside during the day.

Update 2017-08-25: It was noted that ssh(1) does not parse its command line correctly, and therefore the patch above might not be enough in the general case. However, I still think it’s good enough for CVS because it constructs its command line in a way that doesn’t let users exploit that bug.

mksh R56 was released with experimental fixes for the “history no longer persisted when HISTFILE near-full” and interactive shell cannot wait on coprocess by PID issues (I hope they do not introduce any regressioins) and otherwise as a bugfix release. You might wish to know the $EDITOR selection mechanism in dot.mkshrc changed. Some more alias characters are allowed again, and POSIX character classes (for ASCII, and EBCDIC, only) appeared by popular vote.

mksh now has a FAQ; enjoy. Do feel free to contribute (answers, too, of course).

The jupp text editor has also received a new release; asides from being much smaller, and updated (mksh too, btw) to Unicode 10, and some segfault fixes, it features falling back to using /dev/tty if stdin or stdout is not a terminal (for use on GNU with find | xargs jupp, since they don’t have our xargs(1) -o option yet), a new command to exit nonzero (sometimes, utilities invoking the generic visual editor need this), and “presentation mode”.

Presentation mode, crediting Natureshadow, is basically putting your slides as (UTF-8, with fancy stuff inside) plaintext files into one directory, with sorting names (so e.g. zero-padded slide numbers as filenames), presenting them with jupp * in a fullscreen xterm. You’d hit F6 to switch to one-file view first, then present by using F8 to go forward (F7 to go backward), and, for demonstrations, F9 to pipe the entire slide through an external command (could be just “sh”) offering the previous one as default. Simple yet powerful; I imagine Sven Guckes would love it, were he not such a vim user.

The new release is offered as source tarball (as usual) and in distribution packages, but also, again, a Win32 version as PKZIP archive (right-click on setup.inf and hit I̲nstall to install it). Note that this comes with its own (thankfully local) version of the Cygwin32 library (compatible down to Windows 95, apparently), so if you have Cygwin installed yourself you’re better off compiling it there and using your own version instead.

I’ve also released a new DOS version of 2.8 with no code patches but an updated jupprc; the binary (self-extracting LHarc archive) this time comes with all resource files, not just jupp’s.

Today, the jupprc drop-in file for JOE 3.7 got a matching update (and some fixes for bugs discovered during that) and I added a new one for JOE 4.4 (the former being in Debian wheezy, the latter in jessie, stretch and buster/sid). It’s a bit rudimentary (the new shell window functionality is absent) but, mostly, gives the desired jupp feeling, more so than just using stock jstar would.

CVS’ ability to commit to multiple branches of a file at the same time, therefore grouping the commit (by commitid at least, unsure if cvsps et al. can be persuaded to recognise it). If you don’t know what cvs(GNU) is: it is a proper (although not distributed) version control system and the best for centralised tasks. (For decentral tasks, abusing git as pseudo-VCS has won by popularity vote; take this as a comparison.)

If desired, I can make these new versions available in my “WTF” APT repository on request. (Debian buster/sid users: please change “https” to “http” there, the site is only available with TLSv1.0 as it doesn’t require bank-level security.)

I’d welcome it very much if people using an OS which does not yet carry either to package it there. Message me when one more is added, too ☺

In unrelated news I uploaded MuseScore 2.1 to Debian unstable, mostly because the maintainers are busy (though I could comaintain it if needed, I’d just need help with the C++ and CMake details). Bonus side effect is that I can now build 2.2~ test versions with patches of mine added I plan to produce to fix some issues (and submit upstream) ☻

In other news, I’m working on a new i386+sparc MirBSD snapshot more than ever. Mostly to get everything old out from under my feet before tackling the LibreSSL import (to get TLSv1.2 support, due to the aforementioned idio…decision). I’ve yet to see whether our G++ port works on sparc, and I’ve yet to create ports for libGLU and xlock which used to be in the base X system but had to go away for being written in an unmaintainable language (plus a system is only reliable if it has only one libstdc++), but it’ll be a good stepping stone (plus mfny asked for a sparc snapshot on IRC). I was considering distributing ISOs at FrOSCon but, with an installed user base in the single digits (likely), you can imagine how useful that’d be. (Fun side idea: distribute ISOs with a boot menu where you can choose not only MirBSD installer or live system but also “minimal Debian system directly booting into the MirBSD live system running under qemu-kvm”. But I’ve got not enough spare time right now.)

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

MirOS Logo