Design notes

Table of contents

Portability

This tool heavily relies on strace utility, and thus is dependent on Linux for now.

I keep OS dependencies as small and localized as possible to ease porting.
Currently, both dependencies to GNAT for the Spawn procedure, and to the OS (the strace call itself) are in a short separate procedure in smk-main-run_command.adb.

That said, I do not intend to port smk to other platforms myself, but many OSes provide utilities similar to strace, so it may be done.
And obviously, any contribution, including ports, is welcome.

smk is currently only tested on my Debian x86_64 box.

On file systems time stamp

The first issue I came across is the poor resolution (1s) of the GNAT Modification_Time ([cf. to this discussion on com.lang.ada] (https://groups.google.com/forum/#!topic/comp.lang.ada/CM7axwAh1B8)).

Behind the GNAT specific issue, there is a file system, and not all file systems have the same resolution :

File system resolution
FAT/VFAT 2 s for modification time (but 10 ms for creation)
NTFS 100 ns
EXT4 1 ns

But the ext4 time update is based on the cached kernel time that may be updated on interrupt only every 10ms.

Anyway, 10ms would be better than the current Modification_Time results, that is 1s...

  • Note that the need is not really on precision, but on :
    1. resolution, and more specifically about ordering. Having lots of files with the same modification time prevent smk proper working.
    2. coherency between the wall clock time and the file system time stamp.

The first implementation was just asking to strace the list of accessed files. Then, accessed files where classified "targets" or "sources" according to their time stamp relatively to the smk execution time. For example, the wall time Clock when running a command was 20:36:00.12. A file is generated by this command at 20:36:00.27. It should be considered as a target. But as Modification_Time truncates the time to one-second, and returns 20:36:00.00, it was considered as a source.

The current implementation uses a more in deep strace output analysis to identify read vs write operation (Cf. hereafter Strace file operations output analysis). The code is (slightly) more complex, but smk no more relies on files time stamp to determine sources and target.

The major part of the problem is solved, but there is still a bug due to the resolution problem.

If you run in a smkfile file: 1. a command depending on the s1 source at 20:36:00.15 2. a command that "touch" the s1 source at 20:36:00.25 3. a command depending on the s1 source at 20:36:00.45

The third command should re-run. But smk will see that the previous run was at 20:36:00.15, and the (truncated) file time at 20:36:00.00.
And conclude that there is nothing to do...

Hence some sleep 1 in the test Makefile!

As far as you don't run smkfile embedding call to smk recursively, you should not be bothered by this.

Options of make and mk, for inspiration...

mk

−a      Assume all targets to be out of date. Thus, everything is updated.

−d[egp]Produce debugging output (p is for parsing, g for graph building, e for execution).

−e      Explain why each target is made.

−i      Force any missing intermediate targets to be made.

−k      Do as much work as possible in the face of errors.

−n      Print, but do not execute, the commands needed to update the targets.

−s      Make the command line arguments sequentially rather than in parallel.

−t      Touch (update the modified date of) file targets, without executing any recipes.

−wtarget1,target2,...
Pretend the modify time for each target is the current time; useful in conjunction with −n to learn what updates would be triggered by modifying the targets.

make

-B, --always-make
Unconditionally make all targets.

-d
Print debugging information in addition to normal processing. The debugging information says which files are being considered for remaking, which file-times are being compared and with what results, which files actually need to be remade, which implicit rules are considered and which are applied---everything interesting about how make decides what to do.

-i, --ignore-errors
Ignore all errors in commands executed to remake files.

-k, --keep-going
Continue as much as possible after an error. While the target that failed, and those that depend on it, cannot be remade, the other dependencies of these targets can be processed all the same.

-n, --just-print, --dry-run, --recon
Print the commands that would be executed, but do not execute them.

-q, --question
''Question mode''. Do not run any commands, or print anything; just return an exit status that is zero if the specified targets are already up to date, nonzero otherwise.

-s, --silent, --quiet
Silent operation; do not print the commands as they are executed.

-S, --no-keep-going, --stop
Cancel the effect of the -k option. This is never necessary except in a recursive make where

-v, --version
Print the version of the make program plus a copyright, a list of authors and a notice that there is no warranty.

-W file, --what-if=file, --new-file=file, --assume-new=file
Pretend that the target file has just been modified. When used with the -n flag, this shows you what would happen if you were to modify that file. Without -n, it is almost the same as running a touch command on the given file before running make, except that the modification time is changed only in the imagination of make.

Strace file operations output analysis

Write operations (that cause the file to be considered as Target)

  • write - Écrire dans un descripteur de fichier

  • mkdir

  • open, openat : si O_WRONLY, or O_RDWR (The argument flags must include one of the following access modes: O_RDONLY, O_WRONLY, or O_RDWR). These request opening the file read-only, write-only, or read/write, respectively.

  • creat() : A call to creat() is equivalent to calling open() with flags equal to O_CREAT|O_WRONLY|O_TRUNC

  • chown, fchown, lchown

  • chmod, fchmod

  • link - Crée un nouveau nom pour un fichier

  • unlink - Détruire un nom et éventuellement le fichier associé

  • unlinkat - remove a directory entry relative to a directory file descriptor. The unlinkat() system call operates in exactly the same way as either unlink(2) or rmdir(2) (depending on whether or not flags includes the AT_REMOVEDIR flag) except for the differences described in this manual page.

  • fopen, fdopen, freopen - Fonctions d'ouverture de flux, sauf si mode = R ou R+
    L'argument mode pointe vers une chaîne commençant par l'une des séquences suivantes (d'autres caractères peuvent suivre la séquence) :

  • r : Ouvre le fichier en lecture. Le pointeur de flux est placé au début du fichier.

  • r+ : Ouvre le fichier en lecture et écriture. Le pointeur de flux est placé au début du fichier.

  • w : Tronque le fichier à son début ou ouvre le fichier en écriture. Le pointeur de flux est placé au début du fichier.

  • w+ : Ouvre le fichier en lecture et écriture. Le fichier est créé s'il n'existait pas. S'il existait déjà, sa longueur est ramenée à 0. Le pointeur de flux est placé au début du fichier.

  • a : Ouvre le fichier en ajout (écriture à la fin du fichier). Le fichier est créé s'il n'existait pas. Le pointeur de flux est placé à la fin du fichier.

  • a+ : Ouvre le fichier en lecture et ajout (écriture en fin de fichier)Le fichier est créé s'il n'existait pas. La tête de lecture initiale du fichier est placée au début du fichier mais la sortie est toujours ajoutée à la fin du fichier.

  • remove - Détruire un nom et éventuellement le fichier correspondant (appelle unlink ou rmdir suivant le fichier)

  • rename, renameat, renameat2 - change the name or location of a file

Read operations (that cause the file to be considered as Source)

  • open, openat : si O_RDONLY

  • fopen, fdopen, freopen si mode = R ou R+

  • readlink, readlinkat - read value of a symbolic link

I don't know operations (that are currently ignored!)

  • dup() et dup2() créent une copie du descripteur de fichier oldfd.

Ignored

  • close, fcntl, lseek,

  • mknod - Créer un nœud du système de fichiers (NB : sous Linux mknod() ne peut pas être utilisé pour créer des répertoires)

  • mount, umount, umount2 - Monter/démonter des systèmes de fichiers

  • umask() fixe le masque de création de fichiers du processus appelant, utilisé entre autre par open(2), mkdir(2

  • fsync, fdatasync - Synchroniser un fichier en mémoire avec le disque (flush)

  • mmap, munmap - Établir/supprimer une projection en mémoire (map/unmap) des fichiers ou des périphériques

  • fopen, fdopen, freopen - Fonctions d'ouverture de flux

  • access, faccessat - check user's permissions for a file

  • statfs, fstatfs : get filesystem statistics. The statfs() system call returns information about a mounted filesystem.

  • stat, fstat, lstat : Obtenir l'état d'un fichier (file status)

  • fstatat : the fstatat() system call is a more general interface for accessing file information which can still provide exactly the behavior of each of stat(), lstat(), and fstat().

  • getcwd, getwd, get_current_dir_name - get current working directory

On AT_FDCWD

The openat() system call operates in exactly the same way as open(2), except for the differences described in this manual page. If the pathname given in pathname is relative, then it is interpreted relative to the directory referred to by the file descriptor dirfd (rather than relative to the current working directory of the calling process, as is done by open(2) for a relative pathname).

If the pathname given in pathname is relative and dirfd is the special value AT_FDCWD, then pathname is interpreted relative to the current working directory of the calling process (like open(2)).

If the pathname given in pathname is absolute, then dirfd is ignored.

On at sufixed operations

openat() and other similar system calls suffixed "at" are supported for two reasons. First, openat() allows an application to avoid race conditions that could occur when using open(2) to open files in directories other than the current working directory. These race conditions result from the fact that some component of the directory prefix given to open() could be changed in parallel with the call to open(). Such races can be avoided by opening a file descriptor for the target directory, and then specifying that file descriptor as the dirfd argument of openat().

Second, openat() allows the implementation of a per-thread "current working directory", via file descriptor(s) maintained by the application. (This functionality can also be obtained by tricks based on the use of /proc/self/fd/dirfd, but less efficiently.)

  • faccessat
  • fchmodat
  • fchownat
  • fstatat
  • futimesat (obsolete)
  • linkat
  • mkdirat
  • mknodat
  • openat
  • readlinkat
  • renameat
  • symlinkat
  • unlinkat
  • utimensat