ubiquitous

About Ubiquitous

Ubiquitous is a very easy-to-use library for persistent configuration storage. It automatically takes care of finding a suitable place to save your data, and provides simple functions to access and modify the data within.

How To

Load ubiquitous through ASDF or Quicklisp.

(ql:quickload :ubiquitous)

The main functions you will be using are restore, and value. The former loads the configuration according to a designator, and the latter is an accessor allowing you to retrieve and set properties.

(restore 'my-config)
(setf (value :test) "hi!")
(value :test)

Ubiquitous will not perform any loading unless you tell it to. Thus, before the storage is truly persistent, you need to tell it where to restore from. Then, when values are set, it automatically saves the configuration to file. The location of the file is stored in *storage-pathname*, which is automatically computed according to what is most suitable for the given restore designator and OS. On Windows it will be under %HOME%\AppData\Local\common-lisp\ubiquitous\ and everywhere else under ~/.config/common-lisp/ubiquitous/. The exact behaviour of the pathname choosing is documented in designator-pathname.

value doesn't take a single name, but rather a path to a configuration value. The way things are traversed is handled by the field generic accessor. It tries to handle a number of commonly used structures, but you might have to extend it for your own classes, if you want to store those directly and traverse them. If a place does not exist yet, Ubiquitous will try to augment it if possible by creating a hash-table. This allows you to directly write a long path without having to worry about the containers existing.

(setf (value 'something 'that 'goes 6 'levels 'deep) "Calling from the depths!")

Often times for configuration one might want to specify a default value to use.

(defaulted-value "localhost" :hostname)

In case you need to remove a value, there's remvalue.

(remvalue 'something 'that 'goes 6 'levels 'deep)

By default, an extended s-expression format is used to store things in a file. If you need a different format, you can add methods to read-storage and write-storage, and set *storage-type* to your type name. Since (setf value) automatically calls offload to persist the storage, this might lead to a lot of saving all over the place. In order to avoid this, you can bundle a block of operations in a with-transaction form, which will only perform an offload once the block exits.

Ubiquitous in itself does not have any external dependencies, so you may also bundle it into a singular file to just load using ASDF:

(asdf:operate :build-op :ubiquitous)

Which will produce an independent ubiquitous.lisp file in (asdf:system-source-directory :ubiquitous).

Concurrency

By default Ubiquitous does not try to handle concurrent access in any way. The reason for this is not laziness, but merely the desire to avoid dependencies for those that don't need it. However, if you require safe concurrent access and handling of the storage, simply load ubiquitous-concurrent instead of ubiquitous. This will also pull in bordeaux-threads and establish additional methods around the standard definitions that will ensure concurrency safety.

This will still work irregardless of how many different storage objects you use, as the locking on the operations happens on the currently accessed storage object itself, rather than on a global lock. In order to avoid needless locking and unlocking, you should bundle your operations into a with-transaction block, which will only perform a lock once.

Metadata

Since version 2.0, Ubiquitous will output a metadata header into the configuration file. It will typically have the following structure:

; meta (:version 1.0 :package "CL-USER")

The purpose of this header is to ensure print/read consistency. When offloading, the current *package* is output into the header, so that when the configuration is restored, unqualified symbols in the configuration are read with the same home package as they were printed with.

If the header is missing your configuration will still read fine, and since it is a comment, configurations will still be compatible when read with version 1.0 of Ubiquitous. An error is however signalled if the header is malformed, or the referred package cannot be found.

If you implement your own storage format, you should ensure that you output and respect the same header, or a similar header. See maybe-read-metadata, with-processed-metadata, print-metadata, and related functions.

Shortcomings

A couple of shortcomings exist in Ubiquitous, almost by necessity. As you might know out of experience, certain modifying operations are not possible to do without being able to modify the container of the object itself. As an example, poping an element off the head of the list requires setting the variable that contains the list, rather than the list itself. This sort of thing is rather annoying to model in a generic manner without complicating the common case needlessly. Furthermore, in a couple of instances ambiguity arises due to multiple actions being possible.

In detail, the following operations are supported suboptimally, or not at all:

• remfield on a list with one element left that is to be removed. The car of the cons will simply be set to NIL.
• remfield on a vector. This would require shifting fields and potentially adjusting it. The desired effect cannot be estimated and so Ubiquitous does not support the operation at all.
• remfield on a standard-object, as class-slots cannot be removed without the MOP and you probably wouldn't want that to happen anyway.
• setf field on a list or vector where the field is an index exceeding the length. In this case, an error is signalled as extending the object to the required length might not be a desired or possible effect.
• setf field on a list where the field is a symbol or string. Lists are commonly used as alists or plists. Ubiquitous strictly expects alists.

Another shortcoming is in the department of serialisation. Ubiquitous does not try to be overly smart about things, which especially comes into effect when serialising standard-objects. Ubiquitous saves the class' slots and restores it by calling allocate-instance without initargs and then setf slot-value-ing one slot after the other. If you need more tailored support for serialising your object, you must extend define-ubiquitous-reader and define-ubiquitous-writer, or write a new storage format altogether. Furthermore, since the default behaviour is to use the lisp printer and reader (with special handling for hash-table, standard-object, standard-class, and package) to serialise objects, several things might get lost in translation, such as the fill-pointer and adjustability of a vector.

System Information

Definition Index

• UBIQUITOUS

  • ORG.SHIRAKUMO.UBIQUITOUS

  No documentation provided.

  • EXTERNAL SPECIAL-VARIABLE

    *CHANGED*

    Source

    When non-NIL it means a change has occurred and the config should be offloaded.

  • EXTERNAL SPECIAL-VARIABLE

    *COMMIT*

    Source

    When non-NIL, an OFFLOAD is performed after a call to (SETF VALUE) or REMVALUE.

  • EXTERNAL SPECIAL-VARIABLE

    *METADATA-PREFIX*

    Source

    The prefix that is used to recognise the metadata header.

  • EXTERNAL SPECIAL-VARIABLE

    *METADATA-VERSION*

    Source

    The current version of the configuration metadata.
    
    Should be a float.

  • EXTERNAL SPECIAL-VARIABLE

    *STORAGE*

    Source

    Special variable containing the current root storage object.
    Defaults to an EQUAL hash-table.

  • EXTERNAL SPECIAL-VARIABLE

    *STORAGE-PATHNAME*

    Source

    The pathname for the file where the current *STORAGE* is stored.
    Defaults to (DESIGNATOR-PATHNAME :GLOBAL *STORAGE-TYPE*).
    
    See DESIGNATOR-PATHNAME
    See *STORAGE-TYPE*

  • EXTERNAL SPECIAL-VARIABLE

    *STORAGE-TYPE*

    Source

    An indicator for the type of storage format to use.
    Defaults to :LISP
    
    Only used as a discerning argument to READ/WRITE-STORAGE.

  • EXTERNAL CONDITION

    BAD-CONFIGURATION-PACKAGE

    Source

    Error signalled when the metadata header refers to an unknown package.
    
    See BAD-PACKAGE-NAME
    See METADATA-CONDITION

  • EXTERNAL CONDITION

    BAD-METADATA-HEADER

    Source

    Error signalled when the metadata header is malformed.
    
    See HEADER
    See METADATA-CONDITION

  • EXTERNAL CONDITION

    METADATA-CONDITION

    Source

    Superclass for conditions related to configuration metadata.

  • EXTERNAL CONDITION

    NO-STORAGE-FILE

    Source

    Condition signalled when the storage FILE to be RESTOREd does not exist.
    
    This does not usually denote a problem, but can be useful to know about
    if you want perform certain actions on what is probably a first-time
    launch.
    
    See FILE
    See RESTORE

  • EXTERNAL CONDITION

    UNKNOWN-READER-TYPE

    Source

    Error signalled if an unknown structure type is encountered in the config.
    
    See READER-TYPE
    See UBIQUITOUS-READER

  • EXTERNAL CONDITION

    UNKNOWN-VERSION

    Source

    Warning signalled when the metadata header contains an unknown version number.
    
    See VERSION
    See METADATA-CONDITION

  • EXTERNAL FUNCTION

    CONFIG-DIRECTORY

    Source

    Returns a hopefully suitable directory for ubiquitous configuration files.
    
    On Windows this is (USER-HOMEDIR-PATHNAME)/AppData/Local/common-lisp/ubiquitous
    On other systems   (USER-HOMEDIR-PATHNAME)/.config/common-lisp/ubiquitous

  • EXTERNAL FUNCTION

    CONFIG-PATHNAME

    • TYPE

    Source

    Returns a pathname with the proper directory and type set.
    
    See CONFIG-DIRECTORY

  • EXTERNAL FUNCTION

    FIND-METADATA-PACKAGE

    • NAME

    Source

    Attempt to find a package of the given name for metadata package resolution.
    
    This is like FIND-PACKAGE, except that it signals an error of type
    BAD-CONFIGURATION-PACKAGE if no package could be found, and establishes
    the USE-VALUE and CONTINUE restarts to recover from the error.
    
    USE-VALUE lets you pass a package designator to use in place. Note that
    if you do not pass a PACKAGE object, it will re-test the designator the
    same way.
    
    CONTINUE will instead use the current *PACKAGE*. 
    
    Note that using an alternate package may lead to symbols with the wrong
    home package in the processed configuration.
    
    See BAD-CONFIGURATION-PACKAGE

  • EXTERNAL FUNCTION

    GENERATE-METADATA

    Source

    Generates valid metadata for the current environment.

  • EXTERNAL FUNCTION

    LAZY-LOADER

    • ACTION
    • FIELD
    • &OPTIONAL
    • VALUE

    Source

    A function that is to be used as a direct *STORAGE* value to delay the restoring.
    
    When called, the function will call RESTORE and then delegate the given
    action to the proper function (FIELD, (SETF FIELD), REMFIELD) using the
    *STORAGE* as object.
    
    See *STORAGE*
    See FIELD
    See REMFIELD
    See WITH-LOCAL-STORAGE

  • EXTERNAL FUNCTION

    MAYBE-READ-METADATA

    • STREAM

    Source

    Attempts to read a metadata line from the stream.
    
    If the data at the current stream starts with *METADATA-PREFIX*, the line is
    consumed and read. The read metadata structure is returned. If reading fails,
    an error of type BAD-METADATA-HEADER is signalled.
    
    See *METADATA-PREFIX*
    See BAD-METADATA-HEADER

  • EXTERNAL FUNCTION

    PRINT-METADATA

    • STREAM
    • &OPTIONAL
    • METADATA

    Source

    Writes a metadata header to the stream.
    
    Writes a valid metadata header line to the stream, with the METADATA as the
    header contents. The header will start with *METADATA-PREFIX* and will only
    consume a single line.
    
    See *METADATA-PREFIX*
    See GENERATE-METADATA

  • EXTERNAL FUNCTION

    PROCESS-METADATA

    • META

    Source

    Processes the metadata and applies its effects.
    
    This may signal conditions if the metadata is malformed, or the current
    environment is not agreeable. Two restarts are established to recover:
    
    USE-VALUE allows you to supply alternate metadata. The metadata is then
    processed in place of the original.
    
    CONTINUE will simply ignore the metadata and return successfully.
    
    See CHECK-METADATA
    See BAD-METADATA-HEADER
    See BAD-CONFIGURATION-PACKAGE
    See UNKNOWN-VERSION

  • EXTERNAL GENERIC-FUNCTION

    AUGMENT

    • OBJECT
    • FIELD
    • SECONDARY

    Source

    Attempts to augment OBJECT on FIELD to be able to host a SECONDARY place.
    
    This is done by (SETF FIELD) a hash-table on the given OBJECT and FIELD. The
    type of SECONDARY decides the hash-table test to use:
      SYMBOL, INTEGER, CHARACTER          --- EQL
      STRING, BIT-VECTOR, PATHNAME        --- EQUAL
      ARRAY, STRUCTURE-OBJECT, HASH-TABLE --- EQUALP
    
    See FIELD

  • EXTERNAL GENERIC-FUNCTION

    BAD-PACKAGE-NAME

    • CONDITION

    Returns the bad package name that the condition captured.
    
    See BAD-CONFIGURATION-PACKAGE

  • EXTERNAL GENERIC-FUNCTION

    CALL-WITH-TRANSACTION

    • FUNCTION
    • &KEY
    • STORAGE
    • TYPE
    • DESIGNATOR

    Source

    Calls FUNCTION with *COMMIT* set to NIL and offloads if necessary upon exit.
    
    OFFLOAD is only called if *CHANGED* is non-NIL. Otherwise no change is assumed to
    have taken place and the offload is prevented to avoid unnecessary writing.
    
    The keyword parameters replace the bindings for *STORAGE* *STORAGE-TYPE* and
    *STORAGE-PATHNAME* respectively.
    
    See *COMMIT*
    See *CHANGED*
    See OFFLOAD

  • EXTERNAL GENERIC-FUNCTION

    DEFAULTED-VALUE

    • DEFAULT
    • &REST
    • PATH

    Source

    Same as VALUE, but automatically returns and sets DEFAULT if the field cannot be found.
    
    See VALUE

  • EXTERNAL GENERIC-FUNCTION

    DESIGNATOR-PATHNAME

    • DESIGNATOR
    • TYPE

    Source

    Attempts to automatically find the proper pathname for the given DESIGNATOR and TYPE.
    
    If DESIGNATOR is..
      An absolute PATHNAME:
        The pathname is used as-is.
    
      A relative PATHNAME: 
        The pathname is merged with that of CONFIG-DIRECTORY.
    
      A STRING:
        The string is turned into a pathname by PATHNAME and merged with CONFIG-PATHNAME.
    
      A SYMBOL:
        A pathname with the symbol's symbol-name as pathname-name, the CONFIG-PATHNAME's
        pathname-type, and the defaults from PACKAGE-PATHNAME is constructed.
    
    Examples:
    (designator-pathname #p"/a" :lisp)   --- #p"/a"
    (designator-pathname #p"a" :lisp)    --- #p"~/.config/common-lisp/ubiquitous/a"
    (designator-pathname "a" :lisp)      --- #p"~/.config/common-lisp/ubiquitous/a.conf.lisp"
    (designator-pathname :foo :lisp)     --- #p"~/.config/common-lisp/ubiquitous/foo.conf.lisp"
    (designator-pathname #:foo :lisp)    --- #p"~/.config/common-lisp/ubiquitous/foo.conf.lisp"
    (designator-pathname 'cl:find :lisp) --- ERROR
    (designator-pathname 'foo:bar :lisp) --- #p"~/.config/common-lisp/ubiquitous/foo/bar.conf.lisp"
    
    See PACKAGE-PATHNAME
    See CONFIG-DIRECTORY

  • EXTERNAL GENERIC-FUNCTION

    DESTROY

    • &OPTIONAL
    • DESIGNATOR
    • TYPE

    Source

    Destroys *STORAGE* by deleting its file and restoring it to an empty hash table.
    
    The file used to destroy the storage is calculated by passing
    DESIGNATOR (defaulting to *STORAGE-PATHNAME*) and TYPE (defaulting to
    *STORAGE-TYPE*) to DESIGNATOR-PATHNAME.
    
    This sets *STORAGE*, *STORAGE-TYPE*, *STORAGE-PATHNAME*, and *CHANGED*.
    
    See *STORAGE*
    See *STORAGE-TYPE*
    See *STORAGE-PATHNAME*
    See *CHANGED*
    See DESIGNATOR-PATHNAME

  • EXTERNAL GENERIC-FUNCTION

    FIELD

    • OBJECT
    • FIELD
    • &OPTIONAL
    • DEFAULT

    Source

    Access FIELD on OBJECT if possible. Returns DEFAULT if FIELD is not present.
    The secondary return value is a boolean depicting whether the field could be found.
    
    This is SETF-able. However, while some objects and field combinations may be used
    to read a field, an equivalent SETF method must not necessarily exist.
    
    In the case where the object is a function, the function is called as follows:
     (field func field default)      => (funcall func :get field default)
     (setf (field func field) value) => (funcall func :set field value) 
    
    Note that if there is no matching method to look up the requested field, an error
    is signalled.

  • EXTERNAL GENERIC-FUNCTION

    (SETF FIELD)

    • VALUE
    • OBJECT
    • FIELD

    Source

    No documentation provided.
  • EXTERNAL GENERIC-FUNCTION

    FILE

    • CONDITION

    To be used on NO-STORAGE-FILE, returns the pathname to the file that could not be found.

  • EXTERNAL GENERIC-FUNCTION

    (SETF FILE)

    • NEW-VALUE
    • CONDITION

    No documentation provided.
  • EXTERNAL GENERIC-FUNCTION

    HEADER

    • CONDITION

    Returns the header that is malformed.
    
    See BAD-METADATA-HEADER

  • EXTERNAL GENERIC-FUNCTION

    OFFLOAD

    • &OPTIONAL
    • DESIGNATOR
    • TYPE
    • STORAGE

    Source

    Offloads *STORAGE* by writing it to file.
    
    The file used to read the storage is calculated by passing
    DESIGNATOR (defaulting to *STORAGE-PATHNAME*) and TYPE (defaulting to
    *STORAGE-TYPE*) to DESIGNATOR-PATHNAME.
    
    The file is first written to a temporary one and then renamed to the
    actual file to avoid potential errors or interruptions that would result
     in a garbled configuration file.
    
    This sets *STORAGE-TYPE*, *STORAGE-PATHNAME*, and *CHANGED*.
    
    During OFFLOAD, the following restarts are active:
       ABORT  Aborts and does not set any of the usual variables.
    
    See *STORAGE*
    See *STORAGE-TYPE*
    See *STORAGE-PATHNAME*
    See *CHANGED*
    See DESIGNATOR-PATHNAME
    See WRITE-STORAGE

  • EXTERNAL GENERIC-FUNCTION

    PACKAGE-DIRECTORY

    • PACKAGE

    Source

    Returns the directory for config files suitable for this package.
    
    By default:
    For the CL package, an error is signalled.
    For the KEYWORD and NIL packages, the CONFIG-DIRECTORY is returned.
    Otherwise, a subdirectory within the CONFIG-DIRECTORY is returned according to the
    package's name.
    
    The user may add methods to this function to customise the behaviour of their own
    packages.
    
    See CONFIG-DIRECTORY

  • EXTERNAL GENERIC-FUNCTION

    READ-STORAGE

    • TYPE
    • STREAM

    Source

    Reads a storage object from STREAM, which must be stored in a format suitable for TYPE.
    Returns the read storage object.

  • EXTERNAL GENERIC-FUNCTION

    READER-TYPE

    • CONDITION

    Returns the type that was attempted to be read.
    
    See UNKNOWN-READER-TYPE

  • EXTERNAL GENERIC-FUNCTION

    REMFIELD

    • OBJECT
    • FIELD

    Source

    Removes FIELD from OBJECT if possible.
    The secondary return value is a boolean depicting whether the field was removed.
    
    In the case where the object is a function, the function is called as follows:
      (remfield func field)          => (funcall func :remove field)
    
    Note that if there is no matching method to look up the requested field, an error
    is signalled.

  • EXTERNAL GENERIC-FUNCTION

    REMVALUE

    • &REST
    • PATH

    Source

    Removes the value denoted by the PATH.
    The secondary return value is a boolean depicting whether the field could be found.
    
    First traverses *STORAGE* until the last field in PATH by FIELD, then uses REMFIELD
    on the last remaining field. If no PATH is given, the *STORAGE* is reset to an empty
    hash-table.
    
    See FIELD
    See REMFIELD

  • EXTERNAL GENERIC-FUNCTION

    RESTORE

    • &OPTIONAL
    • DESIGNATOR
    • TYPE

    Source

    Restores *STORAGE* by reading it from file if possible and returns it.
    
    The file used to read the storage is calculated by passing
    DESIGNATOR (defaulting to *STORAGE-PATHNAME*) and TYPE (defaulting to
    *STORAGE-TYPE*) to DESIGNATOR-PATHNAME. If it exists, a stream is opened
    and subsequently passed to READ-STORAGE. The result thereof is used as
    the new storage object. If it does not exist, a warning of type 
    NO-STORAGE-FILE is signalled and a new EQUAL hash-table is used for the
    storage object (unless a restart is invoked of course).
    
    This sets *STORAGE*, *STORAGE-TYPE*, *STORAGE-PATHNAME*, and *CHANGED*.
    
    During OFFLOAD, the following restarts are active:
      USE-NEW-STORAGE  Takes one argument to use as the new storage instead.
      ABORT            Aborts and does not set any of the usual variables.
    
    See *STORAGE*
    See *STORAGE-TYPE*
    See *STORAGE-PATHNAME*
    See *CHANGED*
    See NO-STORAGE-FILE
    See DESIGNATOR-PATHNAME
    See READ-STORAGE

  • EXTERNAL GENERIC-FUNCTION

    VALUE

    • &REST
    • PATH

    Source

    Traverses *STORAGE* by the fields in PATH and returns the value if it can be found.
    The secondary return value is a boolean depicting whether the field could be found.
    
    This is SETF-able. If a PATH is set that is made up of fields that do not exist yet,
    then these fields are automatically created as necessary (if possible) by usage of
    AUGMENT. Setting with no PATH given sets the value of *STORAGE*. After setting a value,
    OFFLOAD is called, unless *COMMIT* is NIL
    
    See FIELD
    See AUGMENT
    See OFFLOAD
    See *COMMIT*

  • EXTERNAL GENERIC-FUNCTION

    (SETF VALUE)

    • VALUE
    • &REST
    • PATH

    Source

    No documentation provided.
  • EXTERNAL GENERIC-FUNCTION

    VERSION

    • CONDITION

    Returns the version captured by the condition.
    
    See UNKNOWN-VERSION

  • EXTERNAL GENERIC-FUNCTION

    WRITE-STORAGE

    • TYPE
    • STREAM
    • STORAGE

    Source

    Writes the STORAGE object to STREAM in a format suitable for TYPE.
    Returns the written STORAGE object.

  • EXTERNAL MACRO

    DEFINE-UBIQUITOUS-READER

    • TYPE
    • FORM
    • &BODY
    • BODY

    Source

    Define a new function that produces an object of TYPE by parsing the read FORM.

  • EXTERNAL MACRO

    DEFINE-UBIQUITOUS-WRITER

    • TYPE
    • OBJECT
    • &OPTIONAL
    • PRIORITY
    • &BODY
    • BODY

    Source

    Define a new function that produces a list of objects to be written to reproduce OBJECT of TYPE.

  • EXTERNAL MACRO

    WITH-LOCAL-STORAGE

    • DESIGNATOR
    • &KEY
    • TYPE
    • STORAGE
    • TRANSACTION
    • &BODY
    • BODY

    Source

    Useful for completely encapsulating the storage in a local block.
    
    Unlike WITH-STORAGE, this also binds the *STORAGE-TYPE* and
    *STORAGE-PATHNAME*. If TRANSACTION is non-NIL, WITH-TRANSACTION is
    used, and otherwise a simple LET*. STORAGE defaults to the LAZY-LOADER
    function, meaning that if the storage is never accessed, it is never
    loaded to begin with. This, along with WITH-TRANSACTION can be a
    good optimisation to avoid unnecessary disk access.
    
    See *STORAGE*
    See *STORAGE-TYPE*
    See *STORAGE-PATHNAME*
    See WITH-TRANSACTION
    See LAZY-LOADER

  • EXTERNAL MACRO

    WITH-PROCESSED-METADATA

    • META
    • &BODY
    • BODY

    Source

    Wraps the body in an environment where metadata can be safely applied and processes the given metadata within.
    
    At the moment, this simply binds *PACKAGE* to ensure the package indicated
    by the metadata can be applied without influencing the surrounding environment.
    
    See PROCESS-METADATA

  • EXTERNAL MACRO

    WITH-STORAGE

    • STORAGE
    • &BODY
    • BODY

    Source

    Binds *STORAGE* to the given STORAGE object, ensuring a local configuration.

  • EXTERNAL MACRO

    WITH-TRANSACTION

    • &KEY
    • STORAGE
    • TYPE
    • DESIGNATOR
    • &BODY
    • BODY

    Source

    Executes BODY within a transaction.
    
    See CALL-WITH-TRANSACTION