staple code parser

About Staple-Parser

This system implements a Lisp code parser to implement marking up definition references within code snippets.

How To

You can parse a lisp source snippets using parse:

(staple-code-parser:parse "(defun foo (a) (+ 1 a))")

This will return a list of "parse results". Parse results represent all information about the toplevel source form that was parsed. Typically you will want to pass this to parse-result->definition-list, which will return a list of definitions and their source locations that were found within the parse results.

(staple-code-parser:parse-result->definition-list *)
; => ((#<DEFINITIONS:FUNCTION +> (16 . 17)) (#<DEFINITIONS:MACRO DEFUN> (1 . 6)))

The definitions objects are from the Definitions library. Please see its documentation on how to handle these kinds of objects. This definition list is used in Staple to mark up the respective source parts with HTML links, but you could also use it for your own purposes.

Extending Staple-Parser

Since the parser does not compile or evaluate the code, it is missing a lot of information about what each symbol could be, hampering the quality of definition retrieval. You can help this out by implementing custom walkers for known forms that expand to parse results that are more easily understood.

The way to do this is twofold. You can either use define-walk-compound-form an expand into known parse results, transforming the contents as appropriate, or you can use define-walker-form to define a new parse result type. In the latter case you will also need to add define-sub-forms and define-definition-resolver to handle the traversal and lookup.

Have a look at the source files special-forms, standard-forms, and to-definitions for examples on how to use these.

System Information

Definition Index

• STAPLE-CODE-PARSER

  • ORG.SHIRAKUMO.STAPLE.CODE-PARSER

  No documentation provided.

  • EXTERNAL CLASS

    ENVIRONMENT

    Source

    Container for environment information used during walking.
    
    See PARENT
    See NAMESPACES
    See LOOKUP
    See AUGMENT-ENVIRONMENT!
    See AUGMENTED-ENVIRONMENT

  • EXTERNAL CLASS

    PLACEHOLDER

    Source

    This class represents symbols that are not present in the host.
    
    They are emitted in parsed code snippets in place of symbols that
    cannot be read properly.
    
    See PLACEHOLDER-NAME
    See PLACEHOLDER-PACKAGE
    See PLACEHOLDER-INTERN

  • EXTERNAL FUNCTION

    AUGMENT-ENVIRONMENT!

    • ENVIRONMENT
    • NAMES
    • VALUES

    Source

    Augments the given environment with the new values for the given names.
    
    Returns the modified environment.
    
    See LOOKUP
    See ENVIRONMENT

  • EXTERNAL FUNCTION

    AUGMENTED-ENVIRONMENT

    • PARENT
    • NAMES
    • VALUES
    • &KEY
    • CLASS

    Source

    Returns a new environment with the changed values in place.
    
    The old environment is a parent to the new one.
    
    See AUGMENT-ENVIRONMENT!
    See ENVIRONMENT

  • EXTERNAL FUNCTION

    PARSE-RESULT->DEFINITION-LIST

    • RESULT

    Source

    Turn the parse-result into a list of definitions and source locations.
    
    For instance:
      ((:CALL (0 . 10) (:VARIABLE (1 . 5) NULL) (:LITERAL (6 . 9) NIL)))
      => ((#<DEFINITIONS:FUNCTION NULL> (1 . 5)))
    
    This uses FIND-DEFINITIONS to find suitable definitions for a parse
    result, as well as SUB-RESULTS to traverse the parse result tree.
    
    See FIND-DEFINITIONS
    See SUB-RESULTS

  • EXTERNAL FUNCTION

    TIE-TO-SOURCE

    • SOURCE
    • DEFS

    Source

    Turns each def into a list of source and def.

  • EXTERNAL FUNCTION

    WALK-BINDINGS

    • BINDINGS
    • ENVIRONMENT

    Source

    Walk the set of LET bindings in the environment.
    
    Returns a list of cons cells where the CAR is the variable definition
    of the binding and the cdr is the parse result of the value.
    
    See WALK

  • EXTERNAL FUNCTION

    WALK-BODY

    • CST
    • ENVIRONMENT

    Source

    Same as WALK-IMPLICIT-PROGN, but filters out declarations from the cst.
    
    See WALK-IMPLICIT-PROGN

  • EXTERNAL FUNCTION

    WALK-IMPLICIT-PROGN

    • CST
    • ENVIRONMENT

    Source

    Walks the CST as a list of forms and returns the list of parse-results for each form.
    
    See WALK

  • EXTERNAL FUNCTION

    WALK-LAMBDA-LIKE

    • CST
    • ENVIRONMENT
    • &OPTIONAL
    • LAMBDA-LIST-PARSER

    Source

    Walk a lambda-like structure.
    
    Parses the lambda-list and body forms appropriately and returns a
    parse-result for a lambda. The given parser is used to process the
    lambda-list.
    
    See WALK-IMPLICIT-PROGN

  • EXTERNAL GENERIC-FUNCTION

    FIND-DEFINITIONS

    • TYPE
    • SOURCE
    • ARGS

    Returns any matching definitions for the given parse result.
    
    All parse results have the structure of (TYPE SOURCE . ARGS).
    Thus you can simply destructure it and pass the arguments to this
    function to retrieve its definitions.
    
    See DEFINE-DEFINITION-RESOLVER

  • EXTERNAL GENERIC-FUNCTION

    LOOKUP

    • NAME
    • NAMESPACE
    • ENVIRONMENT

    Looks up the name in the namespace of the environment.
    
    This will traverse the environment chain upwards until no parent can
    be found anymore in case the current environment's namespace does not
    contain the value.
    
    When used as a setf place the value is always stored in the given
    environment's namespace.
    
    See NAMESPACE
    See ENVIRONMENT

  • EXTERNAL GENERIC-FUNCTION

    (SETF LOOKUP)

    • VALUE
    • NAME
    • NAMESPACE
    • ENVIRONMENT

    No documentation provided.
  • EXTERNAL GENERIC-FUNCTION

    PARSE

    • INPUT

    Parses the input and returns a list of parse results, each for one toplevel.
    
    First uses READ-TOPLEVEL to read all toplevel forms, then uses WALK
    for each of the read CSTs to turn them into parse results.
    
    See READ-TOPLEVEL
    See WALK

  • EXTERNAL GENERIC-FUNCTION

    PLACEHOLDER-INTERN

    • OBJECT

    Returns whether the symbol being read is an internal or external symbol.
    
    See PLACEHOLDER

  • EXTERNAL GENERIC-FUNCTION

    PLACEHOLDER-NAME

    • OBJECT

    Returns the symbol-name of the symbol this is a placeholder for.
    
    See PLACEHOLDER

  • EXTERNAL GENERIC-FUNCTION

    PLACEHOLDER-PACKAGE

    • OBJECT

    Returns the symbol-package name of the symbol this is a placeholder for.
    
    See PLACEHOLDER

  • EXTERNAL GENERIC-FUNCTION

    READ-TOPLEVEL

    • INPUT

    Reads the toplevel of an input.
    
    The INPUT may be a string, pathname, or a stream (by default).
    Returns a list of CSTs representing all toplevel forms that were read.

  • EXTERNAL GENERIC-FUNCTION

    SUB-RESULTS

    • TYPE
    • ARGS

    Returns all parse results that are sub-results of this parse result.
    
    All parse results have the structure of (TYPE SOURCE . ARGS).
    Thus you can simply destructure it and pass the arguments to this
    function to retrieve its definitions.
    
    See DEFINE-SUB-RESULTS

  • EXTERNAL GENERIC-FUNCTION

    WALK

    • CST
    • ENVIRONMENT

    Walks the given CST in the environment.
    
    Should return a parse result structure.
    Parse results are lists of the following form:
    
      PARSE-RESULT ::= (TYPE SOURCE . ARGS)
      TYPE         --- The type of the form we've walked. Typically this
                       is a symbol of the form itself, like LAMBDA, or a
                       keyword if a generic variant is encountered like
                       for :CALLs and :MACROs.
      SOURCE       ::= (START . END)
      ARGS         --- Additional arguments for the parse result,
                       including additional parse-results.
    
    Generally see the overall concrete-syntax-tree system for explanations
    on how to use this.
    
    Note that you probably want to define a method on WALK-FORM instead,
    as that is called automatically as appropriate for each CST:CONST-CST,
    and WALK-ATOM is called for each CST:ATOM-CST.
    
    See ENVIRONMENT

  • EXTERNAL GENERIC-FUNCTION

    WALK-ATOM

    • ATOM
    • ENVIRONMENT

    Walks an atom.
    
    If the atom is a symbol, it returns a parse result of a literal for
    keywords and booleans, or a variable for symbols. For everything else
    it returns a parse result for a literal.

  • EXTERNAL GENERIC-FUNCTION

    WALK-FORM

    • OPERATOR-VAR2
    • CST
    • ENVIRONMENT

    Walks a form.
    
    The form is identified by the car of the cons. The entirety of the
    form as a CST, including the operator, are passed along as well.

  • EXTERNAL MACRO

    DEFINE-DEFINITION-RESOLVER

    • TYPE
    • SOURCE
    • &REST
    • ARGS
    • &BODY
    • BODY

    Source

    Shorthand to define a find-definitions method and destructure the arguments of the parse result.
    
    See FIND-DEFINITIONS

  • EXTERNAL MACRO

    DEFINE-SUB-RESULTS

    • TYPE
    • ARGS
    • &BODY
    • BODY

    Source

    Shorthand to define a sub-results method and destructure the arguments of the parse result.
    
    See SUB-RESULTS

  • EXTERNAL MACRO

    DEFINE-WALK-COMPOUND-FORM

    • OPERATOR
    • CST-VAR
    • &OPTIONAL
    • ENVIRONMENT-VAR
    • &BODY
    • BODY

    Source

    Shorthand to define a WALK-FORM method.
    
    Adds local functions for WALK and WALK-IMPLICIT-PROGN that
    automatically pass the environment along so you don't need to repeat
    it.
    
    See WALK-FORM

  • EXTERNAL MACRO

    DEFINE-WALKER-FORM

    • FORM
    • CST-VAR
    • &OPTIONAL
    • ENVIRONMENT-VAR
    • SOURCE
    • &BODY
    • BODY

    Source

    Shorthand to define simple walker forms.
    
    The FORM should be a destructuring description of the kind of form to
    walk. The return value of the BODY should be the list of additional
    arguments for the parse result. The type and source of the parse
    result are automatically added for you.
    
    If you need control over the type or source, look at
    DEFINE-WALK-COMPOUND-FORM instead.
    
    See DEFINE-WALK-COMPOUND-FORM