Tokenlist installation

First you have to copy tokenlist.sl somewhere on your jed_library_path. Optionally you can also copy tkl-modes.sl if you will use the default settings for some modes (SLang, C, python, PHP, HTML, LaTeX). Insert the content of the INITIALIZATION block (from tokenlist.sl) or just

  require("tokenlist");

into your jed.rc (or .jedrc) file (or use the "make_ini" and "home-lib" modes from jedmodes.sf.net). Optionally add some keybindings, e.g.:

  setkey("list_routines", "^R");
  setkey("occur", "^O");

The function list_routines is very simple in its default implementation and is conditionally useful for C and SLang mode. To extend its capabilities you have to add some macros to search for routines and extract them into the list-buffer. Extensions are provided for more modes in the file tkl-modes.sl which can be loaded in tokenlist_hook() which you define in .jedrc (jed.rc):

  define tokenlist_hook()
  {
     if (expand_jedlib_file("tkl-modes"))
       require("tkl-modes");
  }

If you use require("tokenlist") to load tokenlist.sl then you should put the definition of tokenlist_hook() before require("tokenlist").

You can add more definitions to tkl-modes.sl but it is better to write a separate file like mytklist.sl and change tokenlist_hook() to:

  define tokenlist_hook()
  {
     if (expand_jedlib_file("tkl-modes"))
       require("tkl-modes");
     if (expand_jedlib_file("mytklist"))
       require("mytklist");
  }

Defining your own routine-search

You can write the macros for routine-search directly into .jedrc but it is better to add them to a separate file so they will be loaded only when needed. Here is a minimal example for mytklist.sl:

% ----------- mytklist.sl ------
provide("mytklist");

Let us now define a special X-TEXT mode where the titles on the first level start with '::', on the second with '==' an and on the third level with '--'. In the result list the titles should be indented depending on the level of the title.

For the new mode to work we have to define the function x_text_list_routines_setup(opt) where opt is a structure that defines the behaviour of list_routines. It has the following fields:

Field Description Default

mode The name of the mode.

list_regex A list of regular expressions to search for. list_routines will take every item from the list and search for all the occurences in the active buffer that match the item (regular expression). After an expression is found, some text is extracted from the buffer with a call to fn_extract.
An item from the list can be a function reference instead of a regular expression. In this case the function will be called by list_routines until it returns 0. The function takes an integer parameter which is the current index and looks like this:
% Int_Type searc_fn(Int_Type array_index) define searc_fn(idx) { return fsearch("something"); }
{"^[a-zA-Z].*("};

fn_extract A reference to a (private) function that extracts some text at the current point in the active buffer and returns it as a string. It is called after every match that list_routines finds. The function takes an integer parameter which is the current index and looks like this:
% String extract_fn(Int_Type array_index) define extract_fn(idx) { return line_as_string(); }
&_list_routines_extract
The function is defined in tokenlist and returns the current line (line_as_string).

onlistcreated A reference to a (private) function that is called after all the items from list_regex have been processed. The active buffer is the list of matches and it is narrowed to the matches found in the last buffer.
You can use this hook to sort the matched lines or do any other processing of the buffer. There are two helper functions defined in tokenlist.sl to sort the lines by line number (tkl_sort_by_line) or by content (tkl_sort_by_value).
NULL

Field	Description	Default
`mode`	The name of the mode.
`list_regex`	A list of regular expressions to search for. `list_routines` will take every item from the list and search for all the occurences in the active buffer that match the item (regular expression). After an expression is found, some text is extracted from the buffer with a call to `fn_extract`. An item from the list can be a function reference instead of a regular expression. In this case the function will be called by `list_routines` until it returns 0. The function takes an integer parameter which is the current index and looks like this: % Int_Type searc_fn(Int_Type array_index) define searc_fn(idx) { return fsearch("something"); }	`{"^[a-zA-Z].*("};`
`fn_extract`	A reference to a (private) function that extracts some text at the current point in the active buffer and returns it as a string. It is called after every match that `list_routines` finds. The function takes an integer parameter which is the current index and looks like this: % String extract_fn(Int_Type array_index) define extract_fn(idx) { return line_as_string(); }	`&_list_routines_extract` The function is defined in tokenlist and returns the current line (line_as_string).
`onlistcreated`	A reference to a (private) function that is called after all the items from `list_regex` have been processed. The active buffer is the list of matches and it is narrowed to the matches found in the last buffer. You can use this hook to sort the matched lines or do any other processing of the buffer. There are two helper functions defined in `tokenlist.sl` to sort the lines by line number (`tkl_sort_by_line`) or by content (`tkl_sort_by_value`).	`NULL`

Here is the code for the "X-TEXT" mode:

% ----------- mytklist.sl ------
autoload("tkl_sort_by_value", "tokenlist");
autoload("tkl_sort_by_line", "tokenlist");
provide("mytklist");

private define x_text_list_routines_extract(n)
{
  variable chars  = [":", "=", "-"];
  variable prefix = ["", "..", "...."];
  if (n < 3) {
    bol(); 
    skip_chars(chars[n]); 
    push_mark();
    eol();
    return (prefix[n] + strtrim(bufsubstr()));
  }
  return "";
}

define x_text_list_routines_setup(opt)
{
  opt.list_regexp = {
    '^::',
    '^==',
    '^--'
  };
  opt.fn_extract = &x_text_list_routines_extract;
  opt.onlistcreated = &tkl_sort_by_line;
}

If have a buffer TestBufer.xtxt with the content like

::Heading 1
...
==Heading 2
...
--Heading 3
..
==Heading 4

list_routines would generate the following list:

Buffer: TestBufer.xtxt
     3 : Heading 1
    27 : ..Heading 2
    33 : ....Heading 3
    43 : ..Heading 4

You can see that the list is sorted by line number. This is because we set opt.onlistcreated to &tkl_sort_by_line. If we set it to NULL we get the following list:

Buffer: TestBufer.xtxt
     3 : Heading 1
    27 : ..Heading 2
    43 : ..Heading 4
    33 : ....Heading 3

If we wanted to sort the titles by value, we could use tkl_sort_by_value. Unfortunately it would not sort the titles alphabetically because the titles in the buffer are indented.

Mode writers are encouraged to add _list_routines_setup definitions to their modes.

Extending existing modes

Suppose you would like to list all the special blocks like #<INITIALIZATION> in SLang mode. You could change the function slang_list_routines_setup in tkl-modes.sl or copy the contents from tkl-modes.sl to mytklist.sl and change it there. This would be hard to maintain.

That is why another hook was added to tokenlist:

  Void  tokenlist_routine_setup_hook(opt)

This hook is not defiend by default. You can add it to a file like mytklist.sl. In our example we add a new regular expression and modify the extraction function. We store some values from the structure opt for later use.

private variable slang_nregex = NULL;
private variable slang_fn_extract = NULL;
private variable slang_onlistcreated = NULL;

private define my_slang_fn_extract(idx)
{
   if (idx < slang_nregex && slang_fn_extract != NULL)
      return (@slang_fn_extract)(idx);
   else
      return strtrim(line_as_string());
}
  
define tokenlist_routine_setup_hook(opt)
{
   if (opt.mode == "slang")
   {
      slang_nregex = length(opt.list_regex);
      slang_fn_extract = opt.fn_extract;
      slang_onlistcreated = opt.onlistcreated;
      list_append(opt.list_regex, "^#\\<[a-zA-Z]");
      opt.fn_extract = &my_slang_fn_extract;
   }
}

my_slang_fn_extract will use the original extraction function for the original regular expressions and extract the current line for the new regular expression.

Imagine that you woud like to sometimes list only functions and at other times only variables. You would define a global variable and prepare the members of the parameter opt according to the value of the variable:

% 0 - all; 1 - variables; 2 - macros
variable slang_routine_mode = 0;

private define my_slang_extract_variable(n)
{
  return(strtrim(line_as_string()));
}

private define my_slang_extract_macro(n)
{
  return(strtrim(line_as_string()));
}

define tokenlist_routine_setup_hook(opt)
{
   if (opt.mode == "slang")
   {
      if (slang_routine_mode == 1)
      {
         opt.list_regex = { "^variable[ \t]" };
         opt.fn_extract = &my_slang_extract_variable;
      }
      if (slang_routine_mode == 2)
      {
         opt.list_regex = { "^define[ \t]" };
         opt.fn_extract = &my_slang_extract_macro;
      }
   }
}

Future

The structure opt could have more members so tokenlist could be made extremely configurable:

what to do when user presses Enter (replacing tkl_goto_token)
what to do when user presses Space (replacing tkl_display_token)
what other keybindings do we want in the list
what to do when the current line is changed in the list (extending the tkl_update_token_hook)
...