Index project with ctags and cscope with a single button press in emacs.

I should probably comment on the madness happening below, but I'm too tired at the moment, and, more importantly, I don't think anyone needs it anyway.

The short and messy description is that you press f9 and both ctags and cscope databases are built for your project. Pretty convenient.

To make it work with the project of your choice, you need to describe your project in the known-projects list (it's already filled with some examples).

The first part is a list of directories that designate the root of the project (so that you can run the command from any internal dir).

The second part is a blacklist that contains names and paths that should not be indexed, e.g. you would want to blacklist the huge test collection in GCC which will inflate your database tremendously and add an incredible amount of duplicated names, or the equally humongous Linux driver collection (unless you're a driver developer, duh).

1-2 auxiliary subroutines may have been copied from various places on the Internet. The code may hang your editor if you have cyclic links. Also, I can't program in elisp, although that must be strikingly obvious without me saying that.

;; Define the core dir trappings and blacklisted directories here
(setq known-projects '((("gcc" "config" "libgcc" "libiberty") ("/testsuite/" "/test/" "/build/"))
               (("mono" "mcs" "eglib" "docs" "tools") ())
               (("DROD" "DRODLib" "DRODUtil") ())
               (("arch" "drivers" "include" "kernel") ("/drivers/")) ; Linux
 
               (("README" "INSTALL") ("/.pc/")) ; Generic software project #1 (e.g. emacs)
               (("README") ("/.pc/")) ; Generic software project #2.
               ((".") ())          ; Oh well, let's at least index the current dir
               ))              ; if everything else fails.

(defun index-project () (interactive)
  (defun dump-vars-to-file (varlist filename)
    "simplistic dumping of variables in VARLIST to a file FILENAME"
    (save-excursion
      (let ((buf (find-file-noselect filename)))
    (set-buffer buf)
    (erase-buffer)
    (loop for var in varlist do
          (princ (concat var "\n") buf))
    (save-buffer)
    (kill-buffer))))
 
  (defun lookup-project (projects)
    (let ((lookup-failed t) (blacklist nil) (remaining-trappings nil) (lookup-path nil) (project nil))
      (while (and (consp projects) lookup-failed)
    (setq lookup-path buffer-file-name)
    (setq project (pop projects))
    ; (message "Trying project %s" project)
    (while (and (> (string-width lookup-path) 1) lookup-failed)
      ;; For some reason (file-exists-p "main.c/.") = t!
      (when lookup-failed
        (setq lookup-path (replace-regexp-in-string "/[^/]*$" "" lookup-path)))
      ; (message "Trying path %s" lookup-path)
      (setq lookup-failed nil)
      (setq remaining-trappings (nth 0 project))
      (setq blacklist (nth 1 project))
      (while (and (consp remaining-trappings) (not lookup-failed))
        (if (not (file-exists-p (concat lookup-path "/" (pop remaining-trappings))))
        (setq lookup-failed t)))))
      (when lookup-failed (setq lookup-path nil) (setq blacklist nil))
      (list lookup-path blacklist)))

  (let ((project-info (lookup-project known-projects))
    (project-root nil) (project-blacklist nil))

    (setq project-root (nth 0 project-info))
    (setq project-blacklist (nth 1 project-info))

    (message "Project found at %s. Scanning files... (excliding %s)" project-root project-blacklist)

    (when project-root
      (defun is-path-blacklisted (path blacklist)
    (while (and blacklist (not (string-match (car blacklist) path))) (pop blacklist))
    (if blacklist t nil))

      (defun build-cscope-database (root blacklist)
    (setq dirs (list root)) ;; "/usr/include/")) ; I found it to add too much clutter, better add some individual files instead.
    (setq files '())
    (while dirs
      (setq dir (pop dirs))
      (setq ls (directory-files dir))
      (while ls
        (setq file (pop ls))
        (setq file-path (concat dir "/" file))
        (unless (is-path-blacklisted file-path blacklist)
          (if (file-accessible-directory-p file-path)
          (unless (or (string= file ".") (string= file "..")) (add-to-list 'dirs file-path))
        (if (and (file-readable-p file-path) (string-match "\\.\\(c\\|cpp\\|h\\|hpp\\|vala\\|def\\)$" file-path))
            (add-to-list 'files file-path))))))
    ;; Append some commonly used header files. Uncomment the "/usr/include" line above
    ;; and remove/comment this one if you want to see them all their multitude instead
    ;; of this selectivity.
    (dolist (header '("stdio.h" "stdlib.h" "stdarg.h" "ctype.h" "string.h" "math.h" "dlfcn.h"
              "sys/types.h" "sys/time.h" "unistd.h" "assert.h" "limits.h" "poll.h"
              "libgen.h" "signal.h" "fcntl.h"
              "bits/stdio2.h" "bits/string2.h"
              "X11/Xos.h" "X11/Xlib.h" "X11/Xfuncproto.h" "X11/Xatom.h" "X11/Xproto.h"
              "X11/extensions/Xrandr.h" "X11/extensions/shape.h" "X11/cursorfont.h"
              "X11/extensions/XInput2.h" "X11/X.h"
              "boost/bind.hpp" "boost/foreach.hpp"
              "sys/wait.h" "sys/types.h" "sys/stat.h"
              "glib-2.0/glib/gspawn.h" "glib-2.0/glib/gslice.h" "glib-2.0/glib/gmacros.h"
              "glib-2.0/gobject/gobject.h" "glib-2.0/gobject/gtype.h"
              ))
      (add-to-list 'files (concat "/usr/include/" header)))
    files)

      (dump-vars-to-file (build-cscope-database project-root project-blacklist)
             (concat project-root "/" cscope-index-file))

      (message "Processing...")
      (cscope-set-initial-directory project-root)
      (with-temp-buffer (shell-command (concat "cd " project-root "; cscope -b; ctags -eL " cscope-index-file) t)) ; with-temp-buffer is needed to hide output
      (setq tags-file-name (concat project-root "/TAGS"))
      (message "Done processing")))
)

(global-set-key (kbd "<f9>") 'index-project)

No such file or directory after ld.

I was toying with the Linux development tools trying to figure out the whole compilation process of a program (something I should have done a loooong time ago) and ran into this interesting error. Or, rather, quite a boring one, but with a baffling manifestation for a permanent newbie like me.

What I wanted to do was to go through the whole source->compiler->assembler->linker->binary tool invocation chain manually instead of relying on GCC. I made a typical C program:

#include <stdio.h>

main()
{
printf ("Le ha-ha.\n");
}

Ran a typical compiler with the -S option to get a typical assembly source rather than a typical ready-to-go binary:

gcc -S hello.c -o hello.S

Assembled it into a typical ELF object file:

as hello.S -o hello.o

And, finally linked it with libc containing printf() and the crt* wrappers.

ld hello.o /usr/lib/crt* /usr/lib/libc.so -o hello

I say, that was quite simple! Let's run the bastard!

$ ./hello
bash: ./hello: No such file or directory

Huh? I guess there WAS an error, but the stupid tools didn't report it. Let's see which file is missing:

$ ls
hello hello.c hello.o hello.S

Err, what? The binary is present? What about the permissions?

$ ls -lh ./hello
-rwxr-xr-x 1 * * 4.3K 2011-04-03 03:43 hello

WTF?

$ file hello
hello: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not stripped

So the file actually EXISTS and it IS executale. However, when I run it, the system says it's absent.

Guess what? Linux actually doesn't find one file when I invoke my program, however, it is not my binary.

Let's dive into the details. If you run GCC with the -v option, it prints all the commands it executes. You can find the linking stage there too, although it's performed through a wrapper called collect2. My line was the following:

"/usr/lib/gcc/i486-linux-gnu/4.4.1/collect2" "--build-id" "--eh-frame-hdr" "-m" "elf_i386" "--hash-style=both" "-dynamic-linker" "/lib/ld-linux.so.2" "-z" "relro" "/usr/lib/gcc/i486-linux-gnu/4.4.1/../../../../lib/crt1.o" "/usr/lib/gcc/i486-linux-gnu/4.4.1/../../../../lib/crti.o" "/usr/lib/gcc/i486-linux-gnu/4.4.1/crtbegin.o" "-L/usr/lib/gcc/i486-linux-gnu/4.4.1" "-L/usr/lib/gcc/i486-linux-gnu/4.4.1" "-L/usr/lib/gcc/i486-linux-gnu/4.4.1/../../../../lib" "-L/lib/../lib" "-L/usr/lib/../lib" "-L/usr/lib/gcc/i486-linux-gnu/4.4.1/../../.." "-L/usr/lib/i486-linux-gnu" "/tmp/ccWw7lET.o" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/lib/gcc/i486-linux-gnu/4.4.1/crtend.o" "/usr/lib/gcc/i486-linux-gnu/4.4.1/../../../../lib/crtn.o"

After some trial and error I found out that the option I needed was this:

"-dynamic-linker" "/lib/ld-linux.so.2"

It specifies the name of the dynamic linker that will be used on program invocation. But how on earth was I supposed to know that the default dynamic linker wasn't good if the manpage for ld says:

The default dynamic linker is normally correct; don't use this unless you know what you are doing.

Liars. Let's find out what the default linker is:

$ ld hello.o /usr/lib/crt* /usr/lib/libc.so -o hello.without.explicit.dl
$ ./hello.without.explicit.dl
bash: ./hello.without.explicit.dl: No such file or directory
$ ld --dynamic-linker=/lib/ld-linux.so.2 hello.o /usr/lib/crt* /usr/lib/libc.so -o hello.with.explicit.dl
$ ./hello.with.explicit.dl
Le ha-ha.
$ objdump -s hello.without.explicit.dl > hello.without.explicit.dl.objdump
$ objdump -s hello.with.explicit.dl > hello.with.explicit.dl.objdump
$ diff -C1 hello.with.explicit.dl.objdump hello.without.explicit.dl.objdump
*** hello.with.explicit.dl.objdump 2011-04-03 04:14:11.000000000 +0400
--- hello.without.explicit.dl.objdump 2011-04-03 04:14:00.000000000 +0400
***************
*** 1,7 ****

! hello.with.explicit.dl: file format elf32-i386

Contents of section .interp:
! 8048114 2f6c6962 2f6c642d 6c696e75 782e736f /lib/ld-linux.so

! 8048124 2e3200 .2.

Contents of section .note.ABI-tag:

--- 1,7 ----

! hello.without.explicit.dl: file format elf32-i386

Contents of section .interp:

! 8048114 2f757372 2f6c6962 2f6c6962 632e736f /usr/lib/libc.so

! 8048124 2e3100 .1.

Contents of section .note.ABI-tag:

The only difference was the string value in the .interp section (which apparently specifies the path to the dynamic loader). And instead of /usr/lib/libc.so.1 what I needed was /lib/ld-linux.so.2. So... What does libc.so.1 look like?

$ ls /usr/lib/libc.so.1
ls: cannot access /usr/lib/libc.so.1: No such file or directory

There we have it. So the error we saw was the error about a missing dynamic loader, not the binary itself! But how was I supposed to know that from that message without stepping on this rake once? Beats me.

A query to Google shows that /usr/lib/libc.so.1 is used on SCO UnixWare systems, not Linux. Why ld doesn't put the proper linker name on an i386 Ubuntu system and, on top of that, confuses the user by saying not to touch the --dynamic-linker option is another question I can't answer.

Finally, it seems that the older Linux systems used to complain about a "bad ELF interpreter" which was kind of right. I wonder if the modern behaviour can be considered a bug.

Lesson learned? Even robust tools used for many years on a multitude of platforms may try to trick you. Especially robust tools used for many years on a multitude of platforms.