ImpLib SDK Guide

ImpLib SDK v1.8

ImpLib SDK is intended for authoring custom MS-COFF Import Libraries with advanced features, the following being the most relevant:

• Importing symbolic names by ordinal values. For example, there's a DLL function you want to call: MYDLL!dummy_function ORD:20. So, it's symbolic name is dummy_function while it's ordinal value is 20. Usually it's not a good idea to import a symbol by ordinal, since the ordinals may change between different versions of the same DLL. However, if you are sure the DLL version won't change or the developer of the DLL guarantees the backward compatibility of the ordinals (i.e. you are the developer of the DLL and you always keep the ordinals unchanged or you just distribute the same DLL version with your project), then you can benefit from using ordinals. Importing by ordinal significantly reduces the load time of the executable and makes the import section size smaller. Buy the way, if you plan to use always the same DLL build, consider binding your executables. Binding reduces the load time even more.
• Storing symbols from different DLL files in a single import library. For example, you can make a single import library containing the exported entries from KERNEL32, USER32, GDI32 and other common Windows API DLLs and name it something like win32api.lib. Then you can always link to that single file instead of having to specify kernel32.lib, user32.lib, gdi32.lib and so on.
• When interfacing non-STDCALL DLLs from Visual Basic 6, PureBasic and other popular development environments, sometimes the calling convention might not be supported. ImpLib is able to generate thunks converting CDECL and other calling conventions to STDCALL with very little overhead.
• ImpLib is able to generate libraries without original thunks. When linking to such a "stripped" library, the final executable won't have the thunks either. That means you can make your executable smaller when using "stripped" libraries. The only linker that is able to handle "stripped" libraries correctly is polink. MS linker (the one shipped with Visual Studio) is also able to do the job, but with a certain limitation. Sometimes, when mixing "stripped" and regular libraries, the import tables in the final executable get corrupted. So, you'd better use "stripped"-only or regular-only libraries in the same project, but not both, when linking with MS linker. GNU Linker (LD) appears to handle "stripped" libraries correctly, when not mixing with regular ones. Newer versions are more reliable. To make a "stripped" import library add the following command: KEEP_ORIGINAL_THUNKS equ 0 just after the include 'implib.inc'. A "stripped" import directory is not bindable, but otherwise perfectly valid in all Windows.

How does it work?

First, you have to write a script file, containing the list of the symbols (i.e. functions) exported by the given DLL. The syntax is very simple and similar to Microsoft's DEF file format. For example, let's make an import lib with some file I/O functions exported by KERNEL32:

include 'implib.inc'

   ; KERNEL32.DLL
   implib kernel32.dll, CreateFileA
   implib kernel32.dll, CreateFileW
   implib kernel32.dll, ReadFile
   implib kernel32.dll, SetFilePointer
   implib kernel32.dll, CloseHandle

endlib

Everything preceded by a ';' is a comment. Before starting to list the symbols, you have to place a valid reference to implib.inc. A relative or full path is requiered unless you place implib.inc in the same directory where the current script file is located. After completing the list, don't forget to place an endlib command. Every symbol definition starts with an implib command, followed by a number of arguments, the DLL file name and the symbolic name of the function being the most important ones. You may specify the ordinal value instead of the symbolic name, prefixing it with 'ord.':

   implib kernel32.dll, ord.5

If you want a symbol to be indexed by the linker with a different name, append another argument with the name you like:

   implib kernel32.dll, CreateFileA, CreateFile

This way, when referring to CreateFile in your object file, linker will actually resolve it as KERNEL32!CreateFileA. So, a trailing 'A' needs not being specified in the object file by default. This feature is useful mostly for mangling symbolic names. For example, MS tools assume that a STDCALL function CreateFileA with 7 arguments should be named as _CreateFileA@28. So, let's take this into account and rewrite our first example in a way compatible with MS naming conventions:

include 'sdk\implib.inc'

   ; KERNEL32.DLL
   implib kernel32.dll, CreateFileA,    _CreateFileA@28
   implib kernel32.dll, CreateFileW,    _CreateFileW@28
   implib kernel32.dll, ReadFile,       _ReadFile@20
   implib kernel32.dll, SetFilePointer, _SetFilePointer@16
   implib kernel32.dll, CloseHandle,    _CloseHandle@4

endlib

As you see, STDCALL names are all prefixed with a '_' and suffixed with a '@#', # being the number of arguments x4. Actually, # is the number of bytes required to hold the function arguments on the stack, but most of the time it equals to the number of arguments x4.

Save the script. Let's call it kernel32.def, but you can choose any other name. Now we are ready to compile it from the command line:

   bin\fasm kernel32.def kernel32.lib
   flat assembler  version 1.67.23  (533081 kilobytes memory)
   3 passes, 4247 bytes.

We've got a file named kernel32.lib! Buy the way, FASM is really an assembler, but it's preprocessor is so powerful we can use it to run ImpLib scripting engine :-)

There are more arguments to implib. Just open the implib.inc in a plain text editor and read the header for additional info.

DLL2DEF

Since big DLLs might contain tons of useful functions, listing them manually in a text file whould be a quite boring task. There is a tool DLL2DEF available in the \bin subdirectory which is useful for speeding up the process. Let's give it a try:

   dll2def c:\windows\system32\kernel32.dll

We've got a file named kernel32.def with all of the KERNEL32.DLL functions already listed! The implib.inc reference contains the full path name (only if implib.inc is located in the same directory as dll2def.exe), so that you can move the file to any place you want without having to update the implib.inc location. Every symbol definition starts with a couple of comments, i.e.:

   ; KERNEL32.AddVectoredExceptionHandler ORD:9
   ; -> NTDLL.RtlAddVectoredExceptionHandler
   implib kernel32.dll, AddVectoredExceptionHandler

The comments include the full symbolic name (if available), the ordinal value and the forwarder chain, if any. Knowing the ordinal is useful if you have good reasons to replace the symbol definition with it's ordinal. The forwarder chain means that the given symbol is actually located in another DLL. In the example above the symbol AddVectoredExceptionHandler exported by KERNEL32.DLL is actually forwarded to NTDLL. You can update the symbol definition to import the given symbol directly from the target DLL, but generally doing so is not recommended for compatibility reasons. If you don't like to see those comments in the script, just prepend a /COMPACT argument while calling DLL2DEF:

   sdk\dll2def /COMPACT c:\windows\system32\kernel32.dll

Mods

Currently ImpLib contains 2 mods:

• Visual Basic mod useful particularly for interfacing CDECL and other non-STDCALL DLLs in Visual Basic 6.
• PureBasic mod useful for making PureBasic Userlibs interfacing external DLLs, particularly useful with CDECL DLLs.

Replace implib command with vbimplib or pbimplib in the script file to use the Visual Basic or PureBasic mods respectively. While calling DLL2DEF prepend a /VB or /PB argument to make it use the given mod scheme instead of the general purpose implib. Check the tutorials below for a quick start.

Tutorials

Assuming you've already read the above intro, pick one of the following minitutorials for a real usage example:

• General purpose: MSVCRT.lib for MASM32 and NASM.
• PureBasic mod: OpenAL PureBasic Userlib.

MSVCRT.lib for MASM32 and NASM

Microsoft Visual C Run-Time library (MSVCRT.DLL) exports over 700 functions, automating many common programming tasks, like formatted console I/O, memory management, string manipulation, sorting and so on. There is nothing you can't achieve directly using the Windows API, but since the C Run-Time is almost always preinstalled in all Windows (except in Windows 95), sometimes it makes sense linking to CRT DLLs instead of hardcoding printf(), malloc(), etc. So, the goal of our first tutorial is making a couple of sample programs in MASM32 and NASM using MSVCRT.DLL.

Let's start with MASM32. You will probably find a header file for MSVCRT, named msvcrt.inc, in the latest MASM32 release, but there is no msvcrt.lib file required by the linker. If you have Visual Studio installed, you can get the dynamic version of msvcrt.lib from there, but there is one little naming conflict with functions div() and fabs(). You can't use div or fabs as the name of a function in either MASM32 or NASM because there are homonymous mnemonics in the x86 instruction set. The same conflict arises when using names, containing only numeric characters, or any of the words reserved by the compiler, like proc, label, etc. Import libraries in Visual Studio sometimes introduce additional dependencies from OLDNAMES.lib, UUID.lib and others. It seems, you don't have permission to redistribute a Visual Studio's import library and that's why MASM32 doesn't include msvcrt.lib.

So, we'll make our own MSVCRT.lib not copyrighted by any third party, not conflicting with MASM's or NASM's reserved words.

The first step is making the script file, containing all the public names found in MSVCRT.DLL:

   dll2def /COMPACT \windows\system32\msvcrt.dll

Next, we need to solve the naming conflict. So, open msvcrt.def in Notepad and search for the following line:

   implib msvcrt.dll, div

Since div is the unsigned integer division mnemonic, we need to rename this function to something else. Hutch suggested prefixing it with '_crt_' (there's a thread on this topic at masm32's official forum). So, here's the replacement:

   implib msvcrt.dll, div, _crt_div

Do the same to fabs function. It is recommended adding a leading '_' preceded thunk name to all function names not already starting with it, so that calling these functions whould be possible via invoke thunk from MASM32. For example:

   ; MSVCRT.printf ORD:742
   implib msvcrt.dll, printf

It is recommended to update it in the following way:

   ; MSVCRT.printf ORD:742
   implib msvcrt.dll, printf, _printf

The same definition using the ordinal value whould be:

   ; MSVCRT.printf ORD:742
   implib msvcrt.dll, ord.742, _printf, __imp__printf

Since the real funtion name is lost when using it's ordinal, it is recommended to specify the thunk name (_printf) and public import name (__imp__printf) explicitly this time. ImpLib won't complain if you don't specify them, but calling printf() will become a bit tricky ;-)

Procede compiling the library. Since ImpLib runs interpreted by the FASM's preprocessor and the script file contains over 700 functions, it might take some time to complete the task (about a couple of minutes on a fast machine):

   \sdk\fasm msvcrt.def msvcrt.lib
   flat assembler  version 1.67.23  (518320 kilobytes memory)
   3 passes, 35.4 seconds, 460585 bytes.

Now we have our own msvcrt.lib :-) Or a list of syntax error, duplicate symbols and other sad messages. Just correct the typos and compile again. A ready to use msvcrt.def and a precompiled msvcrt.lib are available in \src\test.

In order to call CRT functions in MASM32 using invoke, you need to specify the prototypes. There's an example of a header file with CRT prototypes in \src\test\msvcrt.inc. As an alternative, you can use PROTO instead of externdef syntax, if you wish to call the CRT through import thunks. For example:

   printf PROTO C :DWORD,:VARARG

Calling printf() using invoke with the above prototype will generate a call <jmp msvcrt!printf> instead of a direct call msvcrt!printf. In other words, it will use a call thunk. Since using thunks introduces performance penalties in most cases, it is recommended to prototype the extern functions in the ugly externdef manner:

   cdecl_dword_vararg typedef PROTO C :DWORD,:VARARG
   externdef _imp__printf:PTR cdecl_dword_vararg
   printf equ <_imp__printf>

This time invoking printf() will generate a direct call.

Here's a sample MASM32 code performing a call to MSVCRT!printf:

   .386
   .model flat,stdcall

   ; MSVCRT API
   include msvcrt.inc
   includelib msvcrt.lib

   .CODE
   format db "Hello, world!",0

   start:
      invoke printf,OFFSET format
      ret
   END start

Let's compile, link and run it from the command line:

   \masm32\bin\ml /c /coff test_masm32.asm
   \masm32\bin\link /SUBSYSTEM:CONSOLE test_masm32.obj
   test_masm32
   Hello, world!

The same example for NASM:

   EXTERN __imp__printf
   %define printf [__imp__printf]

   section .text
   format db "Hello, world!",0

   GLOBAL _start
   _start:
      push format
      call printf
      add esp,4 ; fix the stack
      ret

Compile, link and run:

   \nasm\nasmw -fwin32 test_nasm.asm
   \nasm\polink /ENTRY:start /SUBSYSTEM:CONSOLE test_nasm.obj msvcrt.lib
   test_nasm
   Hello, world!

MSVCRT.LIB tutorial ends here.

OpenAL PureBasic Userlib

Our intention is building a PureBasic Userlib interfacing an external DLL. The example DLL is OPENAL32.DLL from the OpenAL v1.0/v1.1 redistributable. Since it uses CDECL calling convention for all of it's API, we can't use PureBasic's DLL Importer tool. Obviously, we could upgrade to PureBasic v4 and use the new ImportC syntax and the OpenAL32.lib file found in the OpenAL SDK, or just use OpenLibrary and then CallCFunctionFast every time we need to invoke an OpenAL function. Even more complex workarounds might exist, like using a wrapper DLL and so on. In this tutorial we'll make a PureBasic Userlib, which will give us direct access to OpenAL, with a very small overhead. The caller thunk will be faster compared to CallCFunctionFast, with prototype checking, inline help and any other advantages a Userlib is able to offer. It will work in PureBasic v3 as well.

First of all, we need to create the import description script:

   dll2def /PB /COMPACT \windows\system32\OpenAL32.dll

This will create a file named openal32.def, listing all the public symbols found in OpenAL32.dll. For example:

   pbimplib OpenAL32.dll, STDCALL, 0, alBuffer3f

The complete pbimplib command syntax is as follows:

   pbimplib dllname, convention, numarguments, real_name, PureBasics_name, public_name

The dllname parameter identifies the DLL's filename. The convention parameter identifies the calling convention used with the specified function. The following values are currently supported:

• STDCALL used by default in most Windows DLLs: KERNEL32.DLL, USER32.DLL, ...
• CDECL used mostly in C DLLs: MSVCRT.DLL, OPENAL32.DLL, ...
• PBCALL is PureBasic's calling convention, not actually supposed to be used in a real world's DLL. It's provided just in case and to make an example of extending the PBIMPLIB functionality. Somebody might consider adding more exotic calling conventions, like PASCAL, FASTCALL, etc.

numarguments tells the amount of doubleword (32-bit) values used in the stack frame for argument storage. It generally matches the amount of arguments, except when dealing with 64-bit parameters (double, __int64 and so on). This value is not really used in STDCALL. real_name is the name actually defined in the DLL's export table. The last 2 parameters are optional. PureBasics_name allows customizing the name of the function in PureBasic. By default, when not specified, real_name is used. public_name is reserved for very specific issues, generally involving low-level assembly programming. It lets you specify the name of the symbol, used to call a function directly, without using any thunks. You don't need to care about it most of the time. Take a look at the generated script file once again:

   pbimplib OpenAL32.dll, STDCALL, 0, alBuffer3f

alBuffer3f() is not a STDCALL function and it doesn't receive 0 arguments. DLL2DEF just can't guess the amount of arguments and the calling convention because it can't read the "OpenAL Programmer's Guide" :-) Using STDCALL, 0 just disables automatic stack fixing after calling the given function via thunk. The correct prototype should be:

   pbimplib OpenAL32.dll, CDECL, 5, alBuffer3f

You can update all the other definitions or just use the already fixed script src\PureBasic\openal32.def. Now, let's compile it:

   fasm openal32.def pbopenal.lib

No errors? - Fine. Now you need to create a library descriptor and name it exactly as the newly obtained import lib with a .desc extension. So, in the current example this file should be named pbopenal.desc. Check the PureBasic\Library SDK\Readme.txt file if not familiar with LibraryMaker.

   ; Langage used to code the library: ASM or C. You need to select C here.
   C

   ; Number of windows DLL than the library need: none in the example.
   0

   ; Library type (Can be OBJ or LIB). Select LIB, since pbopenal.lib is a LIB.
   LIB

The rest of the file depends on the actual library content. There is an example: src\PureBasic\pbopenal.desc. Finally, use LibraryMaker.exe to create the Userlib:

   LibraryMaker.exe pbopenal.desc /COMPRESSED

Just move the Userlib into PureBasic\PureLibraries\UserLibraries and try compiling a simple test application. You can find a PureBasic OpenAL example in uFMOD.

That's all. Hope you've found useful this minitutorial.
 

2006-2008 Quantum.