Questions on building C libraies using ATS

Google Group
1

I moved the discussion under a more informative heading.On Saturday, December 13, 2014 12:12:13 AM UTC-5, gmhwxi wrote:

By Barry Schwartz:

On Saturday, December 13, 2014 12:10:17 AM UTC-5, gmhwxi wrote:

john skaller ska...@users.sourceforge.net skribis:

But its WORSE. In both C and C++ you ALSO have to say something like:

#if !defined(FLX_STATIC_LINK) && FLX_WIN32
#define FLX_EXPORT __declspec(dllexport)
#define FLX_IMPORT __declspec(dllimport)
#else
#define FLX_EXPORT
#define FLX_IMPORT
#endif

#ifdef BUILD_RTL
#define RTL_EXTERN FLX_EXPORT
#else
#define RTL_EXTERN FLX_IMPORT
#endif

etc.

I asked about this a few weeks ago. Yes, it can be done. It is not
done by using =“ext:” but by a more convoluted but otherwise better
method; I am not the one to discuss it, however.

On Saturday, December 13, 2014 12:08:42 AM UTC-5, gmhwxi wrote:

By John Skaller:

On 13/12/2014, at 2:59 PM, Brandon Barker wrote:

In the wiki article there is this, which I think is what you are
referring to:

Make an ATS function available in C (can be implemented in ATS or C):

extern
fun myfun (n: int, res: int): int = “ext#myfun_in_c” (* call as
myfun_in_c(int, int) in C *)

(* Have the C function name be the same as the ATS function name: )
extern
fun myfun (n: int, res: int): int = “ext#” ( call as myfun(int, int)
in C *)

That’s not enough in C++. You have to say

// in header file:
extern “C” void name (int);

// in body code:
void name ( int x) { … }

But its WORSE. In both C and C++ you ALSO have to say something like:

#if !defined(FLX_STATIC_LINK) && FLX_WIN32
#define FLX_EXPORT __declspec(dllexport)
#define FLX_IMPORT __declspec(dllimport)
#else
#define FLX_EXPORT
#define FLX_IMPORT
#endif

#ifdef BUILD_RTL
#define RTL_EXTERN FLX_EXPORT
#else
#define RTL_EXTERN FLX_IMPORT
#endif

Here “BUILD_RTL” has to be supplied as a C/C++ compiler switch when
building
the DLL and not otherwise. FLX_STATIC_LINK has to be supplied if
compiling
for static linkage (and not otherwise).

Now the functions have to be declared like

RTL_EXTERN void flx_trace(flx_trace_t* tr,flx_range_srcref_t sr, char
const *file, int line, char const *msg);

in C++ for a C++ function and

extern “C”

as well if you want to use C liker names. This is usually done like:

#ifdef __cplusplus /* support use by C++ code */
extern “C” {
#endif

RTL_EXTERN void flx_trace(flx_trace_t* tr,flx_range_srcref_t sr, char
const *file, int line, char const *msg);

#ifdef __cplusplus
}
#endif

It used to be only Windows required dllimport/dllexport but this is NOT
so any more.
Now GNU linkers (finally) have visibility control, you have to do it
with all modern gcc
systems (i.e. all Unix, OSX and Linux) platforms too (unless you want to
export everything!)

So clearly this is all non-trivial and it HAS to be supported by ATS
compiler.
I’m pretty sure ATS compiler cannot do the DLL import/export at present.
In particular

(a) the ATS to C compiler has to generate the appropriate code AND

(b) the C/C++ compiler/linker toolchain invoked by ATS driver
program has to provide the required STATIC_LINK and BUILD_LIB switches
as well.

I don’t care about (b), I can organise C/C++ compile/link myself.

All this is in addition to ensuring the C code works under C++ compiler
as well.
(no implicit casts from void * allowed, no assuming enums are ints, no
funny business
with bool either … )

Even if you compile the library with C, the header files MUST work in
C++
as well as C.

getting all this right is not hard but the method is NOT obvious: more
than half the
open source projects in existence get it wrong.

The macro method for building DLLs above is the ONLY correct way to do
it.

In turn this means when you’re translating a library from ATS to C you
MUST
supply the build macro name to the ATS translator. Which I don’t think
ATS
supports at present either.

So … I may be wrong … but I do not think ATS can generate libraries
yet.

Just as an aside, although messy this is the correct protocol:

    export sym; // in provider 
    import sym; // in consumer

C, as usual, gets everything wrong and tries to just use

    extern sym;

for both purposes and that simply does not work. [In particular
the statement

    extern int x;

has unspecified meaning: you can’t tell if it’s a declaration or
definition,
so C’s hackery doesn’t even work in C]

2
  1. Can ATS be used to produce a stand alone C library?

This means: I will specify the functions which must be provided.
They will have extern “C” linkage.

I am thinking of Judy. Judy provides a cache tuned digital trie.
It is the fasted mutable associative store in existence which
provides both random and sequential access. There are 3 primary
data types: word → bit, word → word, and NTBS → word.
[NTBS: null terminated string]

It also scales to terrabyte store without any issues.
All core operations on Judy are O(1) (O(N) for NTBS where
N is the string length).

Judy is slower at some sizes for random access compared
to hash tables (but hash tables can’t do sequential access).
OTOH Judy only handles word keys (hash is more general,
it handles anything with a comparison and hash function).

However the code is horrible, full of macros, and a lot of work would be
required to rewrite it in ATS. We’d use templates to replace the macros.
And we’d want to use the type system to prove correctness.

This would be an interesting job to do because Judy is the best.
And it isn’t used much for some reason (perhaps because the code
is unmaintainable :slight_smile:

Judy does have one downside in a modern context: it is not compatible
with a plugin garbage collector because it is a digital trie (keys are
split into non-contiguous byte streams which means some pointers
to objects will be invisible to a collector not trained in Judy).
My own GC knows about Judy though :slight_smile:

I took a quick look at the Judy repository at sourceforge. This is not for
faint-hearted :slight_smile:

Are there any known problems with this implementation of Judy?

It would be a tremendous task to implement it in ATS even if you use
casts to bypass typechecking. I do see a lot of ifdef’s in the
implementation.
So using templates may give you more freedom to reorganize the code.On Saturday, December 13, 2014 12:14:51 AM UTC-5, gmhwxi wrote:

I moved the discussion under a more informative heading.

On Saturday, December 13, 2014 12:12:13 AM UTC-5, gmhwxi wrote:

By Barry Schwartz:

On Saturday, December 13, 2014 12:10:17 AM UTC-5, gmhwxi wrote:

john skaller ska...@users.sourceforge.net skribis:

But its WORSE. In both C and C++ you ALSO have to say something like:

#if !defined(FLX_STATIC_LINK) && FLX_WIN32
#define FLX_EXPORT __declspec(dllexport)
#define FLX_IMPORT __declspec(dllimport)
#else
#define FLX_EXPORT
#define FLX_IMPORT
#endif

#ifdef BUILD_RTL
#define RTL_EXTERN FLX_EXPORT
#else
#define RTL_EXTERN FLX_IMPORT
#endif

etc.

I asked about this a few weeks ago. Yes, it can be done. It is not
done by using =“ext:” but by a more convoluted but otherwise better
method; I am not the one to discuss it, however.

On Saturday, December 13, 2014 12:08:42 AM UTC-5, gmhwxi wrote:

By John Skaller:

On 13/12/2014, at 2:59 PM, Brandon Barker wrote:

In the wiki article there is this, which I think is what you are
referring to:

Make an ATS function available in C (can be implemented in ATS or C):

extern
fun myfun (n: int, res: int): int = “ext#myfun_in_c” (* call as
myfun_in_c(int, int) in C *)

(* Have the C function name be the same as the ATS function name: )
extern
fun myfun (n: int, res: int): int = “ext#” ( call as myfun(int,
int) in C *)

That’s not enough in C++. You have to say

// in header file:
extern “C” void name (int);

// in body code:
void name ( int x) { … }

But its WORSE. In both C and C++ you ALSO have to say something like:

#if !defined(FLX_STATIC_LINK) && FLX_WIN32
#define FLX_EXPORT __declspec(dllexport)
#define FLX_IMPORT __declspec(dllimport)
#else
#define FLX_EXPORT
#define FLX_IMPORT
#endif

#ifdef BUILD_RTL
#define RTL_EXTERN FLX_EXPORT
#else
#define RTL_EXTERN FLX_IMPORT
#endif

Here “BUILD_RTL” has to be supplied as a C/C++ compiler switch when
building
the DLL and not otherwise. FLX_STATIC_LINK has to be supplied if
compiling
for static linkage (and not otherwise).

Now the functions have to be declared like

RTL_EXTERN void flx_trace(flx_trace_t* tr,flx_range_srcref_t sr, char
const *file, int line, char const *msg);

in C++ for a C++ function and

extern “C”

as well if you want to use C liker names. This is usually done like:

#ifdef __cplusplus /* support use by C++ code */
extern “C” {
#endif

RTL_EXTERN void flx_trace(flx_trace_t* tr,flx_range_srcref_t sr, char
const *file, int line, char const *msg);

#ifdef __cplusplus
}
#endif

It used to be only Windows required dllimport/dllexport but this is NOT
so any more.
Now GNU linkers (finally) have visibility control, you have to do it
with all modern gcc
systems (i.e. all Unix, OSX and Linux) platforms too (unless you want
to export everything!)

So clearly this is all non-trivial and it HAS to be supported by ATS
compiler.
I’m pretty sure ATS compiler cannot do the DLL import/export at
present.
In particular

(a) the ATS to C compiler has to generate the appropriate code AND

(b) the C/C++ compiler/linker toolchain invoked by ATS driver
program has to provide the required STATIC_LINK and BUILD_LIB switches
as well.

I don’t care about (b), I can organise C/C++ compile/link myself.

All this is in addition to ensuring the C code works under C++ compiler
as well.
(no implicit casts from void * allowed, no assuming enums are ints, no
funny business
with bool either … )

Even if you compile the library with C, the header files MUST work in
C++
as well as C.

getting all this right is not hard but the method is NOT obvious: more
than half the
open source projects in existence get it wrong.

The macro method for building DLLs above is the ONLY correct way to do
it.

In turn this means when you’re translating a library from ATS to C you
MUST
supply the build macro name to the ATS translator. Which I don’t think
ATS
supports at present either.

So … I may be wrong … but I do not think ATS can generate libraries
yet.

Just as an aside, although messy this is the correct protocol:

    export sym; // in provider 
    import sym; // in consumer

C, as usual, gets everything wrong and tries to just use

    extern sym;

for both purposes and that simply does not work. [In particular
the statement

    extern int x;

has unspecified meaning: you can’t tell if it’s a declaration or
definition,
so C’s hackery doesn’t even work in C]

3
  1. I have another library which I would think is suitable for
    conversion to ATS: a precise garbage collector. Which uses
    Judy mentioned above. However it is currently a C++ library.
    The virtual dispatch is actually used as follows: the allocator
    used is abstract. The GC itself is split into an abstraction and
    implementation, the core implementation is not thread safe.
    A wrapper implementation is provides as well which is thread safe.

To convert to C is possible, just using a record of function pointers.

The GC is not entirely trivial. It’s a precise naive mark/sweep collector
driven by compiler generated tables. The main sweep mechanism
uses an array of offsets to find pointers. Judy arrays provide repetition
counts to allow variable length arrays to be handled.

When sweeping, the recursive descent posts pointers into a buffer
for later analysis until a fixed recursion depth is reached. Then the
scan falls out and the analysis routine runs through the buffer.
This mechanism is complex, a proof of correctness would be really
valuable.

This GC could also be used with ATS is the ATS compiler could be taught
how to generate the RTTI tables required to run the GC.

Unlike Judy, which it relies upon, the collector is not a very large
piece of code. The longest function is about one page.

I have been wanting to get a GC that can run along the side of another GC
(e.g., Java GC). This can be a big help if one wants to mix ATS and Java.
Using linear types is unlikely to be cheap.On Saturday, December 13, 2014 12:14:51 AM UTC-5, gmhwxi wrote:

I moved the discussion under a more informative heading.

On Saturday, December 13, 2014 12:12:13 AM UTC-5, gmhwxi wrote:

By Barry Schwartz:

On Saturday, December 13, 2014 12:10:17 AM UTC-5, gmhwxi wrote:

john skaller ska...@users.sourceforge.net skribis:

But its WORSE. In both C and C++ you ALSO have to say something like:

#if !defined(FLX_STATIC_LINK) && FLX_WIN32
#define FLX_EXPORT __declspec(dllexport)
#define FLX_IMPORT __declspec(dllimport)
#else
#define FLX_EXPORT
#define FLX_IMPORT
#endif

#ifdef BUILD_RTL
#define RTL_EXTERN FLX_EXPORT
#else
#define RTL_EXTERN FLX_IMPORT
#endif

etc.

I asked about this a few weeks ago. Yes, it can be done. It is not
done by using =“ext:” but by a more convoluted but otherwise better
method; I am not the one to discuss it, however.

On Saturday, December 13, 2014 12:08:42 AM UTC-5, gmhwxi wrote:

By John Skaller:

On 13/12/2014, at 2:59 PM, Brandon Barker wrote:

In the wiki article there is this, which I think is what you are
referring to:

Make an ATS function available in C (can be implemented in ATS or C):

extern
fun myfun (n: int, res: int): int = “ext#myfun_in_c” (* call as
myfun_in_c(int, int) in C *)

(* Have the C function name be the same as the ATS function name: )
extern
fun myfun (n: int, res: int): int = “ext#” ( call as myfun(int,
int) in C *)

That’s not enough in C++. You have to say

// in header file:
extern “C” void name (int);

// in body code:
void name ( int x) { … }

But its WORSE. In both C and C++ you ALSO have to say something like:

#if !defined(FLX_STATIC_LINK) && FLX_WIN32
#define FLX_EXPORT __declspec(dllexport)
#define FLX_IMPORT __declspec(dllimport)
#else
#define FLX_EXPORT
#define FLX_IMPORT
#endif

#ifdef BUILD_RTL
#define RTL_EXTERN FLX_EXPORT
#else
#define RTL_EXTERN FLX_IMPORT
#endif

Here “BUILD_RTL” has to be supplied as a C/C++ compiler switch when
building
the DLL and not otherwise. FLX_STATIC_LINK has to be supplied if
compiling
for static linkage (and not otherwise).

Now the functions have to be declared like

RTL_EXTERN void flx_trace(flx_trace_t* tr,flx_range_srcref_t sr, char
const *file, int line, char const *msg);

in C++ for a C++ function and

extern “C”

as well if you want to use C liker names. This is usually done like:

#ifdef __cplusplus /* support use by C++ code */
extern “C” {
#endif

RTL_EXTERN void flx_trace(flx_trace_t* tr,flx_range_srcref_t sr, char
const *file, int line, char const *msg);

#ifdef __cplusplus
}
#endif

It used to be only Windows required dllimport/dllexport but this is NOT
so any more.
Now GNU linkers (finally) have visibility control, you have to do it
with all modern gcc
systems (i.e. all Unix, OSX and Linux) platforms too (unless you want
to export everything!)

So clearly this is all non-trivial and it HAS to be supported by ATS
compiler.
I’m pretty sure ATS compiler cannot do the DLL import/export at
present.
In particular

(a) the ATS to C compiler has to generate the appropriate code AND

(b) the C/C++ compiler/linker toolchain invoked by ATS driver
program has to provide the required STATIC_LINK and BUILD_LIB switches
as well.

I don’t care about (b), I can organise C/C++ compile/link myself.

All this is in addition to ensuring the C code works under C++ compiler
as well.
(no implicit casts from void * allowed, no assuming enums are ints, no
funny business
with bool either … )

Even if you compile the library with C, the header files MUST work in
C++
as well as C.

getting all this right is not hard but the method is NOT obvious: more
than half the
open source projects in existence get it wrong.

The macro method for building DLLs above is the ONLY correct way to do
it.

In turn this means when you’re translating a library from ATS to C you
MUST
supply the build macro name to the ATS translator. Which I don’t think
ATS
supports at present either.

So … I may be wrong … but I do not think ATS can generate libraries
yet.

Just as an aside, although messy this is the correct protocol:

    export sym; // in provider 
    import sym; // in consumer

C, as usual, gets everything wrong and tries to just use

    extern sym;

for both purposes and that simply does not work. [In particular
the statement

    extern int x;

has unspecified meaning: you can’t tell if it’s a declaration or
definition,
so C’s hackery doesn’t even work in C]

4

Thanks! These are all very good questions.

Right now, you can use ATS code to generate .a and .so files. However,
the .h files for these object files need to be hand-coded. Once it is clear
how this process goes, then we can implement a tool (in whatever language)
to do this by operating on level-2 JSONized syntax.

However, template-based functions in ATS are somewhat like C-macros.
They are not meant to be compiled into object code directly.On Saturday, December 13, 2014 12:14:51 AM UTC-5, gmhwxi wrote:

I moved the discussion under a more informative heading.

On Saturday, December 13, 2014 12:12:13 AM UTC-5, gmhwxi wrote:

By Barry Schwartz:

On Saturday, December 13, 2014 12:10:17 AM UTC-5, gmhwxi wrote:

john skaller ska...@users.sourceforge.net skribis:

But its WORSE. In both C and C++ you ALSO have to say something like:

#if !defined(FLX_STATIC_LINK) && FLX_WIN32
#define FLX_EXPORT __declspec(dllexport)
#define FLX_IMPORT __declspec(dllimport)
#else
#define FLX_EXPORT
#define FLX_IMPORT
#endif

#ifdef BUILD_RTL
#define RTL_EXTERN FLX_EXPORT
#else
#define RTL_EXTERN FLX_IMPORT
#endif

etc.

I asked about this a few weeks ago. Yes, it can be done. It is not
done by using =“ext:” but by a more convoluted but otherwise better
method; I am not the one to discuss it, however.

On Saturday, December 13, 2014 12:08:42 AM UTC-5, gmhwxi wrote:

By John Skaller:

On 13/12/2014, at 2:59 PM, Brandon Barker wrote:

In the wiki article there is this, which I think is what you are
referring to:

Make an ATS function available in C (can be implemented in ATS or C):

extern
fun myfun (n: int, res: int): int = “ext#myfun_in_c” (* call as
myfun_in_c(int, int) in C *)

(* Have the C function name be the same as the ATS function name: )
extern
fun myfun (n: int, res: int): int = “ext#” ( call as myfun(int,
int) in C *)

That’s not enough in C++. You have to say

// in header file:
extern “C” void name (int);

// in body code:
void name ( int x) { … }

But its WORSE. In both C and C++ you ALSO have to say something like:

#if !defined(FLX_STATIC_LINK) && FLX_WIN32
#define FLX_EXPORT __declspec(dllexport)
#define FLX_IMPORT __declspec(dllimport)
#else
#define FLX_EXPORT
#define FLX_IMPORT
#endif

#ifdef BUILD_RTL
#define RTL_EXTERN FLX_EXPORT
#else
#define RTL_EXTERN FLX_IMPORT
#endif

Here “BUILD_RTL” has to be supplied as a C/C++ compiler switch when
building
the DLL and not otherwise. FLX_STATIC_LINK has to be supplied if
compiling
for static linkage (and not otherwise).

Now the functions have to be declared like

RTL_EXTERN void flx_trace(flx_trace_t* tr,flx_range_srcref_t sr, char
const *file, int line, char const *msg);

in C++ for a C++ function and

extern “C”

as well if you want to use C liker names. This is usually done like:

#ifdef __cplusplus /* support use by C++ code */
extern “C” {
#endif

RTL_EXTERN void flx_trace(flx_trace_t* tr,flx_range_srcref_t sr, char
const *file, int line, char const *msg);

#ifdef __cplusplus
}
#endif

It used to be only Windows required dllimport/dllexport but this is NOT
so any more.
Now GNU linkers (finally) have visibility control, you have to do it
with all modern gcc
systems (i.e. all Unix, OSX and Linux) platforms too (unless you want
to export everything!)

So clearly this is all non-trivial and it HAS to be supported by ATS
compiler.
I’m pretty sure ATS compiler cannot do the DLL import/export at
present.
In particular

(a) the ATS to C compiler has to generate the appropriate code AND

(b) the C/C++ compiler/linker toolchain invoked by ATS driver
program has to provide the required STATIC_LINK and BUILD_LIB switches
as well.

I don’t care about (b), I can organise C/C++ compile/link myself.

All this is in addition to ensuring the C code works under C++ compiler
as well.
(no implicit casts from void * allowed, no assuming enums are ints, no
funny business
with bool either … )

Even if you compile the library with C, the header files MUST work in
C++
as well as C.

getting all this right is not hard but the method is NOT obvious: more
than half the
open source projects in existence get it wrong.

The macro method for building DLLs above is the ONLY correct way to do
it.

In turn this means when you’re translating a library from ATS to C you
MUST
supply the build macro name to the ATS translator. Which I don’t think
ATS
supports at present either.

So … I may be wrong … but I do not think ATS can generate libraries
yet.

Just as an aside, although messy this is the correct protocol:

    export sym; // in provider 
    import sym; // in consumer

C, as usual, gets everything wrong and tries to just use

    extern sym;

for both purposes and that simply does not work. [In particular
the statement

    extern int x;

has unspecified meaning: you can’t tell if it’s a declaration or
definition,
so C’s hackery doesn’t even work in C]