CastXML integration #66

goldhoorn · 2015-08-18T13:44:03Z

I started to try to integrate castxml, but it seems that this is NOT a drop-in-replacement.

I currently generate all outputs from gccxml and castxml (castxml is active)
the calls to castxml wen trught, but the gnerated registry is empty.
Someone has to dig into this why the castxml output is not processed correctly.

This is work for either
@doudou, or (Team 8 responsibility) @jmachowinski @saarnold or @jhidalgocarrio

doudou · 2015-08-18T14:05:28Z

Hey, thanks.

You forgot to change the castxml handler to use .hpp files ... (I say "forgot" as you did it in the preprocess method)

doudou · 2015-08-18T14:05:45Z

Still not working with that change, though.

goldhoorn · 2015-08-18T16:00:47Z

This is not needed because the file is not processed later. Only the string is returned .

However semms a bit of work. I can provide tomorrow the preprocessed files.

doudou · 2015-08-18T17:05:08Z

I've brought the stuff up to a point where it looks promising. Basically, the "major" change is that the 'demangled' field has disappeared. The thing is, it was not always there and so using it was making the code that much more complicated.

Code's not clean either. Let's get it to work first.

goldhoorn · 2015-08-19T04:19:05Z

Please keep ind mind that this change is also needed (sorry i forgot to push yesterday):
orocos-toolchain/orogen#64

doudou · 2015-08-19T13:15:58Z

Please keep ind mind that this change is also needed (sorry i forgot to push yesterday)

Actually, it was not (but I merged it anyways, forgetting about that)

The performance optimization of putting all includes in a single temporary file fails with castxml (the type-to-file mappings are not reported properly when doing that). See my comment above.

goldhoorn · 2015-08-19T13:19:07Z

The all in one file causes also this side-problem (or at least would solve it)
orocos-toolchain/orogen#50

goldhoorn · 2015-10-06T07:50:03Z

Ok i spend a bit work in here, the current state is that the importer now roughly works.
I hit a few bugs in typelib (specially the return string <-> nil) problem.

However the current state is that the base/orogen/std could be generated successfull (as far i see).
The problem with base/orogen/types is now definitily the same one as in orocos-toolchain/orogen#50 it seems the headers are reported in a different order by castxml.

However, the orogen bug must be solved before any further development can be done here.
The bug occus here even on a (not-export-all-typekits) setup.

goldhoorn · 2015-10-06T07:52:42Z

@marvin2k (cc)

goldhoorn · 2015-10-06T08:00:26Z

For the sake of clarity: i will not work on this include bug if this header resolution is fixed i might have time to check castxml itself and help for a clear integration. But this special bug needs to be solved by someone else.

marvin2k · 2015-10-06T08:12:33Z

bindings/ruby/lib/typelib/gccxml.rb

@@ -695,6 +647,7 @@ def self.parse_cxx_documentation_before(lines, line)
        def load(required_files, xml)
            @registry = Typelib::Registry.new
            Typelib::Registry.add_standard_cxx_types(registry)
+            registry.alias('/std/basic_string</char>', '/std/string')


Shouldn't this be moved into add_standard_cxx_types? It left my memory why this manual alias is not needed for the gccxml-case...

doudou · 2015-10-29T12:30:56Z

FYI, I've cleaned up the patches, and -- much more importantly -- moved the clang test suite to the common parts and apply it to gccxml, clang and castxml. Currently gccxml/castxml pass most of them (one does not pass because it depends on Eigen ... @marvin2k could you remove the dependency ?). The other one I've not looked into it yet.

What is tested so far is the ability to define the types properly. Metadata is not tested, but it will (especially given how important it is to the orogen side)

I'll from now on have a policy that every single bugfix in the C++ importer should have a corresponding test added.

goldhoorn · 2015-10-29T12:40:51Z

@doudou: do you have a general concept (in mind) how we handle the cxx11 abi's?
Is really ignoring (as here: 64a309d) the way to go?

marvin2k · 2015-10-29T13:12:26Z

Hm, I added the eigen dependency to check the opaque mechanism. Removed them now. I don't know how to run the tests now, so testing this commit is left for the curious among us ;-)

Thanks for attacking this castxml problem, this removes the need for a lot of long-time work on the clang importer!

Ah, I did not edit the generated templates.tlb as it will be generated automatically by running the testsuite?

doudou · 2015-10-29T13:41:14Z

Ah, I did not edit the generated templates.tlb as it will be generated automatically by running the testsuite?

Actually not, the tlb files are used as expected result and compared with the importer result.

marvin2k · 2015-10-29T13:42:41Z

So how are the expected results generated then? By hand? In the cmake-based testing the tlb's were generated in the build-dir and had to be copied manually into the src-folder.

doudou · 2015-10-29T13:44:16Z

So how are the expected results generated then? By hand? In the cmake-based testing the tlb's were generated in the build-dir and had to be copied manually into the src-folder.

That's what I had in mind to add new "expected" file. Generate it, check it by hand to make sure it matches and then add it to the test folder.

marvin2k · 2015-10-29T13:46:14Z

The CMakeList with the target is "gone". I thought this is somehow replaced by minitest?

doudou · 2015-10-29T13:47:35Z

I'm trying to iron out a process here to

easily run a ruby test (currently easy to run the whole test suite, but a pain to run a single test)
save the generated tlb files in the build directory

I'll come back to you

marvin2k · 2015-10-29T13:52:43Z

in shell-speak:

./build/tools/typelib-clang-tlb-importer/typelib-clang-tlb-importer test/ruby/cxx_import_tests/templates.hh -opaquePath=test/ruby/cxx_import_tests/templates.opaques -tlbSavePath=/tmp/templates.tlb
# inspect inspect inspect...
cp /tmp/templates.tlb test/ruby/cxx_import_tests/templates.tlb

doudou · 2015-10-29T14:11:58Z

in shell-speak:

This assumes that you have a clang importer.

doudou · 2015-11-09T16:11:22Z

Rebased and updated things. The castxml and gccxml importers now pass the test suite

One major problem is that castxml does not output the offset of base classes (meaning, it will break for multiple inheritance). I've added a ticket to castxml about that (CastXML/CastXML#34)

doudou · 2015-11-09T16:18:49Z

To run the tests, there's a runner script in build/test/ruby. Just cd in there and run

 ./runner
 ./runner test_cxx_gccxml.rb
 ./runner test_cxx_gccxml.rb -n /test_name_filter/

To generate the tlbs, run

 ./cxx_tlbgen name_of_test_file.hh

hauptmech · 2016-01-04T00:56:32Z

I think I need castxml integrated to build rock with gcc5. How's this going?

One can add a .tlb.IMPORTER_NAME file that is specific to the importer, in which case the test verifies that the loaded registry has all tests for the common registry, and complete equivalence for the loader-specific registry This is because the clang importer currently does not support pointers, but gccxml/castxml does, and I don't want to make it a strong requirement just yet.

Some tests where still selecting their loader based on the TYPELIB_CXX_LOADER envvar, which goes contrary to what we actually want.

…rtefacts

The ones starting with __

After this commit, both gccxml and castxml pass the suite

…s alone

…rs in template Negative integers (e.g. -2) are represented as -0x000000N (e.g. -2 is -0x000000002)

One should add it through the TYPELIB_CASTXML_DEFAULT_OPTIONS environment variable

clang/castxml can handle the vectorization instructions just fine

doudou · 2016-03-29T14:11:38Z

Given the 98% lack of review of PRs (important or otherwise) from anyone to rock-core in the last year or so, I merge this.

goldhoorn changed the title ~~Integrate CastXML~~ DO NOT MERGE: Integrate CastXML Aug 18, 2015

goldhoorn mentioned this pull request Aug 19, 2015

Castxml orocos-toolchain/orogen#64

Merged

marvin2k reviewed Oct 6, 2015
View reviewed changes

planthaber mentioned this pull request Oct 26, 2015

Broken links rock-core/base-doc#42

Open

doudou force-pushed the castxml branch from 0a22ddf to cd75009 Compare October 28, 2015 19:03

doudou force-pushed the castxml branch from 0222b65 to e286895 Compare November 9, 2015 15:15

doudou and others added 26 commits March 17, 2016 11:34

cxx,test: filter source_file_line to make it testable in tlb_cxxgen

de275ee

cxx,test: typedef struct { } NAME; actually defines a named type

bd2d13c

cxx,test: fix loader selection

41397cd

Some tests where still selecting their loader based on the TYPELIB_CXX_LOADER envvar, which goes contrary to what we actually want.

cxx,test: don't pollute generated TLBs with aliases that really are a…

bfb99b0

…rtefacts

cxx,test: fix examples.tlb in C++ common tests

943c219

cxx,test: we do support multi-dimensional arrays

da90f0c

gccxml: ignore private C/C++ typedefs

9932692

The ones starting with __

cxx,test: fix minor discrepancies between castxml and gccxml modes

8053106

After this commit, both gccxml and castxml pass the suite

gccxml: make castxml selectable with the TYPELIB_CXX_IMPORTER envvar

dacb606

gccxml: fix return value of GCCXMLLoader.preprocess

a8ab75f

gccxml: make sure that the importer leaves integer template parameter…

abebbd4

…s alone

cxx,test: move multiple inheritance test from separate to classes.hh

4e1b4e4

gccxml: fix handling of qualified ignored types

600104a

cxx: fix support for nested type resolution

a1f4f47

bindings: fix selecting castxml

50a2958

gccxml: fix wrong combination of && and || in #find_node_by_name

1697ad0

gccxml: ignore types that are declared in private/protected sections

fa4a367

gccxml: fix corner cases in name resolution

8bb363c

gccxml: do not fall back to gccxml if castxml is required

2aa4c18

cxx: hide the output of 'which gccxml.real'

86166ae

add the castxml optional dependency

67ae6c5

add missing dependency on pkg-config

829f850

gccxml: workaround weird gccxml handling of negative integer paramete…

075eee2

…rs in template Negative integers (e.g. -2) are represented as -0x000000N (e.g. -2 is -0x000000002)

castxml: do not mandatorily add '-std=c++11' to the castxml command line

ec66ff5

One should add it through the TYPELIB_CASTXML_DEFAULT_OPTIONS environment variable

castxml: remove EIGEN_DONT_VECTORIZE

096b27e

clang/castxml can handle the vectorization instructions just fine

doudou force-pushed the castxml branch from f6f48dc to 096b27e Compare March 17, 2016 14:37

doudou merged commit 2ab2375 into master Mar 29, 2016

doudou deleted the castxml branch March 29, 2016 14:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CastXML integration #66

CastXML integration #66

goldhoorn commented Aug 18, 2015

doudou commented Aug 18, 2015

doudou commented Aug 18, 2015

goldhoorn commented Aug 18, 2015

doudou commented Aug 18, 2015

goldhoorn commented Aug 19, 2015

doudou commented Aug 19, 2015

goldhoorn commented Aug 19, 2015

goldhoorn commented Oct 6, 2015

goldhoorn commented Oct 6, 2015

goldhoorn commented Oct 6, 2015

marvin2k Oct 6, 2015

doudou commented Oct 29, 2015

goldhoorn commented Oct 29, 2015

marvin2k commented Oct 29, 2015

doudou commented Oct 29, 2015

marvin2k commented Oct 29, 2015

doudou commented Oct 29, 2015

marvin2k commented Oct 29, 2015

doudou commented Oct 29, 2015

marvin2k commented Oct 29, 2015

doudou commented Oct 29, 2015

doudou commented Nov 9, 2015

doudou commented Nov 9, 2015

hauptmech commented Jan 4, 2016

doudou commented Mar 29, 2016

CastXML integration #66

CastXML integration #66

Conversation

goldhoorn commented Aug 18, 2015

doudou commented Aug 18, 2015

doudou commented Aug 18, 2015

goldhoorn commented Aug 18, 2015

doudou commented Aug 18, 2015

goldhoorn commented Aug 19, 2015

doudou commented Aug 19, 2015

goldhoorn commented Aug 19, 2015

goldhoorn commented Oct 6, 2015

goldhoorn commented Oct 6, 2015

goldhoorn commented Oct 6, 2015

marvin2k Oct 6, 2015

Choose a reason for hiding this comment

doudou commented Oct 29, 2015

goldhoorn commented Oct 29, 2015

marvin2k commented Oct 29, 2015

doudou commented Oct 29, 2015

marvin2k commented Oct 29, 2015

doudou commented Oct 29, 2015

marvin2k commented Oct 29, 2015

doudou commented Oct 29, 2015

marvin2k commented Oct 29, 2015

doudou commented Oct 29, 2015

doudou commented Nov 9, 2015

doudou commented Nov 9, 2015

hauptmech commented Jan 4, 2016

doudou commented Mar 29, 2016