From ondrej at certik.cz Wed Mar 3 21:22:04 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 3 Mar 2010 12:22:04 -0800 Subject: [Cython] using the C API Declarations in several C++ files Message-ID: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> Hi, I am using the C API Declarations[0], which generate the file _hermes_common_api.h, that I include in my C++ files. I am having problems with initializing the module properly. Here is an example of my C++ function, that internally calls scipy's sparse solver to solve the system: void solve_linear_system_scipy_cg(CooMatrix *mat, double *res) { if (import__hermes_common()) throw std::runtime_error("hermes_common failed to import."); CSRMatrix M(mat); insert_object("m", c2py_CSRMatrix(&M)); insert_object("rhs", c2numpy_double_inplace(res, mat->get_size())); cmd("A = m.to_scipy_csr()"); cmd("from scipy.sparse.linalg import cg"); cmd("x, res = cg(A, rhs)"); double *x; int n; numpy2c_double_inplace(get_object("x"), &x, &n); memcpy(res, x, n*sizeof(double)); } the functions insert_object(), cmd(), c2numpy_double_inplace() etc. are declared in my .pyx file to ease C++ and Python integration (the whole code is at [1]). Everything works, as long as I call the "import__hermes_common()" function. Since I call Python things all over my C++ code, it's kind of annoying to always call this by hand. Also, doesn't it slow things down? What is the best way to do this? It occurred to me that I could create a "Python" C++ class, whose constructor would import the module if needed, and it would be used like this: Python p; p.insert_object(...) p.cmd(...) p.get_object(...) Another annoying thing is that I currently need to have the following at the beginning of my C++ main() function: // This is a hack, this should be set somewhere else: putenv((char *)"PYTHONPATH=../.."); Py_Initialize(); PySys_SetArgv(argc, argv); I would like this to be handled automatically somehow, probably in the constructor of the C++ class "Python". The problem that I am having is that it seems to me, that I need to call import__hermes_common() in every single .cpp file that I have, even though that I link all of them into a shared library. It segfaults if I don't do it. This confuses me. Here is a part of the code in _hermes_common_api.h: static PyObject *(*c2py_CooMatrix)(struct CooMatrix *); static PyObject *(*c2py_CSRMatrix)(struct CSRMatrix *); static void (*cmd)(const char*); static void (*set_verbose_cmd)(int); static void (*insert_object)(const char*, PyObject *); those have to be initalized by calling import__hermes_common(), that is clear, but why isn't it enough to call it once per the project/shared lib? I call it in matrix.cpp, but if I don't call it in python_solvers.cpp as well, it seems it won't get initialized and it segfaults. Maybe it has something to do with how C++ header files work, but I thought that symbols are shared in the whole program/library. Thanks for any hints. Ondrej [0] http://docs.cython.org/docs/external_C_code.html#c-api-declarations [1] http://github.com/certik/hermes_common From dalcinl at gmail.com Wed Mar 3 22:55:48 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Mar 2010 18:55:48 -0300 Subject: [Cython] using the C API Declarations in several C++ files In-Reply-To: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> References: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> Message-ID: On 3 March 2010 17:22, Ondrej Certik wrote: > Hi, > > I am using the C API Declarations[0], which generate the file > _hermes_common_api.h, that I include in my C++ files. I am having > problems with initializing the module properly. > > Here is an example of my C++ function, that internally calls scipy's > sparse solver to solve the system: > > void solve_linear_system_scipy_cg(CooMatrix *mat, double *res) > { > ? ?if (import__hermes_common()) > ? ? ? ?throw std::runtime_error("hermes_common failed to import."); > ? ?CSRMatrix M(mat); > ? ?insert_object("m", c2py_CSRMatrix(&M)); > ? ?insert_object("rhs", c2numpy_double_inplace(res, mat->get_size())); > ? ?cmd("A = m.to_scipy_csr()"); > ? ?cmd("from scipy.sparse.linalg import cg"); > ? ?cmd("x, res = cg(A, rhs)"); > ? ?double *x; > ? ?int n; > ? ?numpy2c_double_inplace(get_object("x"), &x, &n); > ? ?memcpy(res, x, n*sizeof(double)); > } > > the functions insert_object(), cmd(), c2numpy_double_inplace() etc. > are declared in my .pyx file to ease C++ and Python integration (the > whole code is at [1]). Everything works, as long as I call the > "import__hermes_common()" function. Since I call Python things all > over my C++ code, it's kind of annoying to always call this by hand. > Also, doesn't it slow things down? > > What is the best way to do this? > > It occurred to me that I could create a "Python" C++ class, whose > constructor would import the module if needed, and it would be used > like this: > > Python p; > p.insert_object(...) > p.cmd(...) > p.get_object(...) > That's the way I would do it... Moreover, all C++ -- Python interactions should go via this object, so you should wrap the C API calls of your module as methos of the "Python" class. > Another annoying thing is that I currently need to have the following > at the beginning of my C++ main() function: > > ? ? ? ?// This is a hack, this should be set somewhere else: > ? ? ? ?putenv((char *)"PYTHONPATH=../.."); > ? ? ? ?Py_Initialize(); > ? ? ? ?PySys_SetArgv(argc, argv); > > I would like this to be handled automatically somehow, probably in the > constructor of the C++ class "Python". > Yep.. > The problem that I am having is that it seems to me, that I need to > call import__hermes_common() in every single .cpp file that I have, > even though that I link all of them into a shared library. It > segfaults if I don't do it. This confuses me. Here is a part of the > code in _hermes_common_api.h: > > static PyObject *(*c2py_CooMatrix)(struct CooMatrix *); > static PyObject *(*c2py_CSRMatrix)(struct CSRMatrix *); > static void (*cmd)(const char*); > static void (*set_verbose_cmd)(int); > static void (*insert_object)(const char*, PyObject *); > > those have to be initalized by calling import__hermes_common(), that > is clear, but why isn't it enough to call it once per the > project/shared lib? Because these are "static" ... and these are not visible outside compilation units (i.e. outside each C source) > I call it in matrix.cpp, but if I don't call it in > python_solvers.cpp as well, it seems it won't get initialized and it > segfaults. Of course. Each source has it's own copy of these pointers, then you have to init them at each source... Again, if you route stuff using an globally accessible instance of a "Python" class, this is no longer an issue. > Maybe it has something to do with how C++ header files > work, but I thought that symbols are shared in the whole > program/library. > Yes. Symbols are shared, as long as then are not "static" :-) ... Cython could grow support for your use case... If you really feel you will need this (as opposed to implementing the "Python" class), perhaps I could take a look at it and implement something... -- Lisandro Dalcin --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Thu Mar 4 00:28:52 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 3 Mar 2010 15:28:52 -0800 Subject: [Cython] using the C API Declarations in several C++ files In-Reply-To: References: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> Message-ID: <85b5c3131003031528gdc143a6s5e046eb87dd24f5@mail.gmail.com> On Wed, Mar 3, 2010 at 1:55 PM, Lisandro Dalcin wrote: > On 3 March 2010 17:22, Ondrej Certik wrote: >> Hi, >> >> I am using the C API Declarations[0], which generate the file >> _hermes_common_api.h, that I include in my C++ files. I am having >> problems with initializing the module properly. >> >> Here is an example of my C++ function, that internally calls scipy's >> sparse solver to solve the system: >> >> void solve_linear_system_scipy_cg(CooMatrix *mat, double *res) >> { >> ? ?if (import__hermes_common()) >> ? ? ? ?throw std::runtime_error("hermes_common failed to import."); >> ? ?CSRMatrix M(mat); >> ? ?insert_object("m", c2py_CSRMatrix(&M)); >> ? ?insert_object("rhs", c2numpy_double_inplace(res, mat->get_size())); >> ? ?cmd("A = m.to_scipy_csr()"); >> ? ?cmd("from scipy.sparse.linalg import cg"); >> ? ?cmd("x, res = cg(A, rhs)"); >> ? ?double *x; >> ? ?int n; >> ? ?numpy2c_double_inplace(get_object("x"), &x, &n); >> ? ?memcpy(res, x, n*sizeof(double)); >> } >> >> the functions insert_object(), cmd(), c2numpy_double_inplace() etc. >> are declared in my .pyx file to ease C++ and Python integration (the >> whole code is at [1]). Everything works, as long as I call the >> "import__hermes_common()" function. Since I call Python things all >> over my C++ code, it's kind of annoying to always call this by hand. >> Also, doesn't it slow things down? >> >> What is the best way to do this? >> >> It occurred to me that I could create a "Python" C++ class, whose >> constructor would import the module if needed, and it would be used >> like this: >> >> Python p; >> p.insert_object(...) >> p.cmd(...) >> p.get_object(...) >> > > That's the way I would do it... Moreover, all C++ -- Python > interactions should go via this object, so you should wrap the C API > calls of your module as methos of the "Python" class. > >> Another annoying thing is that I currently need to have the following >> at the beginning of my C++ main() function: >> >> ? ? ? ?// This is a hack, this should be set somewhere else: >> ? ? ? ?putenv((char *)"PYTHONPATH=../.."); >> ? ? ? ?Py_Initialize(); >> ? ? ? ?PySys_SetArgv(argc, argv); >> >> I would like this to be handled automatically somehow, probably in the >> constructor of the C++ class "Python". >> > > Yep.. > >> The problem that I am having is that it seems to me, that I need to >> call import__hermes_common() in every single .cpp file that I have, >> even though that I link all of them into a shared library. It >> segfaults if I don't do it. This confuses me. Here is a part of the >> code in _hermes_common_api.h: >> >> static PyObject *(*c2py_CooMatrix)(struct CooMatrix *); >> static PyObject *(*c2py_CSRMatrix)(struct CSRMatrix *); >> static void (*cmd)(const char*); >> static void (*set_verbose_cmd)(int); >> static void (*insert_object)(const char*, PyObject *); >> >> those have to be initalized by calling import__hermes_common(), that >> is clear, but why isn't it enough to call it once per the >> project/shared lib? > > Because these are "static" ... and these are not visible outside > compilation units (i.e. outside each C source) Ah --- that explains everything. I am still learning C++. :) > >> I call it in matrix.cpp, but if I don't call it in >> python_solvers.cpp as well, it seems it won't get initialized and it >> segfaults. > > Of course. Each source has it's own copy of these pointers, then you > have to init them at each source... > > Again, if you route stuff using an globally accessible instance of a > "Python" class, this is no longer an issue. In this case, that'd be cool. It'd be a bit tedious to repeat all the functions in the Python class, but maybe I'll figure out some shortcut. > >> Maybe it has something to do with how C++ header files >> work, but I thought that symbols are shared in the whole >> program/library. >> > > Yes. Symbols are shared, as long as then are not "static" :-) ... Got it --- everything is clear now. > > Cython could grow support for your use case... If you really feel you > will need this (as opposed to implementing the "Python" class), > perhaps I could take a look at it and implement something... Do you have some ideas how Cython could be improved to make my use case easier? I can try to write something too --- I am still in the stage of figuring things out, not sure yet what I really want in details. I only know my general goal --- it should be easy for C++ users to use it, should be robust (no segfaults) and Python internals should not be visible at all. I am half way there already. Thanks for the help, Ondrej From dalcinl at gmail.com Thu Mar 4 00:54:32 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Mar 2010 20:54:32 -0300 Subject: [Cython] using the C API Declarations in several C++ files In-Reply-To: <85b5c3131003031528gdc143a6s5e046eb87dd24f5@mail.gmail.com> References: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> <85b5c3131003031528gdc143a6s5e046eb87dd24f5@mail.gmail.com> Message-ID: On 3 March 2010 20:28, Ondrej Certik wrote: >> >> Cython could grow support for your use case... If you really feel you >> will need this (as opposed to implementing the "Python" class), >> perhaps I could take a look at it and implement something... > > Do you have some ideas how Cython could be improved to make my use case easier? > The easier way to get something working would be to use a preprocessor macro to control the storage (i.e. static vs. extern) of the symbols you get in the generated C header... Then you just have to #include the module API header like this: #define CYTHON_USE_GLOBAL_API #include You will still need to call import_mymodule(), but you just need to do it earlier in your code, likely a few lines below Py_Initialize() Does this look good/easy enough for you? > I can try to write something too --- I am still in the stage of > figuring things out, not sure yet what I really want in details. I > only know my general goal --- it should be easy for C++ users to use > it, should be robust (no segfaults) and Python internals should not be > visible at all. I am half way there already. > I have to insist: The best (though admittedly complex) way to do this is you have a 'Python' class, in such a way its instances serve as a execution context... If you start with this, using separate interpreters (like Apache's mod_python) in the near future could be straightforward. -- Lisandro Dalcin --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Thu Mar 4 02:01:01 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 3 Mar 2010 17:01:01 -0800 Subject: [Cython] using the C API Declarations in several C++ files In-Reply-To: References: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> <85b5c3131003031528gdc143a6s5e046eb87dd24f5@mail.gmail.com> Message-ID: <85b5c3131003031701s1ed40e5ay5e0081e8849e1a05@mail.gmail.com> On Wed, Mar 3, 2010 at 3:54 PM, Lisandro Dalcin wrote: > On 3 March 2010 20:28, Ondrej Certik wrote: >>> >>> Cython could grow support for your use case... If you really feel you >>> will need this (as opposed to implementing the "Python" class), >>> perhaps I could take a look at it and implement something... >> >> Do you have some ideas how Cython could be improved to make my use case easier? >> > > The easier way to get something working would be to use a preprocessor > macro to control the storage (i.e. static vs. extern) of the symbols > you get in the generated C header... Then you just have to #include > the module API header like this: > > #define CYTHON_USE_GLOBAL_API > #include > > You will still need to call import_mymodule(), but you just need to do > it earlier in your code, likely a few lines below Py_Initialize() > > Does this look good/easy enough for you? That looks cool, I'll give it a shot. > >> I can try to write something too --- I am still in the stage of >> figuring things out, not sure yet what I really want in details. I >> only know my general goal --- it should be easy for C++ users to use >> it, should be robust (no segfaults) and Python internals should not be >> visible at all. I am half way there already. >> > > I have to insist: The best (though admittedly complex) way to do this > is you have a 'Python' class, in such a way its instances serve as a > execution context... If you start with this, using separate > interpreters (like Apache's mod_python) in the near future could be > straightforward. Yes, I currently store the context in the "namespace" global dictionary in Cython. I'll give it a shot too. I don't mind using one interpreter, but at least each Python class instance should have it's own namespace. One more question --- can there be any problem if I call for example from Python: m.add() and add() is implemented in C++, and internally it calls some stuff in Python (like scipy). In principle it could be recursively nested couple times. Can the Python interpreter be confused by this "nesting"? Normally it's not an issue since one usually just wraps a C/C++ code, which doesn't call Python. Ondrej From dalcinl at gmail.com Thu Mar 4 02:50:53 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Mar 2010 22:50:53 -0300 Subject: [Cython] using the C API Declarations in several C++ files In-Reply-To: <85b5c3131003031701s1ed40e5ay5e0081e8849e1a05@mail.gmail.com> References: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> <85b5c3131003031528gdc143a6s5e046eb87dd24f5@mail.gmail.com> <85b5c3131003031701s1ed40e5ay5e0081e8849e1a05@mail.gmail.com> Message-ID: On 3 March 2010 22:01, Ondrej Certik wrote: > On Wed, Mar 3, 2010 at 3:54 PM, Lisandro Dalcin wrote: >> On 3 March 2010 20:28, Ondrej Certik wrote: >>>> >>>> Cython could grow support for your use case... If you really feel you >>>> will need this (as opposed to implementing the "Python" class), >>>> perhaps I could take a look at it and implement something... >>> >>> Do you have some ideas how Cython could be improved to make my use case easier? >>> >> >> The easier way to get something working would be to use a preprocessor >> macro to control the storage (i.e. static vs. extern) of the symbols >> you get in the generated C header... Then you just have to #include >> the module API header like this: >> >> #define CYTHON_USE_GLOBAL_API >> #include >> >> You will still need to call import_mymodule(), but you just need to do >> it earlier in your code, likely a few lines below Py_Initialize() >> >> Does this look good/easy enough for you? > > That looks cool, I'll give it a shot. > OK, but you will need an additional step... You have to create an additional C source, #include the generated API header, and make it 'define' all these pointers... So perhaps you will need an additional macro (let say, named CYTHON_DEFINE_GLOBAL_API) to tell the header file "define these symbols at this point" Am I being clear enough? >> >>> I can try to write something too --- I am still in the stage of >>> figuring things out, not sure yet what I really want in details. I >>> only know my general goal --- it should be easy for C++ users to use >>> it, should be robust (no segfaults) and Python internals should not be >>> visible at all. I am half way there already. >>> >> >> I have to insist: The best (though admittedly complex) way to do this >> is you have a 'Python' class, in such a way its instances serve as a >> execution context... If you start with this, using separate >> interpreters (like Apache's mod_python) in the near future could be >> straightforward. > > Yes, I currently store the context in the "namespace" global > dictionary in Cython. I'll give it a shot too. I don't mind using one > interpreter, but at least each Python class instance should have it's > own namespace. > That should work (don't remember the C-API equivalent of "exec in namespace", but Cython sources have the trick in order to implement the 'exec' statement) > One more question --- can there be any problem if I call for example > from Python: > > m.add() > > and add() is implemented in C++, and internally it calls some stuff in > Python (like scipy). In principle it could be recursively nested > couple times. Normal recursion limit applies here... A couple of times should not be a problem, a couple thousand times, better don't do that... > Can the Python interpreter be confused by this > "nesting"? Normally it's not an issue since one usually just wraps a > C/C++ code, which doesn't call Python. > Cython itself can call Python, that calls Cython that call Python, that calls ... -- Lisandro Dalcin --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Thu Mar 4 03:08:24 2010 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 3 Mar 2010 18:08:24 -0800 Subject: [Cython] using the C API Declarations in several C++ files In-Reply-To: References: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> <85b5c3131003031528gdc143a6s5e046eb87dd24f5@mail.gmail.com> <85b5c3131003031701s1ed40e5ay5e0081e8849e1a05@mail.gmail.com> Message-ID: <85b5c3131003031808i48f71b0al2f08b49793c65a5b@mail.gmail.com> On Wed, Mar 3, 2010 at 5:50 PM, Lisandro Dalcin wrote: > On 3 March 2010 22:01, Ondrej Certik wrote: >> On Wed, Mar 3, 2010 at 3:54 PM, Lisandro Dalcin wrote: >>> On 3 March 2010 20:28, Ondrej Certik wrote: >>>>> >>>>> Cython could grow support for your use case... If you really feel you >>>>> will need this (as opposed to implementing the "Python" class), >>>>> perhaps I could take a look at it and implement something... >>>> >>>> Do you have some ideas how Cython could be improved to make my use case easier? >>>> >>> >>> The easier way to get something working would be to use a preprocessor >>> macro to control the storage (i.e. static vs. extern) of the symbols >>> you get in the generated C header... Then you just have to #include >>> the module API header like this: >>> >>> #define CYTHON_USE_GLOBAL_API >>> #include >>> >>> You will still need to call import_mymodule(), but you just need to do >>> it earlier in your code, likely a few lines below Py_Initialize() >>> >>> Does this look good/easy enough for you? >> >> That looks cool, I'll give it a shot. >> > > OK, but you will need an additional step... You have to create an > additional C source, #include the generated API header, and make it > 'define' all these pointers... So perhaps you will need an additional > macro (let say, named CYTHON_DEFINE_GLOBAL_API) to tell the header > file "define these symbols at this point" > > Am I being clear enough? Yes, that is clear to me, thanks. > >>> >>>> I can try to write something too --- I am still in the stage of >>>> figuring things out, not sure yet what I really want in details. I >>>> only know my general goal --- it should be easy for C++ users to use >>>> it, should be robust (no segfaults) and Python internals should not be >>>> visible at all. I am half way there already. >>>> >>> >>> I have to insist: The best (though admittedly complex) way to do this >>> is you have a 'Python' class, in such a way its instances serve as a >>> execution context... If you start with this, using separate >>> interpreters (like Apache's mod_python) in the near future could be >>> straightforward. >> >> Yes, I currently store the context in the "namespace" global >> dictionary in Cython. I'll give it a shot too. I don't mind using one >> interpreter, but at least each Python class instance should have it's >> own namespace. >> > > That should work (don't remember the C-API equivalent of "exec in > namespace", but Cython sources have the trick in order to implement > the 'exec' statement) That is already implemented, here is my exec function: cdef api object run_cmd(const_char_p text, object namespace): try: verbose = namespace.get("verbose") if verbose: print "got a text:", text if verbose: print "evaluting in the namespace:" print namespace code = compile(text, "", "exec") eval(code, {}, namespace) if verbose: print "new namespace:" print namespace return namespace except SystemExit, e: try: exit_code = int(e) except: exit_code = -1 exit(exit_code) except: etype, value, tb = sys.exc_info() s = "".join(traceback.format_exception(etype, value, tb)) s = "Exception raised in the Python code:\n" + s throw_exception(s) > >> One more question --- can there be any problem if I call for example >> from Python: >> >> m.add() >> >> and add() is implemented in C++, and internally it calls some stuff in >> Python (like scipy). In principle it could be recursively nested >> couple times. > > Normal recursion limit applies here... A couple of times should not be > a problem, a couple thousand times, better don't do that... > >> Can the Python interpreter be confused by this >> "nesting"? Normally it's not an issue since one usually just wraps a >> C/C++ code, which doesn't call Python. >> > > Cython itself can call Python, that calls Cython that call Python, > that calls ... Ok, great! i was worried if there were some issues, but it seems everything will work very smoothly and the C++ guys in my group will not even notice that they are using Python. :) Ondrej From dalcinl at gmail.com Thu Mar 4 03:23:19 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 3 Mar 2010 23:23:19 -0300 Subject: [Cython] using the C API Declarations in several C++ files In-Reply-To: <85b5c3131003031808i48f71b0al2f08b49793c65a5b@mail.gmail.com> References: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> <85b5c3131003031528gdc143a6s5e046eb87dd24f5@mail.gmail.com> <85b5c3131003031701s1ed40e5ay5e0081e8849e1a05@mail.gmail.com> <85b5c3131003031808i48f71b0al2f08b49793c65a5b@mail.gmail.com> Message-ID: On 3 March 2010 23:08, Ondrej Certik wrote: >> >> Am I being clear enough? > > Yes, that is clear to me, thanks. > OK. Once you have something working, let's polish it and ask other for opinions, then push to cython-devel. > > That is already implemented, here is my exec function: > > cdef api object run_cmd(const_char_p text, object namespace): > ? ?try: > ? ? ? ?verbose = namespace.get("verbose") > ? ? ? ?if verbose: > ? ? ? ? ? ?print "got a text:", text > ? ? ? ?if verbose: > ? ? ? ? ? ?print "evaluting in the namespace:" > ? ? ? ? ? ?print namespace > ? ? ? ?code = compile(text, "", "exec") > ? ? ? ?eval(code, {}, namespace) > ? ? ? ?if verbose: > ? ? ? ? ? ?print "new namespace:" > ? ? ? ? ? ?print namespace > ? ? ? ?return namespace > ? ?except SystemExit, e: > ? ? ? ?try: > ? ? ? ? ? ?exit_code = int(e) > ? ? ? ?except: > ? ? ? ? ? ?exit_code = -1 > ? ? ? ?exit(exit_code) > ? ?except: > ? ? ? ?etype, value, tb = sys.exc_info() > ? ? ? ?s = "".join(traceback.format_exception(etype, value, tb)) > ? ? ? ?s = "Exception raised in the Python code:\n" + s > ? ? ? ?throw_exception(s) > Mmm, are you sure you want to handle SystemExit by actually calling exit(e)? Is this 'exit' actually 'sys.exit' ? In such case, why just not use 'raise' to re-raise the exception? > > Ok, great! i was worried if there were some issues, but it seems > everything will work very smoothly and the C++ guys in my group will > not even notice that they are using Python. :) > But they should notice, just to have a chance to fall in love with the beast :-) -- Lisandro Dalcin --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From greg.ewing at canterbury.ac.nz Thu Mar 4 04:06:23 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 04 Mar 2010 16:06:23 +1300 Subject: [Cython] using the C API Declarations in several C++ files In-Reply-To: <85b5c3131003031701s1ed40e5ay5e0081e8849e1a05@mail.gmail.com> References: <85b5c3131003031222n26bdd2beo983448ea6ce35a2d@mail.gmail.com> <85b5c3131003031528gdc143a6s5e046eb87dd24f5@mail.gmail.com> <85b5c3131003031701s1ed40e5ay5e0081e8849e1a05@mail.gmail.com> Message-ID: <4B8F23AF.7070701@canterbury.ac.nz> Ondrej Certik wrote: > and add() is implemented in C++, and internally it calls some stuff in > Python (like scipy). In principle it could be recursively nested > couple times. Can the Python interpreter be confused by this > "nesting"? No, there shouldn't be any problem calling back and forth between Python and extension code as deep as you want (subject to the Python recursion depth limitation). -- Greg From yytomo at googlemail.com Thu Mar 4 04:22:19 2010 From: yytomo at googlemail.com (Toni Moll) Date: Thu, 4 Mar 2010 04:22:19 +0100 Subject: [Cython] pxd pyx parser bug for mismatch in function argument with default value Message-ID: <9010d0261003031922g47c7fb3cv466e075b8eac106f@mail.gmail.com> Hello, this is a bug report for cython 0.12.1. The bug occurs when a function argument with default value is named differently in the .pyx and the .pxd file. A minimal example is: foo.pyx contains: =========================== cpdef test(test=5): pass =========================== foo.pxd contains: =========================== cpdef test(FJKHFKLJDHF=*) =========================== the traceback is: /tmp/Cython-0.12.1$ python2.5 cython.py foo.pyx Traceback (most recent call last): File "cython.py", line 8, in main(command_line = 1) File "/tmp/Cython-0.12.1/Cython/ Compiler/Main.py", line 743, in main result = compile(sources, options) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 718, in compile return compile_multiple(source, options) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 690, in compile_multiple result = run_pipeline(source, options) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 561, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 221, in run_pipeline data = phase(data) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 154, in generate_pyx_code module_node.process_implementation(options, result) File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 72, in process_implementation self.generate_c_code(env, options, result) File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 274, in generate_c_code self.body.generate_function_definitions(env, code) File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 341, in generate_function_definitions stat.generate_function_definitions(env, code) File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1106, in generate_function_definitions self.generate_argument_parsing_code(env, code) File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1493, in generate_argument_parsing_code code.putln('%s = %s->%s;' % (arg.cname, Naming.optional_args_cname, self.type.opt_arg_cname(declarator.name))) File "/tmp/Cython-0.12.1/Cython/Compiler/PyrexTypes.py", line 1571, in opt_arg_cname return self.op_arg_struct.base_type.scope.lookup(arg_name).cname AttributeError: 'NoneType' object has no attribute 'cname' best regards and thanks a lot for cython, I like it a lot! Toni -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100304/3e6b57bf/attachment-0001.htm From yytomo at googlemail.com Thu Mar 4 04:13:52 2010 From: yytomo at googlemail.com (Toni Moll) Date: Thu, 4 Mar 2010 04:13:52 +0100 Subject: [Cython] parser bug for mismatches in function arguments with default value Message-ID: <9010d0261003031913v4d8220f3ufa3fada390b7ba75@mail.gmail.com> Hello, this is a bug report for cython 0.12.1. The bug occurs when an function argument with default value is named differently in the .pyx and the .pxd file. A minimal example is: foo.pyx contains: =========================== cpdef test(test=5): pass =========================== foo.pxd contains: =========================== cpdef test(FJKHFKLJDHF=*) =========================== the traceback is: /tmp/Cython-0.12.1$ python2.5 cython.py foo.pyx Traceback (most recent call last): File "cython.py", line 8, in main(command_line = 1) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 743, in main result = compile(sources, options) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 718, in compile return compile_multiple(source, options) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 690, in compile_multiple result = run_pipeline(source, options) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 561, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 221, in run_pipeline data = phase(data) File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 154, in generate_pyx_code module_node.process_implementation(options, result) File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 72, in process_implementation self.generate_c_code(env, options, result) File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 274, in generate_c_code self.body.generate_function_definitions(env, code) File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 341, in generate_function_definitions stat.generate_function_definitions(env, code) File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1106, in generate_function_definitions self.generate_argument_parsing_code(env, code) File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1493, in generate_argument_parsing_code code.putln('%s = %s->%s;' % (arg.cname, Naming.optional_args_cname, self.type.opt_arg_cname(declarator.name))) File "/tmp/Cython-0.12.1/Cython/Compiler/PyrexTypes.py", line 1571, in opt_arg_cname return self.op_arg_struct.base_type.scope.lookup(arg_name).cname AttributeError: 'NoneType' object has no attribute 'cname' best regards and thanks a lot for cython, I like it a lot! Toni -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100304/007f2e57/attachment.htm From robertwb at math.washington.edu Thu Mar 4 10:00:08 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 4 Mar 2010 01:00:08 -0800 Subject: [Cython] parser bug for mismatches in function arguments with default value In-Reply-To: <9010d0261003031913v4d8220f3ufa3fada390b7ba75@mail.gmail.com> References: <9010d0261003031913v4d8220f3ufa3fada390b7ba75@mail.gmail.com> Message-ID: <9921D75D-E42A-40E9-98D2-B9E5035CCBBE@math.washington.edu> Thanks. I've filed a report at http://trac.cython.org/cython_trac/ticket/512 I'm not sure when I'll have time to look at it, but it doesn't look like it'll be too hard to fix. - Robert On Mar 3, 2010, at 7:13 PM, Toni Moll wrote: > Hello, > > this is a bug report for cython 0.12.1. The bug occurs when an > function argument with default value is named differently in > the .pyx and the .pxd file. > A minimal example is: > > foo.pyx contains: > =========================== > cpdef test(test=5): > pass > =========================== > > foo.pxd contains: > =========================== > cpdef test(FJKHFKLJDHF=*) > =========================== > > the traceback is: > > /tmp/Cython-0.12.1$ python2.5 cython.py foo.pyx > Traceback (most recent call last): > File "cython.py", line 8, in > main(command_line = 1) > File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 743, in main > result = compile(sources, options) > File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 718, in > compile > return compile_multiple(source, options) > File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 690, in > compile_multiple > result = run_pipeline(source, options) > File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 561, in > run_pipeline > err, enddata = context.run_pipeline(pipeline, source) > File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 221, in > run_pipeline > data = phase(data) > File "/tmp/Cython-0.12.1/Cython/Compiler/Main.py", line 154, in > generate_pyx_code > module_node.process_implementation(options, result) > File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 72, > in process_implementation > self.generate_c_code(env, options, result) > File "/tmp/Cython-0.12.1/Cython/Compiler/ModuleNode.py", line 274, > in generate_c_code > self.body.generate_function_definitions(env, code) > File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 341, in > generate_function_definitions > stat.generate_function_definitions(env, code) > File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1106, in > generate_function_definitions > self.generate_argument_parsing_code(env, code) > File "/tmp/Cython-0.12.1/Cython/Compiler/Nodes.py", line 1493, in > generate_argument_parsing_code > code.putln('%s = %s->%s;' % (arg.cname, > Naming.optional_args_cname, self.type.opt_arg_cname(declarator.name))) > File "/tmp/Cython-0.12.1/Cython/Compiler/PyrexTypes.py", line > 1571, in opt_arg_cname > return self.op_arg_struct.base_type.scope.lookup(arg_name).cname > AttributeError: 'NoneType' object has no attribute 'cname' > > best regards and thanks a lot for cython, I like it a lot! > Toni > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Thu Mar 4 13:05:35 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 04 Mar 2010 13:05:35 +0100 Subject: [Cython] Library for ODEs with Cython bindings? Message-ID: <4B8FA20F.4010309@student.matnat.uio.no> Are anybody aware of a library for solving ODEs with fast callbacks to Cython? The ones in SciPy require a Python callback at every step... My needs aren't particularily complicated. I'll just wrap something in C otherwise, of course (such as GSL). Dag Sverre From dalcinl at gmail.com Thu Mar 4 14:22:27 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 4 Mar 2010 10:22:27 -0300 Subject: [Cython] Library for ODEs with Cython bindings? In-Reply-To: <4B8FA20F.4010309@student.matnat.uio.no> References: <4B8FA20F.4010309@student.matnat.uio.no> Message-ID: On 4 March 2010 09:05, Dag Sverre Seljebotn wrote: > Are anybody aware of a library for solving ODEs with fast callbacks to > Cython? The ones in SciPy require a Python callback at every step... > IIRC, Jon Olav Vik was working on a Sundials wrapper... Search cython-devel. -- Lisandro Dalcin --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From gregor.thalhammer at gmail.com Thu Mar 4 17:16:56 2010 From: gregor.thalhammer at gmail.com (Gregor Thalhammer) Date: Thu, 4 Mar 2010 17:16:56 +0100 Subject: [Cython] Problems with callback function declaration Message-ID: <42de02941003040816t448de9f4wc7292d26e9aab3cc@mail.gmail.com> Dear all, I experience problems with callbacks declared with the __stdcall calling convention and custom data types as return type. To show the problem I modified the cython callback demo cheese.pyx, other files are the same like in http://hg.cython.org/cython-devel/file/2137466cab4a/Demos/callback cheese.pyx: <<<<<<<<<<<<<<<<<<<<<<<< cdef extern from "cheesefinder.h": ctypedef void myvoid ctypedef myvoid (__stdcall *cheesefunc)(char *name, void *user_data) #this gives 'Syntax error in ctypedef statement' #ctypedef void (__stdcall *cheesefunc)(char *name, void *user_data) #OK void find_cheeses(cheesefunc user_func, void *user_data) def find(f): find_cheeses(callback, f) cdef void __stdcall callback(char *name, void *f): (