From sturla at molden.no Fri Jan 1 04:03:55 2010 From: sturla at molden.no (Sturla Molden) Date: Fri, 1 Jan 2010 04:03:55 +0100 Subject: [Cython] Cython 0.12 not working on Windows XP In-Reply-To: <4B3CA848.6020207@behnel.de> References: <4B2A71F0.1030105@student.matnat.uio.no> <4B2A7CF4.7050006@behnel.de> <473FCA5A-D155-42FC-B1CB-355DEB87B756@math.washington.edu> <4B2B2AF4.10100@behnel.de> <4B2B3154.7060708@behnel.de> <4B3706AF.4040803@earthlink.net> <4B372272.10808@behnel.de> <4B39749C.2050203@earthlink.net> <71a02c81a7f80ae6d31e45bf173d172e.squirrel@webmail.uio.no> <4B3BCDD2.9020309@earthlink.net> <4B3C713C.9010801@molden.no> <4B3CA848.6020207@behnel.de> Message-ID: <74e60ecaed82ca83fdd29b8836922b66.squirrel@webmail.uio.no> [This is perhaps OT on the Cython list. Please forgive me the noise.] Stefan, CreateProcess in Win32 API and fork in Interix (SUA/SFU) both call the kernel function ZwCreateProcess in ntdll.dll. If NULL is passed as section handle to ZwCreateProcess, it will clone the current process. The copy-on-write optimization actually happen in hardware. Most modern processors have a paging memory-management unit that can tag pages as shared and copy-on-write. The reason that early versions of Unix or Linux did not copy-on-write optimize fork but modern do, is that CPUs have evolved. You can read about Windows kernel programming in Nebbet's book on NT kernel internals. It even has example code for a boiler-plate implementation of fork. Cygwin's problem is that hooking up a process created by ZwCreateProcess to the Win32 or SUA subsystem is undocumented; neither MS documentation or Nebbet cover that. That is why Cygwin does not implement a copy-on-write fork yet, although most modern hardware supports it. I recommend using Interix (SFU/SUA) instead of Cygwin. You will need Windows XP professional or enterprice, or Windows Vista / Windows 7 Ultimate. The codebase of Interix is mostly derived from OpenBSD. This means it is very safe. The old POSIX subsystem form Windows NT4 is deprecated in favour of Interix. You'll find more information at www.interix.com. Prebuilt binaries for common Unix tools can be downloaded from the Warehouse in SUA community. Note that SUA is UNIX, not Windows, it just happens to share kernel. This means the Windows API cannot be used; processes do not run on the Win32 subsystem. For graphics you e.g. need to run an X-server on Win32, and good X-servers for Windows tend to be expensive. With modern CPUs that hardware accelerate virtualization (AMD-V or Intel VT-X), I prefer to use virtualization software instead of Interix or Cygwin. My current favourite is Sun Microsystems' VirtualBox, which is free for personal use. Note that the free license for VirtualBox also allow commercial use in an organization; what is not covered is automated deployment on multiple computers. Even without HW accelerated virtualization, VirtualBox or VMware Workstation/Server can be quite efficient. On my Windows 7 desktop computer I run Mandriva Linux 2010 in VirtualBox. (I have also tested Ubuntu and OpenSolaris.) With virtualization, we also avoid having to buy an expensive X-server for Windows. And most software are easier to build and better tested on a modern Linux than Interix. Sturla Stefan Behnel wrote: > Sturla Molden, 31.12.2009 10:39: >> If you need full Unix compatibility, you should use SFU (add-on to Win >> XP) or SUA (Vista and Win7 Ultimate), which gives you a certified UNIX >> on top of the NT-kernel (Interix). > > Interesting, that's the first time I hear about these (although I knew that > NT always had a POSIX-like kernel personality). I found a couple of your > posts to the python MLs that seem to suggest that they even have an > efficient copy-on-write fork implementation. That (alongside with some > other performance issues) is the main drawback of cygwin, IMHO. Do you have > any references that back this claim? I couldn't find anything on a quick > web search. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Fri Jan 1 13:11:20 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 01 Jan 2010 13:11:20 +0100 Subject: [Cython] [OT] Windows and Unix In-Reply-To: <74e60ecaed82ca83fdd29b8836922b66.squirrel@webmail.uio.no> References: <4B2A71F0.1030105@student.matnat.uio.no> <4B2A7CF4.7050006@behnel.de> <473FCA5A-D155-42FC-B1CB-355DEB87B756@math.washington.edu> <4B2B2AF4.10100@behnel.de> <4B2B3154.7060708@behnel.de> <4B3706AF.4040803@earthlink.net> <4B372272.10808@behnel.de> <4B39749C.2050203@earthlink.net> <71a02c81a7f80ae6d31e45bf173d172e.squirrel@webmail.uio.no> <4B3BCDD2.9020309@earthlink.net> <4B3C713C.9010801@molden.no> <4B3CA848.6020207@behnel.de> <74e60ecaed82ca83fdd29b8836922b66.squirrel@webmail.uio.no> Message-ID: <4B3DE668.6040401@student.matnat.uio.no> Sturla Molden wrote: > [This is perhaps OT on the Cython list. Please forgive me the noise.] Well, I for one think it is interesting, so continuing OT... > > Stefan, > > CreateProcess in Win32 API and fork in Interix (SUA/SFU) both call the > kernel function ZwCreateProcess in ntdll.dll. If NULL is passed as section > handle to ZwCreateProcess, it will clone the current process. The > copy-on-write optimization actually happen in hardware. Most modern > processors have a paging memory-management unit that can tag pages as > shared and copy-on-write. The reason that early versions of Unix or Linux > did not copy-on-write optimize fork but modern do, is that CPUs have > evolved. You can read about Windows kernel programming in Nebbet's book on > NT kernel internals. It even has example code for a boiler-plate > implementation of fork. > > Cygwin's problem is that hooking up a process created by ZwCreateProcess > to the Win32 or SUA subsystem is undocumented; neither MS documentation or > Nebbet cover that. That is why Cygwin does not implement a copy-on-write > fork yet, although most modern hardware supports it. > > I recommend using Interix (SFU/SUA) instead of Cygwin. You will need > Windows XP professional or enterprice, or Windows Vista / Windows 7 > Ultimate. The codebase of Interix is mostly derived from OpenBSD. This > means it is very safe. The old POSIX subsystem form Windows NT4 is > deprecated in favour of Interix. You'll find more information at > www.interix.com. Prebuilt binaries for common Unix tools can be downloaded > from the Warehouse in SUA community. How fun is it to develop something which requires Windows Ultimate/Enterprise though? For applications one only uses oneself that is fine, but then one might just as well use Linux. In my experience, Cygwin is often used as an easy way out for porting open source software to Windows, and SFU/SUA seem to exclude at least most of the home users and quite a few laptop users. Can one compile gcc for SFU/SUA? Does that have less or more problems than gcc for Cygwin? -- Dag Sverre From sturla at molden.no Sun Jan 3 07:23:20 2010 From: sturla at molden.no (Sturla Molden) Date: Sun, 3 Jan 2010 07:23:20 +0100 Subject: [Cython] [OT] Windows and Unix In-Reply-To: <4B3DE668.6040401@student.matnat.uio.no> References: <4B2A71F0.1030105@student.matnat.uio.no> <4B2A7CF4.7050006@behnel.de> <473FCA5A-D155-42FC-B1CB-355DEB87B756@math.washington.edu> <4B2B2AF4.10100@behnel.de> <4B2B3154.7060708@behnel.de> <4B3706AF.4040803@earthlink.net> <4B372272.10808@behnel.de> <4B39749C.2050203@earthlink.net> <71a02c81a7f80ae6d31e45bf173d172e.squirrel@webmail.uio.no> <4B3BCDD2.9020309@earthlink.net> <4B3C713C.9010801@molden.no> <4B3CA848.6020207@behnel.de> <74e60ecaed82ca83fdd29b8836922b66.squirrel@webmail.uio.no> <4B3DE668.6040401@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > In my experience, Cygwin is often used as an > easy way out for porting open source software to Windows, and SFU/SUA > seem to exclude at least most of the home users and quite a few laptop > users. First, note that Cygwin is GPL unless you buy a commercial license (for a fee undisclosed by Red Hat). The Cygwin fork call is not copy-on-write optimized, which makes Cygwin unsuited for fork-based internet servers. Also, programs that require inter-process read-only access to huge memory buffers can avoid using shared memory by forking. This trick will not work on Cygwin. And then there is the security issue. How safe is Cygwin against various exploits? As far as I can tell, there is no auto-update of system components. > Can one compile gcc for SFU/SUA? Does that have less or more problems > than gcc for Cygwin? gcc is the system C compiler on Interix. Microsoft is for some reason not using their own C compiler, but rather relying on gcc. Perhaps Visual C++ did not pass UNIX certification; or perhaps this is a decision to make porting from Linux easier. I don't know. But in any case, gcc is preinstalled, you can build your own, or download one here: http://www.suacommunity.com/tool_warehouse.htm Personally I prefer Sun's VirtualBox (PUEL license, not the GPL version) with a modern Linux or Unix to Cygwin or Interix. http://www.virtualbox.org/ From zstone at gmail.com Sun Jan 3 07:58:44 2010 From: zstone at gmail.com (Zak Stone) Date: Sat, 2 Jan 2010 20:58:44 -1000 Subject: [Cython] quick novice question about type conversion In-Reply-To: <4eb86d71291c5891e1d6e3edf660df5b.squirrel@webmail.uio.no> References: <6629e85c0912270106r22e40e91tea5d92c5165ed24b@mail.gmail.com> <1d6da7d58994947f56d9ba771f87c5af.squirrel@webmail.uio.no> <6629e85c0912271340oecb4a1bj48d6b25d08e0c7de@mail.gmail.com> <4eb86d71291c5891e1d6e3edf660df5b.squirrel@webmail.uio.no> Message-ID: <6629e85c1001022258j2036d57bp274d63dd3021e3a8@mail.gmail.com> Thank you very much for these additional details, Dag! They were exactly what I needed. Many thanks to all of you for your inspiring work on Cython, and happy new year! Zak On Mon, Dec 28, 2009 at 4:23 AM, Dag Sverre Seljebotn wrote: >> Thanks for the quick reply. Could you say just a little more about how >> to work around this issue? The numpy.pxd file is close to 1000 lines >> long, and I don't know enough about Cython or NumPy to dive in and do >> a hot-fix. > > You simply do > > cdef ndarray[int] myarray = np.zeros((100,), dtype=np.intc) > > "int" on the left is compile-time context and refers to c int, while > np.intc on the right side is a run-time object (where numpy.pxd doesn't > really come into play). > > What you need to fix NumPy pxd for is so that you can type the completely > equivalent > > cdef ndarray[np.intc_t] myarray = np.zeros((100,), dtype=np.intc) > > Dag > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From hoytak at gmail.com Mon Jan 4 23:43:43 2010 From: hoytak at gmail.com (Hoyt Koepke) Date: Mon, 4 Jan 2010 14:43:43 -0800 Subject: [Cython] ANN: TreeDict, a new cython package Message-ID: <4db580fd1001041443i75abaae6ra85d2c5a0b1ef832@mail.gmail.com> Hello, I'd like to announce a new open source python/cython package, TreeDict, that the cython community might be interested in. I've been using it and testing it for a number of months and have finally documented it sufficiently to release. This is my first real open source software package that I hope can be generally useful, particularly to the scientific community, and I'd greatly appreciate feedback on any and all parts of the design/coding/project/release/documentation from the more experienced developers here. -------------------------------------------------------- TreeDict is a dictionary-like, hierarchical python container to simplify the bookkeeping surrounding parameters, variables and data. It aims to be fast, lightweight, intuitive, feature-rich and stable. It's written in python/cython and BSD licensed. While intended for general python development, it includes a number of features particularly useful for scientific programming. It is similar in basic functionality to MATLAB structures in terms of concise syntax and implicit branch creation. In addition, TreeDict implements all the methods of regular dictionaries, pickling, fast non-intersecting hashing for efficient caching, manipulations on the tree structure, fuzzy key matching for helpful error messages, and a system for forward-referencing branches to make lists of parameters more readable. Documentation and examples are available at http://www.stat.washington.edu/~hoytak/code/treedict/. To install, use "easy_install treedict", download it directly from the python cheeseshop at http://pypi.python.org/pypi/treedict, or pull from my github repository with "git clone git://github.com/hoytak/treedict.git". Note that the source tarballs include the generated C source, but the git repository does not. Thanks! --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From njs at vorpus.org Tue Jan 5 03:30:07 2010 From: njs at vorpus.org (Nathaniel Smith) Date: Mon, 4 Jan 2010 18:30:07 -0800 Subject: [Cython] ANN: TreeDict, a new cython package In-Reply-To: <4db580fd1001041443i75abaae6ra85d2c5a0b1ef832@mail.gmail.com> References: <4db580fd1001041443i75abaae6ra85d2c5a0b1ef832@mail.gmail.com> Message-ID: <961fa2b41001041830l25cc767eu3ea71873703b771e@mail.gmail.com> On Mon, Jan 4, 2010 at 2:43 PM, Hoyt Koepke wrote: > I'd like to announce a new open source python/cython package, > TreeDict, that the cython community might be interested in. ?I've been > using it and testing it for a number of months and have finally > documented it sufficiently to release. ?This is my first real open > source software package that I hope can be generally useful, > particularly to the scientific community, and I'd greatly appreciate > feedback on any and all parts of the > design/coding/project/release/documentation from the more experienced > developers here. Neat! From a quick read of the docs, I have a few minor questions: -- Why invent a new mechanism for global variables (getTree/treeExists)? It seems redundant (and violates "there's only one way to do it", if you care about that prescription for pythonicity ;-) -- I'd prefer PEP8-complaint method names, e.g. branch_name rather than branchName. -- What advantage do you get from using cython instead of regular python? I guess the edit distance calculation for similarity matching is CPU-bound, but I wouldn't expect even that to be a bottleneck in regular usage. -- Nathaniel From hoytak at gmail.com Tue Jan 5 05:10:22 2010 From: hoytak at gmail.com (Hoyt Koepke) Date: Mon, 4 Jan 2010 20:10:22 -0800 Subject: [Cython] ANN: TreeDict, a new cython package In-Reply-To: <961fa2b41001041830l25cc767eu3ea71873703b771e@mail.gmail.com> References: <4db580fd1001041443i75abaae6ra85d2c5a0b1ef832@mail.gmail.com> <961fa2b41001041830l25cc767eu3ea71873703b771e@mail.gmail.com> Message-ID: <4db580fd1001042010n74d87196wd22273468c8f7ad4@mail.gmail.com> > Neat! From a quick read of the docs, I have a few minor questions: Thanks for the look & the helpful feedback! I appreciate it. > ?-- Why invent a new mechanism for global variables > (getTree/treeExists)? It seems redundant (and violates "there's only > one way to do it", if you care about that prescription for pythonicity > ;-) I see what you mean, but there are a couple, possibly weak, reasons why I thought those would be useful. First, my inspiration was the python logging module with its getLogger function that instantiates/returns a logger with that given name, and the resulting logger is common across modules. Second, it always felt a little weird to me to "write" important information to an imported module's namespace without some sort of function that lets you know what you're doing. However, a bit of googling indicates that this is an accepted way of doing things if there's a global config file imported everywhere and all the config parameters are set in that file. Now I'm +0 on keeping them in. > ?-- I'd prefer PEP8-complaint method names, e.g. branch_name rather > than branchName. Ah... I was not familiar with that PEP, but it makes sense. I chose to do it this way since I usually have variable names in that style, which become attributes with TreeDict, and wanted to avoid conflicts. However, I think a PEP would overrule that. Since I'm currently the only one using it, I can change over to that. > ?-- What advantage do you get from using cython instead of regular > python? I guess the edit distance calculation for similarity matching > is CPU-bound, but I wouldn't expect even that to be a bottleneck in > regular usage. I admit it is possible that I over-designed it, but I felt like there is enough to check that would be much faster with C than in Python. For example, analogously to dict, iteration locks all the branches affected by the iteration such that setting a value when an iterator is active raises an exception. Thus when setting a value on a branch, I need to walk from that branch all the way to the root checking for such a flag. In python, this could require a number of (fast) lookups; however, with my cython code the entire check process is C code. There's similar checks required for frozen trees, updating, etc. Since I wanted this data structure to be fast enough to use in inner loops (the attribute-style syntax suggests such a use), I thought cython would be a better choice. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From mattcbro at earthlink.net Wed Jan 6 02:04:48 2010 From: mattcbro at earthlink.net (Matthew Bromberg) Date: Tue, 05 Jan 2010 20:04:48 -0500 Subject: [Cython] Arrays of Python Objects Message-ID: <4B43E1B0.5040908@earthlink.net> I need to create a fast hashing function to cache the creation and deletion of an extension class that uses an external C buffer. (It's actually a buffer on a hardware device.) I can't afford to keep creating and deleting these items so I want to save them all off and reuse them as needed. So I thought it would be easy just to create an array of python objects and use a simple hash into an array written in cython and maybe a little C. I was wrong. A buffer of the type cdef object[int] myhash can only be declared locally inside some function or method. I don't see any suitable python class. The closest would be set, but this needs to be minimalistic and as fast as possible. Set is just too heavy and probably slow for my needs. In fact it would be nice if I could override the creation and destruction of my extension classes so that the class and it's python attribute objects don't get garbage collected. My best hope was storing it in some kind of python aware cython array, but I'm not sure this will work. If I store objects in C arrays somehow, then the objects will get garbage collected anyway. It seems to be a bit of a conundrum. Any thoughts? From sturla at molden.no Wed Jan 6 14:01:43 2010 From: sturla at molden.no (Sturla Molden) Date: Wed, 6 Jan 2010 14:01:43 +0100 Subject: [Cython] Arrays of Python Objects In-Reply-To: <4B43E1B0.5040908@earthlink.net> References: <4B43E1B0.5040908@earthlink.net> Message-ID: <9bca51a9d7736db724f5fe9782d1c829.squirrel@webmail.uio.no> > I was wrong. A buffer of the type > cdef object[int] myhash > can only be declared locally inside some function or method. Then write an extension class that wraps the buffer, and store one on module scope. Something like this (not tested): DEF BUF_SIZE = 32 import myclass # your extension class from python cimport PyObject, Py_INCREF, Py_DECREF cdef class bufferwrapper: cdef PyObject *buffer[BUF_SIZE] def __cinit__(bufferwrapper self): cdef int n for n in range(BUF_SIZE): self.buffer[n] = NULL def __init__(bufferwrapper self): cdef int n cdef object tmp for n in range(BUF_SIZE): tmp = myclass.myclass() Py_INCREF(tmp) self.buffer[n] = tmp def __dealloc__(bufferwrapper self): cdef int n for n in range(BUF_SIZE): if self.buffer[n] != NULL: Py_DECREF( self.buffer[n]) cdef bufferwrapper _buffer = bufferwrapper() def foobar(int hashvalue): cdef object obj obj = _buffer.buffer[hashvalue] Regards, Sturla Molden From sturla at molden.no Wed Jan 6 14:22:53 2010 From: sturla at molden.no (Sturla Molden) Date: Wed, 6 Jan 2010 14:22:53 +0100 Subject: [Cython] Arrays of Python Objects In-Reply-To: <4B43E1B0.5040908@earthlink.net> References: <4B43E1B0.5040908@earthlink.net> Message-ID: Matthew Bromberg wrote: > I was wrong. A buffer of the type > cdef object[int] myhash > can only be declared locally inside some function or method. Ignore my last post!!! You are confusing PEP 3118 buffers with C arrays. You probably want this: DEF BUF_SIZE = 32 cdef object myhash[BUF_SIZE] which should work at module scope. This says "myhash is a C array of object, of length BUF_SIZE." Your statement cdef object[int] myhash is not a Cython syntax error, but means something else: "object is a type that exposes a PEP 3118 Py_buffer storing C integers, and myhash is an instance of PEP 3118 Py_buffer." A Py_buffer can only be used inside a method because of the auxillary variables to which it unboxes for fast array lookup. Hence the error message. PEP 3118 syntax is mostly used with NumPy arrays for numerical programming. Sturla Molden From stefan_ml at behnel.de Wed Jan 6 14:28:03 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 Jan 2010 14:28:03 +0100 Subject: [Cython] Arrays of Python Objects In-Reply-To: References: <4B43E1B0.5040908@earthlink.net> Message-ID: <4B448FE3.1050901@behnel.de> Sturla Molden, 06.01.2010 14:22: > You probably want this: > > DEF BUF_SIZE = 32 > cdef object myhash[BUF_SIZE] And, in case the array size needs to be determined at runtime, why not just use a regular Python tuple? Stefan From sturla at molden.no Wed Jan 6 14:46:13 2010 From: sturla at molden.no (Sturla Molden) Date: Wed, 6 Jan 2010 14:46:13 +0100 Subject: [Cython] Arrays of Python Objects In-Reply-To: <4B448FE3.1050901@behnel.de> References: <4B43E1B0.5040908@earthlink.net> <4B448FE3.1050901@behnel.de> Message-ID: > And, in case the array size needs to be determined at runtime, why not > just > use a regular Python tuple? I agree. It seems array lookup is not the bottleneck here, but object creation. He should just store a container on module scope. I would have used a Python list as it is mutable. cdef list myhash = [extclass() for n in range(bufsize)] S.M. From stefan_ml at behnel.de Wed Jan 6 18:37:27 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 Jan 2010 18:37:27 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? Message-ID: <4B44CA57.2010201@behnel.de> Hi, I think it would be nice to support 'cdef inline' methods in extension types and let Cython call them without the normal method indirection, e.g. cdef class MyType: cdef list large_list cdef size_t y cdef big_method(self): for x in self.large_list: self.small_method(x) cdef inline small_method(self, x): self.y += x would emit a direct call to the ..._small_method(self, x) C function in big_method(), instead of passing through self->__pyx_vtab->small_method. This would require these methods to be non-overridable, which I think makes sense for something declared 'inline'. Thoughts? Stefan From mattcbro at earthlink.net Wed Jan 6 19:57:02 2010 From: mattcbro at earthlink.net (Matthew Bromberg) Date: Wed, 06 Jan 2010 13:57:02 -0500 Subject: [Cython] Arrays of Python Objects In-Reply-To: <9bca51a9d7736db724f5fe9782d1c829.squirrel@webmail.uio.no> References: <4B43E1B0.5040908@earthlink.net> <9bca51a9d7736db724f5fe9782d1c829.squirrel@webmail.uio.no> Message-ID: <4B44DCFE.10509@earthlink.net> Thanks that's pretty close to what I'm doing now. I'm a bit new to writing Python extension classes and I'm just working out the logic of the reference counting mechanism. It turns out that the tricky part is overriding the allocation and deallocation of the python objects associated with my extension class, as well as the extension class structure itself. The logical place to try to capture a cython class that's being destroyed is in __dealloc__(). However as I mentioned earlier, once you are actually in __dealloc__() there is no way to save the allocated data structure associated with self that I could see. I have no idea when __del__() is supposed to be called. I put a print statement in that attribute and it never was called. Thus even if I had the code as you've written it, it would fail if I had attempted to save the object by using a Py_INCREF. Once you're in __dealloc__() self will be destroyed no matter what the ref count is, as far as I can tell (cython 0.11.2 behavior). So my current strategy is to create a new container class for the hash function that simply holds all the attributes of self. I'll have to test if I need a Py_INCREF for any attributes that are python objects contained in self. It's not quite as nice since there will be an extra python/cython allocation of the hash container class. Earlier I had thought the python objects contained in self were also being automatically destroyed if their refcounts were 0, but such behavior would be a bug I think. If there was a way to intercept a garbage collect prior to __dealloc__() or to override the default behavior of destroying self, then I could avoid the extra allocation of the hash container. I'll be doing some tests soon to see how much it costs me. -Matt On 2:59 PM, Sturla Molden wrote: > >> I was wrong. A buffer of the type >> cdef object[int] myhash >> can only be declared locally inside some function or method. >> > > Then write an extension class that wraps the buffer, and store one on > module scope. Something like this (not tested): > > > DEF BUF_SIZE = 32 > > import myclass # your extension class > > from python cimport PyObject, Py_INCREF, Py_DECREF > > cdef class bufferwrapper: > > cdef PyObject *buffer[BUF_SIZE] > > def __cinit__(bufferwrapper self): > cdef int n > for n in range(BUF_SIZE): > self.buffer[n] = NULL > > def __init__(bufferwrapper self): > cdef int n > cdef object tmp > for n in range(BUF_SIZE): > tmp = myclass.myclass() > Py_INCREF(tmp) > self.buffer[n] = tmp > > def __dealloc__(bufferwrapper self): > cdef int n > for n in range(BUF_SIZE): > if self.buffer[n] != NULL: > Py_DECREF( self.buffer[n]) > > cdef bufferwrapper _buffer = bufferwrapper() > > def foobar(int hashvalue): > cdef object obj > obj = _buffer.buffer[hashvalue] > > > > Regards, > Sturla Molden > > > > > > From dagss at student.matnat.uio.no Wed Jan 6 20:18:22 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 Jan 2010 20:18:22 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B44CA57.2010201@behnel.de> References: <4B44CA57.2010201@behnel.de> Message-ID: <4B44E1FE.7000900@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > I think it would be nice to support 'cdef inline' methods in extension > types and let Cython call them without the normal method indirection, e.g. > > cdef class MyType: > cdef list large_list > cdef size_t y > > cdef big_method(self): > for x in self.large_list: > self.small_method(x) > > cdef inline small_method(self, x): > self.y += x > > would emit a direct call to the ..._small_method(self, x) C function in > big_method(), instead of passing through self->__pyx_vtab->small_method. > > This would require these methods to be non-overridable, which I think makes > sense for something declared 'inline'. > > Thoughts? I remember wanting this feature when I first saw Cython (then never got around to it). So very much +1 on the feature, not sure about syntax. Java already has the concept of non-polymorphic dispatches through the "final" keyword. So what I was thinking was a new keyword would be used, so that one did "cdef inline final small_method". But -- there is probably not a use for inline non-final methods as they can't be inlined anyway, and I suppose the times one want a non-inline final method are very scarce (I suppose it would be for a method which took some time to execute but one wanted to actively prevent subclasses from overriding it -- which is not the rationale behind the feature at all). So I guess I'm +1 on the syntax proposed as well.. -- Dag Sverre From mattcbro at earthlink.net Wed Jan 6 21:50:45 2010 From: mattcbro at earthlink.net (Matthew Bromberg) Date: Wed, 06 Jan 2010 15:50:45 -0500 Subject: [Cython] Arrays of Python Objects Message-ID: <4B44F7A5.8060502@earthlink.net> How does tuple or list compare speed wise with dict? Ultimately I have to hash into my list using size information. This also still does not address my confusion with regards to how to capture a python object before it get's destroyed. I can't do it in __dealloc__() as I mentioned earlier since no ref count increment will save the self object there. I guess I could put it in the hash when I first create it, but I would need some logic that prevents it being allocated while it is still alive. Hmm that's no good because the ref count would never go to zero and it creates a defacto memory leak. I still can't figure out how to do this without creating a container class that holds the attributes of the class object I'm trying to hash. -Matt --------------------------- Sturla Molden Wed, 06 Jan 2010 05:46:50 -0800 > And, in case the array size needs to be determined at runtime, why not > just > use a regular Python tuple? I agree. It seems array lookup is not the bottleneck here, but object creation. He should just store a container on module scope. I would have used a Python list as it is mutable. cdef list myhash = [extclass() for n in range(bufsize)] S.M. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100106/40c3f9ca/attachment.htm From stefan_ml at behnel.de Thu Jan 7 09:05:37 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 09:05:37 +0100 Subject: [Cython] Arrays of Python Objects In-Reply-To: <4B44F7A5.8060502@earthlink.net> References: <4B44F7A5.8060502@earthlink.net> Message-ID: <4B4595D1.7030907@behnel.de> Matthew Bromberg, 06.01.2010 21:50: > How does tuple or list compare speed wise with dict? Like apples and oranges, basically. > Ultimately I have to hash into my list using size information. Any specific reason why you /can't/ use a dict? > This also still does not address my confusion with regards to how to > capture a python object before it get's destroyed. As long as there is a reference to it (e.g. in the hash table), it won't get deallocated. So: use a Python list for your hash table, stop caring about ref-counts and it will just work. Stefan From stefan_ml at behnel.de Thu Jan 7 09:19:13 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 09:19:13 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B44E1FE.7000900@student.matnat.uio.no> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> Message-ID: <4B459901.2000401@behnel.de> Dag Sverre Seljebotn, 06.01.2010 20:18: > there is probably not a use for inline non-final methods as they > can't be inlined anyway, and I suppose the times one want a non-inline > final method are very scarce That was my intuition, too. I doubt that this feature is any interesting beyond inlining. If the non-overridable function is large enough to prohibit inlining, it's likely not worth speeding up the call anyway. Also, even if the C compiler sees the 'inline' keyword, it's free to ignore it if it considers the function too large to have any positive impact. Stefan From robertwb at math.washington.edu Thu Jan 7 09:54:33 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jan 2010 00:54:33 -0800 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B44E1FE.7000900@student.matnat.uio.no> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> Message-ID: <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> On Jan 6, 2010, at 11:18 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> I think it would be nice to support 'cdef inline' methods in >> extension >> types and let Cython call them without the normal method >> indirection, e.g. >> >> cdef class MyType: >> cdef list large_list >> cdef size_t y >> >> cdef big_method(self): >> for x in self.large_list: >> self.small_method(x) >> >> cdef inline small_method(self, x): >> self.y += x >> >> would emit a direct call to the ..._small_method(self, x) C >> function in >> big_method(), instead of passing through self->__pyx_vtab- >> >small_method. >> >> This would require these methods to be non-overridable, which I >> think makes >> sense for something declared 'inline'. >> >> Thoughts? > > I remember wanting this feature when I first saw Cython (then never > got > around to it). So very much +1 on the feature, not sure about syntax. This has been floating around in the back of my head as well, so +1. > Java already has the concept of non-polymorphic dispatches through the > "final" keyword. So what I was thinking was a new keyword would be > used, > so that one did "cdef inline final small_method". > > But -- there is probably not a use for inline non-final methods as > they > can't be inlined anyway, and I suppose the times one want a non-inline > final method are very scarce (I suppose it would be for a method which > took some time to execute but one wanted to actively prevent > subclasses > from overriding it -- which is not the rationale behind the feature at > all). So I guess I'm +1 on the syntax proposed as well.. One of the primary reasons for the Java final keyword is compiler optimization (either inlining and faster dispatching). Given that we already support the inline keyword, I think it makes sense to have it imply non-overridable. - Robert From robertwb at math.washington.edu Thu Jan 7 10:15:39 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jan 2010 01:15:39 -0800 Subject: [Cython] Arrays of Python Objects In-Reply-To: <4B4595D1.7030907@behnel.de> References: <4B44F7A5.8060502@earthlink.net> <4B4595D1.7030907@behnel.de> Message-ID: On Jan 7, 2010, at 12:05 AM, Stefan Behnel wrote: > Matthew Bromberg, 06.01.2010 21:50: >> How does tuple or list compare speed wise with dict? > > Like apples and oranges, basically. If you're trying to index into it with an int, especially a c int, lists and tuples will be much faster. >> Ultimately I have to hash into my list using size information. > > Any specific reason why you /can't/ use a dict? Which will likely be just as fast, if not faster, than hashing into a list manually yourself. >> This also still does not address my confusion with regards to how to >> capture a python object before it get's destroyed. > > As long as there is a reference to it (e.g. in the hash table), it > won't > get deallocated. So: use a Python list for your hash table, stop > caring > about ref-counts and it will just work. +1. Ideally, you should never have to worry about reference counts when working with Cython at all. I'm still not quite sure exactly what you're trying to do, but if it's creating and deleting thousands of these objects a second and that's killing you (the actual allocation/deallocation, not the initialization) then what you might want to do is something like http://hg.sagemath.org/sage-main/file/21efb0b3fc47/sage/rings/real_double.pyx#l2260 which is a bit hackish and will probably need to be adapted to your specific case. If initializing is expensive, than you can probably keep around a pool of initalized pointers/buffers/whatever, and have the object creation just set/unset these fields (much cleaner). - Robert From robertwb at math.washington.edu Thu Jan 7 10:26:35 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jan 2010 01:26:35 -0800 Subject: [Cython] [OT] Windows and Unix In-Reply-To: References: <4B2A71F0.1030105@student.matnat.uio.no> <4B2A7CF4.7050006@behnel.de> <473FCA5A-D155-42FC-B1CB-355DEB87B756@math.washington.edu> <4B2B2AF4.10100@behnel.de> <4B2B3154.7060708@behnel.de> <4B3706AF.4040803@earthlink.net> <4B372272.10808@behnel.de> <4B39749C.2050203@earthlink.net> <71a02c81a7f80ae6d31e45bf173d172e.squirrel@webmail.uio.no> <4B3BCDD2.9020309@earthlink.net> <4B3C713C.9010801@molden.no> <4B3CA848.6020207@behnel.de> <74e60ecaed82ca83fdd29b8836922b66.squirrel@webmail.uio.no> <4B3DE668.6040401@student.matnat.uio.no> Message-ID: On Jan 2, 2010, at 10:23 PM, Sturla Molden wrote: > Dag Sverre Seljebotn wrote: > >> In my experience, Cygwin is often used as an >> easy way out for porting open source software to Windows, and SFU/SUA >> seem to exclude at least most of the home users and quite a few >> laptop >> users. > > First, note that Cygwin is GPL unless you buy a commercial license > (for a > fee undisclosed by Red Hat). > > The Cygwin fork call is not copy-on-write optimized, which makes > Cygwin > unsuited for fork-based internet servers. Also, programs that require > inter-process read-only access to huge memory buffers can avoid using > shared memory by forking. This trick will not work on Cygwin. > > And then there is the security issue. How safe is Cygwin against > various > exploits? As far as I can tell, there is no auto-update of system > components. > >> Can one compile gcc for SFU/SUA? Does that have less or more problems >> than gcc for Cygwin? > > gcc is the system C compiler on Interix. Microsoft is for some > reason not > using their own C compiler, but rather relying on gcc. Perhaps > Visual C++ > did not pass UNIX certification; or perhaps this is a decision to make > porting from Linux easier. I don't know. But in any case, gcc is > preinstalled, you can build your own, or download one here: > > http://www.suacommunity.com/tool_warehouse.htm > > Personally I prefer Sun's VirtualBox (PUEL license, not the GPL > version) > with a modern Linux or Unix to Cygwin or Interix. > > http://www.virtualbox.org/ And I prefer running a modern Linux or Unix natively on my system :). Virtualbox is a nice option for Windows users when it works though. Of course, from the developers point of view, no matter how nice we set up our own environment of choice, the issue remains that we don't want to require users to have to upgrade Windows Ultimate to compile/ run our code. - Robert From dagss at student.matnat.uio.no Thu Jan 7 11:38:21 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 Jan 2010 11:38:21 +0100 Subject: [Cython] [OT] Windows and Unix In-Reply-To: References: <4B2A71F0.1030105@student.matnat.uio.no> <4B2A7CF4.7050006@behnel.de> <473FCA5A-D155-42FC-B1CB-355DEB87B756@math.washington.edu> <4B2B2AF4.10100@behnel.de> <4B2B3154.7060708@behnel.de> <4B3706AF.4040803@earthlink.net> <4B372272.10808@behnel.de> <4B39749C.2050203@earthlink.net> <71a02c81a7f80ae6d31e45bf173d172e.squirrel@webmail.uio.no> <4B3BCDD2.9020309@earthlink.net> <4B3C713C.9010801@molden.no> <4B3CA848.6020207@behnel.de> <74e60ecaed82ca83fdd29b8836922b66.squirrel@webmail.uio.no> <4B3DE668.6040401@student.matnat.uio.no> Message-ID: <1262860701.24489.11.camel@mini> On Thu, 2010-01-07 at 01:26 -0800, Robert Bradshaw wrote: > On Jan 2, 2010, at 10:23 PM, Sturla Molden wrote: > > > Dag Sverre Seljebotn wrote: > > > >> In my experience, Cygwin is often used as an > >> easy way out for porting open source software to Windows, and SFU/SUA > >> seem to exclude at least most of the home users and quite a few > >> laptop > >> users. > > > > First, note that Cygwin is GPL unless you buy a commercial license > > (for a > > fee undisclosed by Red Hat). > > > > The Cygwin fork call is not copy-on-write optimized, which makes > > Cygwin > > unsuited for fork-based internet servers. Also, programs that require > > inter-process read-only access to huge memory buffers can avoid using > > shared memory by forking. This trick will not work on Cygwin. > > > > And then there is the security issue. How safe is Cygwin against > > various > > exploits? As far as I can tell, there is no auto-update of system > > components. > > > >> Can one compile gcc for SFU/SUA? Does that have less or more problems > >> than gcc for Cygwin? > > > > gcc is the system C compiler on Interix. Microsoft is for some > > reason not > > using their own C compiler, but rather relying on gcc. Perhaps > > Visual C++ > > did not pass UNIX certification; or perhaps this is a decision to make > > porting from Linux easier. I don't know. But in any case, gcc is > > preinstalled, you can build your own, or download one here: > > > > http://www.suacommunity.com/tool_warehouse.htm > > > > Personally I prefer Sun's VirtualBox (PUEL license, not the GPL > > version) > > with a modern Linux or Unix to Cygwin or Interix. > > > > http://www.virtualbox.org/ > > And I prefer running a modern Linux or Unix natively on my system :). > Virtualbox is a nice option for Windows users when it works though. > > Of course, from the developers point of view, no matter how nice we > set up our own environment of choice, the issue remains that we don't > want to require users to have to upgrade Windows Ultimate to compile/ > run our code. It's not only Windows Ultimate -- it is also e.g. Windows 7 enterprise, which e.g. my entire university and its employees will soon have access to. There's a lot of users on Windows Enterprise. If something was my software which I had a hard time porting to Windows, I wouldn't hesitate to embrace SUA as a first solution, rather than spending many times more time going native or with Cygwin. (I never tried SUA, but the existance of a Gentoo port speaks volumes I think.) Dag Sverre From sturla at molden.no Thu Jan 7 12:10:50 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 7 Jan 2010 12:10:50 +0100 Subject: [Cython] Arrays of Python Objects In-Reply-To: <4B44DCFE.10509@earthlink.net> References: <4B43E1B0.5040908@earthlink.net> <9bca51a9d7736db724f5fe9782d1c829.squirrel@webmail.uio.no> <4B44DCFE.10509@earthlink.net> Message-ID: <3b7bee6acd924a111ad94bd0cf0a601d.squirrel@webmail.uio.no> > It turns out that the tricky part is overriding the allocation and > deallocation of the python objects associated with my extension class, > as well as the extension class structure itself. I don't really understand what you are trying to do, but it seems a bit confused. Why don't you just cache the expensive objects in a dict global to the module? Instead of trapping garbage collection, you can simply avoid that it happens by keeping a reference somewhere (e.g. in a dict). Also don't try to outsmart Python by implementing your own hashing table. Dicts and sets have supernatural performance compared to anything you can make on your own in reasonable time. If you want more precise control over reference counts etc., there are weakrefs, weakvalue dicts and weakref dicts. I used these to manage a pool of share memory segments. It might be similar to what you are trying to do. http://folk.uio.no/sturlamo/python/sharedmem-feb13-2009.zip S.M. From sturla at molden.no Thu Jan 7 12:25:02 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 7 Jan 2010 12:25:02 +0100 Subject: [Cython] [OT] Windows and Unix In-Reply-To: References: <4B2A71F0.1030105@student.matnat.uio.no> <4B2A7CF4.7050006@behnel.de> <473FCA5A-D155-42FC-B1CB-355DEB87B756@math.washington.edu> <4B2B2AF4.10100@behnel.de> <4B2B3154.7060708@behnel.de> <4B3706AF.4040803@earthlink.net> <4B372272.10808@behnel.de> <4B39749C.2050203@earthlink.net> <71a02c81a7f80ae6d31e45bf173d172e.squirrel@webmail.uio.no> <4B3BCDD2.9020309@earthlink.net> <4B3C713C.9010801@molden.no> <4B3CA848.6020207@behnel.de> <74e60ecaed82ca83fdd29b8836922b66.squirrel@webmail.uio.no> <4B3DE668.6040401@student.matnat.uio.no> Message-ID: > On Jan 2, 2010, at 10:23 PM, Sturla Molden wrote: > And I prefer running a modern Linux or Unix natively on my system :). Fair enough, but it is always a PITA to make things compile. :-( On Windows we can rely on nice guys like David Cournapeau to make binary installers for us :-) > Of course, from the developers point of view, no matter how nice we > set up our own environment of choice, the issue remains that we don't > want to require users to have to upgrade Windows Ultimate to compile/ > run our code. I guess that depends on the user. If we are programming for the masses: no. If we are programming for a particular client, we can often require whatever system setup we want. From stefan_ml at behnel.de Thu Jan 7 14:08:16 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 14:08:16 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> Message-ID: <4B45DCC0.2020606@behnel.de> Robert Bradshaw, 07.01.2010 09:54: > On Jan 6, 2010, at 11:18 AM, Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> I think it would be nice to support 'cdef inline' methods in >>> extension types and let Cython call them without the normal method >>> indirection, e.g. >>> >>> cdef class MyType: >>> cdef list large_list >>> cdef size_t y >>> >>> cdef big_method(self): >>> for x in self.large_list: >>> self.small_method(x) >>> >>> cdef inline small_method(self, x): >>> self.y += x >>> >>> would emit a direct call to the ..._small_method(self, x) C function >>> in big_method(), instead of passing through self->__pyx_vtab->small_method. >>> This would require these methods to be non-overridable, which I >>> think makes sense for something declared 'inline'. >>> >> I remember wanting this feature when I first saw Cython (then never >> got around to it). So very much +1 on the feature, not sure about syntax. > > This has been floating around in the back of my head as well, so +1. http://trac.cython.org/cython_trac/ticket/474 Stefan From nludban at osc.edu Thu Jan 7 15:05:02 2010 From: nludban at osc.edu (Neil Ludban) Date: Thu, 7 Jan 2010 09:05:02 -0500 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> Message-ID: <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> >> Stefan Behnel wrote: ... > >>> cdef inline small_method(self, x): >>> self.y += x >>> >>> would emit a direct call to the ..._small_method(self, x) C >>> function in >>> big_method(), instead of passing through self->__pyx_vtab- >>>> small_method. >>> >>> This would require these methods to be non-overridable, which I >>> think makes >>> sense for something declared 'inline'. >>> >>> Thoughts? This obscures the intent of the code by abusing a C keyword, just borrow the Java keyword that already has the associated semantics: cdef final small_method(self, x): ... Add an implicit inline in the C code declaration and emit the direct calls, and it does exactly what new cython users (like me) expect. From sturla at molden.no Thu Jan 7 15:45:37 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 7 Jan 2010 15:45:37 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> Message-ID: <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> > This obscures the intent of the code by abusing a C keyword, just > borrow the Java keyword that already has the associated semantics: > > cdef final small_method(self, x): I think a decorator would be better than a keyword: @cython.nonvirtual cdef void small_method(self, x): pass In a perfect world, Cython should just look like a Python library. Also I agree that the C/C++ semantics of inline should be kept. Or preferably, inline should just be a decorator as well. @cython.nonvirtual @cython.inline cdef void small_method(self, x): pass S.M. From sturla at molden.no Thu Jan 7 15:49:29 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 7 Jan 2010 15:49:29 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B44CA57.2010201@behnel.de> References: <4B44CA57.2010201@behnel.de> Message-ID: > would emit a direct call to the ..._small_method(self, x) C function in > big_method(), instead of passing through self->__pyx_vtab->small_method. Or just declare self->__pyx_vtab->small_method a const pointer, so the C compiler knows it does not change. SM From stefan_ml at behnel.de Thu Jan 7 15:50:39 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jan 2010 15:50:39 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> Message-ID: <4B45F4BF.7060408@behnel.de> Sturla Molden, 07.01.2010 15:45: >> This obscures the intent of the code by abusing a C keyword, just >> borrow the Java keyword that already has the associated semantics: >> >> cdef final small_method(self, x): > > I think a decorator would be better than a keyword: > > @cython.nonvirtual > cdef void small_method(self, x): > pass > > In a perfect world, Cython should just look like a Python library. > > Also I agree that the C/C++ semantics of inline should be kept. Or > preferably, inline should just be a decorator as well. > > @cython.nonvirtual > @cython.inline > cdef void small_method(self, x): > pass But wouldn't '@cython.inline' still imply '@cython.nonvirtual' here? The question is: what use case is there where you would want one without the other? Stefan From sturla at molden.no Thu Jan 7 16:10:29 2010 From: sturla at molden.no (Sturla Molden) Date: Thu, 7 Jan 2010 16:10:29 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B45F4BF.7060408@behnel.de> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> Message-ID: <1ec93c1a76dc2e132a703101a08de66c.squirrel@webmail.uio.no> > The question is: what use case is there where you would want one without > the other? In C++ there is actually something called "inline virtual" methods. It means that it can be inlined if the compiler can infer the type of the class. This can e.g. happen if the compiler sees an object (global or on the stack) rather than a pointer. Sturla From nludban at osc.edu Thu Jan 7 17:37:07 2010 From: nludban at osc.edu (Neil Ludban) Date: Thu, 7 Jan 2010 11:37:07 -0500 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B45F4BF.7060408@behnel.de> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> Message-ID: <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> On Jan 7, 2010, at 9:50 AM, Stefan Behnel wrote: > >> @cython.nonvirtual >> @cython.inline >> cdef void small_method(self, x): >> pass > > But wouldn't '@cython.inline' still imply '@cython.nonvirtual' here? > > The question is: what use case is there where you would want one without > the other? > > Stefan First use case: a super-optimizing cython would look for methods that make many calls to inline non-final methods and generate two code paths depending on whether self is exactly this class. Of course, inline would also be automatically detected. Second use case: other languages use "final" as a software design technique (not an optimization), but that would be adding a new feature to Python, probably not the intended goal. Using "final" still seems much less evil than surprising end users with unexpected (vs C/C++ and Python) semantics for "inline". From nludban at osc.edu Thu Jan 7 17:40:28 2010 From: nludban at osc.edu (Neil Ludban) Date: Thu, 7 Jan 2010 11:40:28 -0500 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> Message-ID: Forgot to ask, what if I wanted to add final/inline/whatever to the entire class? On Jan 7, 2010, at 11:37 AM, Neil Ludban wrote: > On Jan 7, 2010, at 9:50 AM, Stefan Behnel wrote: >> >>> @cython.nonvirtual >>> @cython.inline >>> cdef void small_method(self, x): >>> pass >> >> But wouldn't '@cython.inline' still imply '@cython.nonvirtual' here? >> >> The question is: what use case is there where you would want one without >> the other? >> >> Stefan > > First use case: a super-optimizing cython would look for methods > that make many calls to inline non-final methods and generate two > code paths depending on whether self is exactly this class. Of > course, inline would also be automatically detected. > > Second use case: other languages use "final" as a software design > technique (not an optimization), but that would be adding a new > feature to Python, probably not the intended goal. > > Using "final" still seems much less evil than surprising end users > with unexpected (vs C/C++ and Python) semantics for "inline". From mattcbro at earthlink.net Thu Jan 7 17:59:23 2010 From: mattcbro at earthlink.net (Matthew) Date: Thu, 7 Jan 2010 11:59:23 -0500 (GMT-05:00) Subject: [Cython] Arrays of Python Objects Message-ID: <27647438.1262883563373.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> Well I guess I outsmarted myself on this one. After implementing my object hasher using dict, my code slowed down by nearly a factor of 10 LOL. The hashing code itself shows up as a minimal contribution to the overall time in the profiler. However it does require a little extra logic in the initializer and a copy of a few pointers. Basically I copy all the C device pointers to a new cython class which then get's saved in a dictionary upon the deallocation of a matrix object. If that object exists in the dictionary it is given to the allocator instead of allocating a new block. I'm wondering just what kind of overhead Im up against with regards to the python initialization of my cython defined class? This almost suggests that the overhead problem is actually in python/cython and not just in my C code. -Matt Robert Bradshaw writes: > > On Jan 7, 2010, at 12:05 AM, Stefan Behnel wrote: > > > Matthew Bromberg, 06.01.2010 21:50: > >> How does tuple or list compare speed wise with dict? > > > > Like apples and oranges, basically. > > If you're trying to index into it with an int, especially a c int, > lists and tuples will be much faster. > > >> Ultimately I have to hash into my list using size information. > > > > Any specific reason why you /can't/ use a dict? > > Which will likely be just as fast, if not faster, than hashing into a > list manually yourself. > > >> This also still does not address my confusion with regards to how to > >> capture a python object before it get's destroyed. > > > > As long as there is a reference to it (e.g. in the hash table), it > > won't > > get deallocated. So: use a Python list for your hash table, stop > > caring > > about ref-counts and it will just work. > > +1. Ideally, you should never have to worry about reference counts > when working with Cython at all. > > I'm still not quite sure exactly what you're trying to do, but if it's > creating and deleting thousands of these objects a second and that's > killing you (the actual allocation/deallocation, not the > initialization) then what you might want to do is something like > > http://hg.sagemath.org/sage-main/file/21efb0b3fc47/sage/rings/real_double.pyx#l2260 > > which is a bit hackish and will probably need to be adapted to your > specific case. If initializing is expensive, than you can probably > keep around a pool of initalized pointers/buffers/whatever, and have > the object creation just set/unset these fields (much cleaner). > > - Robert > From robertwb at math.washington.edu Thu Jan 7 21:06:32 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jan 2010 12:06:32 -0800 Subject: [Cython] Arrays of Python Objects In-Reply-To: <27647438.1262883563373.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> References: <27647438.1262883563373.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> Message-ID: <5F33C150-762B-4875-BA0E-DD30F8EC3625@math.washington.edu> On Jan 7, 2010, at 8:59 AM, Matthew wrote: > Well I guess I outsmarted myself on this one. After implementing my > object hasher using dict, > my code slowed down by nearly a factor of 10 LOL. > > The hashing code itself shows up as a minimal contribution to the > overall time in the profiler. > However it does require a little extra logic in the initializer and > a copy of a few pointers. > > Basically I copy all the C device pointers to a new cython class > which then get's saved in a dictionary upon the deallocation > of a matrix object. If that object exists in the dictionary it is > given to the allocator instead of allocating a new block. > > I'm wondering just what kind of overhead Im up against with regards > to the python initialization of my cython defined class? > This almost suggests that the overhead problem is actually in python/ > cython and not just in my C code. This could probably be discovered via some profiling. If this is the case, take a look at http://trac.cython.org/cython_trac/ticket/238 (the modern version of the former PY_NEW trick). Maybe you're passing lots of (keyword?) arguments? > > -Matt > > Robert Bradshaw writes: > >> >> On Jan 7, 2010, at 12:05 AM, Stefan Behnel wrote: >> >>> Matthew Bromberg, 06.01.2010 21:50: >>>> How does tuple or list compare speed wise with dict? >>> >>> Like apples and oranges, basically. >> >> If you're trying to index into it with an int, especially a c int, >> lists and tuples will be much faster. >> >>>> Ultimately I have to hash into my list using size information. >>> >>> Any specific reason why you /can't/ use a dict? >> >> Which will likely be just as fast, if not faster, than hashing into a >> list manually yourself. >> >>>> This also still does not address my confusion with regards to how >>>> to >>>> capture a python object before it get's destroyed. >>> >>> As long as there is a reference to it (e.g. in the hash table), it >>> won't >>> get deallocated. So: use a Python list for your hash table, stop >>> caring >>> about ref-counts and it will just work. >> >> +1. Ideally, you should never have to worry about reference counts >> when working with Cython at all. >> >> I'm still not quite sure exactly what you're trying to do, but if >> it's >> creating and deleting thousands of these objects a second and that's >> killing you (the actual allocation/deallocation, not the >> initialization) then what you might want to do is something like >> >> http://hg.sagemath.org/sage-main/file/21efb0b3fc47/sage/rings/real_double.pyx#l2260 >> >> which is a bit hackish and will probably need to be adapted to your >> specific case. If initializing is expensive, than you can probably >> keep around a pool of initalized pointers/buffers/whatever, and have >> the object creation just set/unset these fields (much cleaner). >> >> - Robert >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Thu Jan 7 21:08:15 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jan 2010 12:08:15 -0800 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> Message-ID: <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> On Jan 7, 2010, at 8:40 AM, Neil Ludban wrote: > Forgot to ask, what if I wanted to add final/inline/whatever to the > entire class? I don't think it's possible to prevent a class from being subclassed in Python-land. (Well, not without checking raising some kind of a runtime exception on instantiation or something like that...) - Robert From nludban at osc.edu Thu Jan 7 21:34:53 2010 From: nludban at osc.edu (Neil Ludban) Date: Thu, 7 Jan 2010 15:34:53 -0500 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> Message-ID: <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> On Jan 7, 2010, at 3:08 PM, Robert Bradshaw wrote: > On Jan 7, 2010, at 8:40 AM, Neil Ludban wrote: > >> Forgot to ask, what if I wanted to add final/inline/whatever to the >> entire class? > > I don't think it's possible to prevent a class from being subclassed > in Python-land. (Well, not without checking raising some kind of a > runtime exception on instantiation or something like that...) > > - Robert Classes defined in extensions must explicitly indicate that they support subclassing from pure Python. NoneType and bool are core types that don't, there may be others. The intended point was that the semantics of making a class "final" are understood and would imply the requested optimizations, while the "inline" keyword wouldn't fit at all in this context. From robertwb at math.washington.edu Thu Jan 7 21:41:44 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jan 2010 12:41:44 -0800 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: References: <4B44CA57.2010201@behnel.de> Message-ID: On Jan 7, 2010, at 6:49 AM, Sturla Molden wrote: > >> would emit a direct call to the ..._small_method(self, x) C >> function in >> big_method(), instead of passing through self->__pyx_vtab- >> >small_method. > > Or just declare self->__pyx_vtab->small_method a const pointer, so > the C > compiler knows it does not change. That might help, but I don't think either of these indirections could be resolved at compile time, even if the latter could be declared const. - Robert From robertwb at math.washington.edu Thu Jan 7 22:06:33 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jan 2010 13:06:33 -0800 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> Message-ID: <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> On Jan 7, 2010, at 12:34 PM, Neil Ludban wrote: > On Jan 7, 2010, at 3:08 PM, Robert Bradshaw wrote: >> On Jan 7, 2010, at 8:40 AM, Neil Ludban wrote: >> >>> Forgot to ask, what if I wanted to add final/inline/whatever to the >>> entire class? >> >> I don't think it's possible to prevent a class from being subclassed >> in Python-land. (Well, not without checking raising some kind of a >> runtime exception on instantiation or something like that...) >> >> - Robert > > Classes defined in extensions must explicitly indicate that they > support subclassing from pure Python. NoneType and bool are core > types that don't, there may be others. Ah, I did not know that. That does make a final modifier more attractive. This is a larger change to the language than allowing inline methods (with an error if one attempts to override them). > The intended point was that the semantics of making a class "final" > are understood and would imply the requested optimizations, while > the "inline" keyword wouldn't fit at all in this context. I think it depends on the perspective you're coming from. Is the intent to have the method inlined, which implies final as a consequence, or is the intent to have the method final, with possible inlining as an optimization detail? The inline keyword is nice because we already use it for functions (where the final keyword makes no sense). I think it's probably worth introducing both keywords, where inline would control emission of the C inline directive. (Of course for non-inline final methods, the C compiler could decide whether or not to actually inline.) A class could be final, with the same semantics as java. One question is would inline imply final, or require it? On another note, I assume inline methods could be placed in pxd files if desired as well? As for decorators vs. inline (pun intended) function modifiers, we should certainly support both. - Robert From Chris.Barker at noaa.gov Thu Jan 7 22:52:05 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 07 Jan 2010 13:52:05 -0800 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> Message-ID: <4B465785.7080706@noaa.gov> Robert Bradshaw wrote: > I think it depends on the perspective you're coming from. Is the > intent to have the method inlined, which implies final as a > consequence, I think that that's the only good reason -- this is Python (with static optimization), after all. or would "final" provide some optimization? > A class could be final What is the benefit of that? Again, this is Python after all! (though I suppose the NoneType and bool examples prove the need) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mattcbro at earthlink.net Thu Jan 7 23:04:23 2010 From: mattcbro at earthlink.net (Matthew) Date: Thu, 7 Jan 2010 17:04:23 -0500 (EST) Subject: [Cython] Arrays of Python Objects Message-ID: <30350529.1262901863739.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> OK I'm going to give the __new__ hack a try from http://trac.cython.org/cython_trac/ticket/238 I don't really need to overload __new__ do I, so I don't have to change matrix.pxd? Just for completeness here is a recent profile on a very fast machine. (All the other machines take 10 times as long. Hmm they weren't running cython 0.12 either but 0.11.3) 1155781 function calls in 11.978 CPU seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 107520 2.928 0.000 3.009 0.000 matrix.pyx:569(__cinit__) 1024 2.223 0.002 2.632 0.003 matrix.pyx:1144(qr) 107516 2.009 0.000 2.091 0.000 matrix.pyx:604(__dealloc__) 40640 1.536 0.000 1.536 0.000 matrix.pyx:83(__dealloc__) 40640 1.170 0.000 1.176 0.000 matrix.pyx:37(__cinit__) 9920 0.337 0.000 0.394 0.000 matrix.pyx:1361(tomat) 3584 0.193 0.000 4.395 0.001 muddect.py:92(sigdelay) 19008 0.127 0.000 0.331 0.000 matrix.pyx:897(__mul__) 1792 0.076 0.000 3.669 0.002 muddect.py:151(despread) ### The calls to the hash table implemented via python dict are below, almost two order of magnitudes less expensive than the ### matrix.pyx cinit in line 1. #################### 90304 0.074 0.000 0.074 0.000 memcache.pyx:56(hashcget) 91260 0.065 0.000 0.065 0.000 memcache.pyx:19(__cinit__) 3584 0.056 0.000 0.285 0.000 matrix.pyx:1050(fft) 3584 0.055 0.000 1.429 0.000 matrix.pyx:344(expj) 3584 0.054 0.000 0.285 0.000 matrix.pyx:1071(ifft) 7168 0.052 0.000 0.318 0.000 matrix.pyx:1324(r2cmat) 7168 0.052 0.000 0.620 0.000 matrix.pyx:861(__and__) ... To give some perspective, my matlab code performing the same operation, is running on octave on a single cpu and takes about the same amount of time (13 seconds.). This cython application is farmed out to a GPU with at least 240 processesors on it. I'd like to see well over an order of magnitude improvement. With a few exceptions the lions share of the cython processing is being sucked away on lousy memory allocations. Even my qr decomposition is also getting killed by allocations and memory transfers I believe. Here is the __cinit__() and dealloc that dominates the time: cdef class cmat : """ Real matrix class binds to both vsipl and numpy ndarray. The ndarray object is returned via getarr. The array is bound back to vsipl via admit """ def __cinit__(self, int r, int c): cdef vsip_cblock_f *myb cdef int istrue cdef np.ndarray[cfloat, ndim=2] carr cdef tuple t = (r,c) cdef bool isinhash self.parent = None # assumes not a submatrix if r <= 0 : self.vm = NULL return isinhash = hashcget( r, c, self) if isinhash : # already done loading return # print "create an nd array float buffer" carr = np.empty(shape=(r,c),dtype=np.complex64,order='Fortran') # print "assigning to arr" self.arr = carr # print " bind the ndarray to a vsipl block" myb = vsip_cblockbind_f( carr.data, NULL, r * c, VSIP_MEM_NONE) if myb == NULL: raise MemoryError("Could not allocate real matrix block") # print " admit the data so that vsipl owns it" istrue = vsip_cblockadmit_f( myb, 0) # create a view self.vm = vsip_cmbind_f( myb, 0, 1, r, r, c) if self.vm == NULL: raise MemoryError("Could not allocate real matrix view") self.fftdata = NULL def __dealloc__(self): # print "freeing vm " cdef hashcmat hc cdef bool isin = True if (self.vm) : if self.parent is None : # add this to the cache if it isn't already hc = hashcmat(self) if not hc.isin : # saving it in the hash instead return vsip_cmalldestroy_f(self.vm) else : vsip_cmdestroy_f(self.vm) # print "freeing fftdata" if (self.fftdata) : vsip_fft_destroy_f(self.fftdata) The vsipl vendor tells me that the only real expensive operation is the cblockbind() within __cinit__(). However the __dealloc()__ routine is very expensive as well (Given the number of times it's being called). It would be nice if I could profile on a line by line basis. I'm not sure if the python cProfile tool supports this or not. I can't just chalk up this result to the vsipl code, since the hash routine is not giving me any performance gain and seemed to be making things worse. (Though I probably need to do some more debugging to see if I have a lot of cache misses,or some bug in my logic.) For the life of me I could not figure out how to just put the matrix object itself into my hash indexed memory cache. It seemed like my python objects were always being garbage collected once I hit the __dealloc__ routine (the self.arr ndarray as an example). Later I found out that the cython class get's stripped of it's attributes if it's stored in a dictionary. Only those attributes written to the classes internal dictionary in the __init__() method seem to get saved, as far as I can tell from my experiments. Of course I'd actually like to avoid calling __init__(). I really didn't intend to try to learn the internals of python or cython for that matter, but I do need to figure out how to optimize this code. Regards, -Matt Robert Bradshaw writes: > > On Jan 7, 2010, at 8:59 AM, Matthew wrote: > > > Well I guess I outsmarted myself on this one. After implementing my > > object hasher using dict, > > my code slowed down by nearly a factor of 10 LOL. > > > > The hashing code itself shows up as a minimal contribution to the > > overall time in the profiler. > > However it does require a little extra logic in the initializer and > > a copy of a few pointers. > > > > Basically I copy all the C device pointers to a new cython class > > which then get's saved in a dictionary upon the deallocation > > of a matrix object. If that object exists in the dictionary it is > > given to the allocator instead of allocating a new block. > > > > I'm wondering just what kind of overhead Im up against with regards > > to the python initialization of my cython defined class? > > This almost suggests that the overhead problem is actually in python/ > > cython and not just in my C code. > > This could probably be discovered via some profiling. If this is the > case, take a look at > > http://trac.cython.org/cython_trac/ticket/238 (the modern version of > the former PY_NEW trick). Maybe you're passing lots of (keyword?) > arguments? > > > > > -Matt > > > > Robert Bradshaw writes: > > > >> > >> On Jan 7, 2010, at 12:05 AM, Stefan Behnel wrote: > >> > >>> Matthew Bromberg, 06.01.2010 21:50: > >>>> How does tuple or list compare speed wise with dict? > >>> > >>> Like apples and oranges, basically. > >> > >> If you're trying to index into it with an int, especially a c int, > >> lists and tuples will be much faster. > >> > >>>> Ultimately I have to hash into my list using size information. > >>> > >>> Any specific reason why you /can't/ use a dict? > >> > >> Which will likely be just as fast, if not faster, than hashing into a > >> list manually yourself. > >> > >>>> This also still does not address my confusion with regards to how > >>>> to > >>>> capture a python object before it get's destroyed. > >>> > >>> As long as there is a reference to it (e.g. in the hash table), it > >>> won't > >>> get deallocated. So: use a Python list for your hash table, stop > >>> caring > >>> about ref-counts and it will just work. > >> > >> +1. Ideally, you should never have to worry about reference counts > >> when working with Cython at all. > >> > >> I'm still not quite sure exactly what you're trying to do, but if > >> it's > >> creating and deleting thousands of these objects a second and that's > >> killing you (the actual allocation/deallocation, not the > >> initialization) then what you might want to do is something like > >> > >> http://hg.sagemath.org/sage-main/file/21efb0b3fc47/sage/rings/real_double.pyx#l2260 > >> > >> which is a bit hackish and will probably need to be adapted to your > >> specific case. If initializing is expensive, than you can probably > >> keep around a pool of initalized pointers/buffers/whatever, and have > >> the object creation just set/unset these fields (much cleaner). > >> > >> - Robert > >> > > _______________________________________________ > > Cython-dev mailing list > > Cython-dev at ... > > http://codespeak.net/mailman/listinfo/cython-dev > > From stefan_ml at behnel.de Fri Jan 8 07:58:37 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 08 Jan 2010 07:58:37 +0100 Subject: [Cython] Arrays of Python Objects In-Reply-To: <30350529.1262901863739.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> References: <30350529.1262901863739.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> Message-ID: <4B46D79D.5010500@behnel.de> Matthew, 07.01.2010 23:04: > It would be nice if I could profile on a line by line basis. I'm not > sure if the python cProfile tool supports this or not. You can always extract some interesting sections to an inline function. > Though I probably need to do some more debugging to see if I have a lot > of cache misses,or some bug in my logic. Yes, I'd certainly investigate that first. Try to find out if the caching works as expected. > For the life of me I could not figure out how to just put the matrix > object itself into my hash indexed memory cache. It seemed like my python > objects were always being garbage collected once I hit the __dealloc__ > routine (the self.arr ndarray as an example). __dealloc__ is part of the deallocation process, i.e. it is already clear at the point of calling it that the object is no longer referenced and can be discarded. Also, at least during a garbage collector run, the Python attributes of the object will already have been cleared before calling __dealloc__, so you can't use them anymore. Only C type attributes are safe to access here. Read the Cython and CPython docs for that. > Later I found out that the > cython class get's stripped of it's attributes if it's stored in a > dictionary. Certainly not, but you didn't show us your hashing code, so I can't give a better comment on why you might think this is the case. Stefan From robertwb at math.washington.edu Fri Jan 8 20:07:49 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 8 Jan 2010 11:07:49 -0800 Subject: [Cython] Arrays of Python Objects In-Reply-To: <30350529.1262901863739.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> References: <30350529.1262901863739.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> Message-ID: <26E46043-27CC-462C-8CF5-46BBBCA48170@math.washington.edu> On Jan 7, 2010, at 2:04 PM, Matthew wrote: > OK I'm going to give the __new__ hack a try from > http://trac.cython.org/cython_trac/ticket/238 > I don't really need to overload __new__ do I, so I don't have to > change matrix.pxd? Sorry, this is the ticket number that I meant to refer to: http://trac.cython.org/cython_trac/ticket/443 , though this takes no arguments, so may not apply to you. > The vsipl vendor tells me that the only real expensive operation is > the cblockbind() within __cinit__(). However the __dealloc()__ > routine > is very expensive as well (Given the number of times it's being > called). It would be nice if I could profile on a line by > line basis. I'm not sure if the python cProfile tool supports this > or not. It does, but we don't have that implemented in Cython yet. Given that it's a deterministic rather than (external) probabilistic profiler, the profiling itself may significantly impact the speed and results. Try commenting stuff out, or factoring it into an (inline) function. > I can't just chalk up this result to the vsipl code, since the hash > routine is not giving me any performance gain and seemed to be > making things worse. (Though I probably need to do some more > debugging to see if I have a lot of cache misses,or some bug in my > logic.) Wel, maybe hashing is slightly more expensive than the vsipl call. On a completely unrelated note, getting data to/from a GPU can be a bottleneck as well, and due to its asynchronous nature may not show up as obviously in the main CPU profiling results. > For the life of me I could not figure out how to just put the matrix > object itself into my hash indexed memory cache. It seemed like my > python objects were always being garbage collected once I hit the > __dealloc__ routine (the self.arr ndarray as an example). Later I > found out that the cython class get's stripped of it's attributes if > it's stored in a dictionary. Only those attributes written to the > classes internal dictionary in the __init__() method seem to get > saved, as far as I can tell from my experiments. Of course I'd > actually like to avoid calling __init__(). I really didn't intend > to try to learn the internals of python or cython for that matter, > but I do need to figure > out how to optimize this code. I think you're trying to make things way more complicated than necessary. The easiest approach is to only expose wrapper classes, and cache the expensive initalization in an internal class. See http://sage.math.washington.edu/home/robertwb/cython/mat.html http://sage.math.washington.edu/home/robertwb/cython/mat.pyx (I'm sure there's some more room for optimization, and the caching algorithm could be improved as well.) Also, note that creating the numpy arrays is expensive as well. In [1]: from mat import * In [2]: %time make_np(10**5) CPU times: user 0.56 s, sys: 0.43 s, total: 0.99 s Wall time: 0.99 s In [4]: %time make_CMat(10**5) CPU times: user 0.68 s, sys: 0.45 s, total: 1.13 s Wall time: 1.14 s In [6]: %time make_CachedCMat(10**5) CPU times: user 0.14 s, sys: 0.00 s, total: 0.14 s Wall time: 0.14 s In [8]: %time make_Empty(10**5) CPU times: user 0.02 s, sys: 0.00 s, total: 0.02 s Wall time: 0.02 s - Robert From mattcbro at earthlink.net Sat Jan 9 01:59:39 2010 From: mattcbro at earthlink.net (Matthew Bromberg) Date: Fri, 08 Jan 2010 19:59:39 -0500 Subject: [Cython] Arrays of Python Objects In-Reply-To: <26E46043-27CC-462C-8CF5-46BBBCA48170@math.washington.edu> References: <30350529.1262901863739.JavaMail.root@elwamui-ovcar.atl.sa.earthlink.net> <26E46043-27CC-462C-8CF5-46BBBCA48170@math.washington.edu> Message-ID: <4B47D4FB.7020507@earthlink.net> Thanks for the mini example and timings. Very instructive. By making CMat an object encapsulated in CachedCMat you avoided the 'self destruct' problem, in that the self object is doomed once it reaches __dealloc__. The only annoying thing about that is that one has to carry around CachedCMat along with the CMat object when it's being used or possibly rewrite the CMat class itself to do the extra indirection when you are doing operations such as add, multiply, qr etc. The only Python object in my cmat class is an ndarray, so just copying a reference to that object and a couple of C pointers to a container class in the hash presumably shouldn't be that bad and accomplishes nearly the same thing. It does create the overhead of creating the container class whenever a cmat get's hashed. However my cython timings seem to suggest that that's not such a big deal. I did some more experiments by the way and discovered that my probability of cache miss was 38%. That's still too high. I either need to cache multiple matrices of the same size or move to a different memory management scheme. I've used a circular buffer in the past for temp matrices somewhat successfully, but you have to size your buffer pretty well to avoid overwriting live objects. I'm not certain but it may be that cython 0.12 runs the code a bit faster as well. More tests on Monday I guess. -Matt On 1/8/2010 2:07 PM, Robert Bradshaw wrote: > On Jan 7, 2010, at 2:04 PM, Matthew wrote: > >> OK I'm going to give the __new__ hack a try from >> http://trac.cython.org/cython_trac/ticket/238 >> I don't really need to overload __new__ do I, so I don't have to >> change matrix.pxd? > > Sorry, this is the ticket number that I meant to refer to: > http://trac.cython.org/cython_trac/ticket/443 , though this takes no > arguments, so may not apply to you. > > >> The vsipl vendor tells me that the only real expensive operation is >> the cblockbind() within __cinit__(). However the __dealloc()__ routine >> is very expensive as well (Given the number of times it's being >> called). It would be nice if I could profile on a line by >> line basis. I'm not sure if the python cProfile tool supports this >> or not. > > It does, but we don't have that implemented in Cython yet. Given that > it's a deterministic rather than (external) probabilistic profiler, > the profiling itself may significantly impact the speed and results. > Try commenting stuff out, or factoring it into an (inline) function. > >> I can't just chalk up this result to the vsipl code, since the hash >> routine is not giving me any performance gain and seemed to be making >> things worse. (Though I probably need to do some more debugging to >> see if I have a lot of cache misses,or some bug in my logic.) > > Wel, maybe hashing is slightly more expensive than the vsipl call. On > a completely unrelated note, getting data to/from a GPU can be a > bottleneck as well, and due to its asynchronous nature may not show up > as obviously in the main CPU profiling results. > >> For the life of me I could not figure out how to just put the matrix >> object itself into my hash indexed memory cache. It seemed like my >> python objects were always being garbage collected once I hit the >> __dealloc__ routine (the self.arr ndarray as an example). Later I >> found out that the cython class get's stripped of it's attributes if >> it's stored in a dictionary. Only those attributes written to the >> classes internal dictionary in the __init__() method seem to get >> saved, as far as I can tell from my experiments. Of course I'd >> actually like to avoid calling __init__(). I really didn't intend to >> try to learn the internals of python or cython for that matter, but >> I do need to figure >> out how to optimize this code. > > I think you're trying to make things way more complicated than > necessary. The easiest approach is to only expose wrapper classes, and > cache the expensive initalization in an internal class. See > > http://sage.math.washington.edu/home/robertwb/cython/mat.html > http://sage.math.washington.edu/home/robertwb/cython/mat.pyx > > (I'm sure there's some more room for optimization, and the caching > algorithm could be improved as well.) Also, note that creating the > numpy arrays is expensive as well. > > In [1]: from mat import * > > In [2]: %time make_np(10**5) > CPU times: user 0.56 s, sys: 0.43 s, total: 0.99 s > Wall time: 0.99 s > > In [4]: %time make_CMat(10**5) > CPU times: user 0.68 s, sys: 0.45 s, total: 1.13 s > Wall time: 1.14 s > > In [6]: %time make_CachedCMat(10**5) > CPU times: user 0.14 s, sys: 0.00 s, total: 0.14 s > Wall time: 0.14 s > > In [8]: %time make_Empty(10**5) > CPU times: user 0.02 s, sys: 0.00 s, total: 0.02 s > Wall time: 0.02 s > > - Robert > > > > No virus found in this incoming message. > Checked by AVG - www.avg.com > Version: 9.0.725 / Virus Database: 270.14.130/2607 - Release Date: 01/08/10 02:35:00 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100108/53c9a3d0/attachment.htm From robertwb at math.washington.edu Sat Jan 9 19:52:41 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 9 Jan 2010 10:52:41 -0800 Subject: [Cython] Cython 0.12 not working on Windows XP In-Reply-To: <4B39749C.2050203@earthlink.net> References: <4B2A71F0.1030105@student.matnat.uio.no> <4B2A7CF4.7050006@behnel.de> <473FCA5A-D155-42FC-B1CB-355DEB87B756@math.washington.edu> <4B2B2AF4.10100@behnel.de> <4B2B3154.7060708@behnel.de> <4B3706AF.4040803@earthlink.net> <4B372272.10808@behnel.de> <4B39749C.2050203@earthlink.net> Message-ID: <8678E0FC-ED4D-4512-A84D-D7A87A46FB4C@math.washington.edu> On Dec 28, 2009, at 7:16 PM, Matthew Bromberg wrote: >>> >>> >>> test.c >>> test.c(812) : error C2143: syntax error : missing ';' before 'type' >>> test.c(812) : error C2275: 'PyFrameObject' : illegal use of this >>> type >>> as an expression >>> C:\python26\include\frameobject.h(47) : see declaration of >>> 'PyFrameObject' >>> [...] >>> Finally I show the relevant section of test.c >>> >>> --------------- test.c ----------------------------- >>> >>> /* "E:\Projects\MUD\test.pyx":19 >>> * cdef class rmat : >>> * >>> * def __cinit__(self, int r, int c): #<<<<<<<<<<<<<< >>> * """ Generate a real matrix of r rows and c cols >>> unitialized. """ >>> * cdef vsip_block_f *myb >>> */ >>> >>> static int __pyx_pf_4test_4rmat___cinit__(PyObject *__pyx_v_self, >>> PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/ >>> static char __pyx_doc_4test_4rmat___cinit__[] = " Generate a real >>> matrix of r rows and c cols unitialized. "; >>> static int __pyx_pf_4test_4rmat___cinit__(PyObject *__pyx_v_self, >>> PyObject *__pyx_args, PyObject *__pyx_kwds) { >>> int __pyx_v_r; >>> int __pyx_v_c; >>> int __pyx_r; >>> static PyObject **__pyx_pyargnames[] = >>> {&__pyx_n_s__r,&__pyx_n_s__c,0}; >>> __Pyx_RefNannySetupContext("__cinit__"); >>> __Pyx_TraceCall("__cinit__", __pyx_f[0], 19); /* This is line 812 >>> */ >>> if (unlikely(__pyx_kwds)) { >>> >>> >>> It looks like a bug to me. Apparently C2143 has a couple hundred meanings in MSVC I think I've figured out what's going on. When the refnanny is turned off , __Pyx_RefNannySetupContext("__cinit__"); gets expanded to the empty line ; rather than a declaration, so the declarations in __Pyx_TraceCall come after "code" making MSVC reject them. (This is why all the tests run fine). I moved the declarations out--see http://hg.cython.org/cython-devel/rev/3e685896dd7b . Try the latest cython-devel to see if that resolves your issue. - Robert From stefan_ml at behnel.de Sat Jan 9 21:06:18 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 09 Jan 2010 21:06:18 +0100 Subject: [Cython] Support for borrowed references Message-ID: <4B48E1BA.7000304@behnel.de> Hi, I would like to get support for borrowed references in Cython. My use case is a package I recently wrote, which uses an automaton (DFA) for multi-keyword text search in a unicode string. The automaton was originally modeled as a set of state objects that basically contain a (non-dict) map from characters to subsequent state objects. The transformation is currently written in Python. The search engine itself, written in Cython, is rather straight forward. It starts with a reference to one state and jumps from one state to the next while reading the sequence of input characters. The engine doesn't create any new objects until a match was found. All it does is update a reference to the current state. If that reference was an unmanaged reference, the whole engine could run without requiring the GIL. However, since it's not, Cython requires the GIL to update the reference for each new input character. (Note that a PyObject* won't work here as this would prevent the code from accessing the state's attributes). I ended up rewriting the engine to use a struct instead, which added quite a bit to the LOC count and also a bit to the memory footprint (due to duplicated pointers). The code was nice and simple before that, now it's still somewhat short, but it clearly became less beautiful. I'm not sure yet what would be needed to support borrowed references, but I don't think it's trivial. There was an older discussion about borrowed references on cython-dev: http://comments.gmane.org/gmane.comp.python.cython.devel/6864 However, that only dealt with stolen function arguments and borrowed return values. My use case above makes me believe that it would be just as useful for local variables and (potentially) object attributes. So you could write cdef borrowed object borrowed_ref and Cython would disable ref-counting for "borrowed_ref", i.e. borrowed_ref = some_value # no incref some_normal_var = borrowed_ref # normal incref However, this becomes problematic when a new reference is (accidentally?) assigned to the variable, e.g. borrowed_ref = [] The above could raise an error at compile time, and I actually think that we could use the same mechanism as for e.g. bytes->char* conversions of temporary values to detect incorrect code. Also, functions could be required to declare their return values as "borrowed" to allow such an assignment. That would provide a reasonable level of safety IMO. Comments? Stefan From stefan_ml at behnel.de Sun Jan 10 18:23:50 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 10 Jan 2010 18:23:50 +0100 Subject: [Cython] Support for borrowed references In-Reply-To: <4B48E1BA.7000304@behnel.de> References: <4B48E1BA.7000304@behnel.de> Message-ID: <4B4A0D26.7070807@behnel.de> Stefan Behnel, 09.01.2010 21:06: > borrowed_ref = [] > > The above could raise an error at compile time, and I actually think that > we could use the same mechanism as for e.g. bytes->char* conversions of > temporary values to detect incorrect code. Thinking about this some more, why not just represent a borrowed reference as a pointer? So you could write cdef list some_list = [] cdef list* borrowed_ref = some_list and borrowed_ref would be a non-refcounted pointer to a Python list. Assignments back to a normal reference would be allowed: cdef list my_list = borrowed_ref # increfs the pointer After all, a non-refcounted reference to a Python object is not more than a bare pointer to a well-defined Python builtin/extension type (including "object*"). Stefan From stefan_ml at behnel.de Mon Jan 11 10:00:05 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jan 2010 10:00:05 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> Message-ID: <4B4AE895.6020804@behnel.de> Robert Bradshaw, 07.01.2010 22:06: > On Jan 7, 2010, at 12:34 PM, Neil Ludban wrote: >> Classes defined in extensions must explicitly indicate that they >> support subclassing from pure Python. NoneType and bool are core >> types that don't, there may be others. > > Ah, I did not know that. That does make a final modifier more > attractive. This is a larger change to the language than allowing > inline methods (with an error if one attempts to override them). I'm +0 on 'final' classes. I don't see a major use case, although there may be. Some extension types are really just meant as containers for C types, and when they are created from module internal code only, there may not be much value in subtyping them (although lxml.etree makes a good counter example). Anyway, if it's supported by CPython, it should be ok to support it in Cython, I guess, as long as users are aware that this really means that they put a serious restriction on the usage of their code, and that this should be used with caution. I certainly have seen enough Java code where 'final' was getting in the way at some point (and wasn't easy to remove from externally maintained code), or where the keyword was used totally arbitrarily in places where the compiler can easily infer the semantics anyway, and where it just adds noise to the code. I always expect an important intention when I see this keyword in code, and when there isn't one, it tends to confuse me, because it doesn't give me the feeling that I have understood the code. >> The intended point was that the semantics of making a class "final" >> are understood and would imply the requested optimizations, while >> the "inline" keyword wouldn't fit at all in this context. IMHO, "inline" fits exactly the intended optimisation, whereas "final" doesn't imply (and certainly not request) any optimisations at all. > The inline keyword is nice because we already use it for functions > (where the final keyword makes no sense). I think it's probably worth > introducing both keywords, where inline would control emission of the > C inline directive. (Of course for non-inline final methods, the C > compiler could decide whether or not to actually inline.) A class > could be final, with the same semantics as java. One question is would > inline imply final, or require it? It makes no sense without final, so it should imply it (and certainly not require it). > On another note, I assume inline > methods could be placed in pxd files if desired as well? I guess that would be a suitable extension to this feature, yes. > As for decorators vs. inline (pun intended) function modifiers, we > should certainly support both. Personally, I think adding new keywords is bad. I hope that 'final' would be used rarely enough not to require a new inline keyword here. A decorator would suite the intention perfectly. Stefan From dagss at student.matnat.uio.no Mon Jan 11 12:37:42 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 Jan 2010 12:37:42 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B4AE895.6020804@behnel.de> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> Message-ID: <4B4B0D86.20405@student.matnat.uio.no> Stefan Behnel wrote: > Robert Bradshaw, 07.01.2010 22:06: >> The inline keyword is nice because we already use it for functions >> (where the final keyword makes no sense). I think it's probably worth >> introducing both keywords, where inline would control emission of the >> C inline directive. (Of course for non-inline final methods, the C >> compiler could decide whether or not to actually inline.) A class >> could be final, with the same semantics as java. One question is would >> inline imply final, or require it? > > It makes no sense without final, so it should imply it (and certainly not > require it). Actually, in code like this a = MyClass() a.foo() the compiler would be able (in theory) to infer the exact type of "a" and treat any method calls on it as final (skip vtable), and then whether INLINE is declared or not C-side is an orthogonal issue. I.e. "cdef inline foo(self)" could mean "allow overriding the method, but make it inline in situations where the vtable can be skipped". Not that I think there's a real use for this... -- Dag Sverre From stefan_ml at behnel.de Mon Jan 11 13:48:32 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Jan 2010 13:48:32 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B4B0D86.20405@student.matnat.uio.no> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <4B4B0D86.20405@student.matnat.uio.no> Message-ID: <4B4B1E20.3090406@behnel.de> Dag Sverre Seljebotn, 11.01.2010 12:37: > Stefan Behnel wrote: >> Robert Bradshaw, 07.01.2010 22:06: >>> The inline keyword is nice because we already use it for functions >>> (where the final keyword makes no sense). I think it's probably worth >>> introducing both keywords, where inline would control emission of the >>> C inline directive. (Of course for non-inline final methods, the C >>> compiler could decide whether or not to actually inline.) A class >>> could be final, with the same semantics as java. One question is would >>> inline imply final, or require it? >> It makes no sense without final, so it should imply it (and certainly not >> require it). > > Actually, in code like this > > a = MyClass() > a.foo() > > the compiler would be able (in theory) to infer the exact type of "a" > and treat any method calls on it as final (skip vtable) I think that's a worthy optimisation anyway. It's mostly a generalisation of ticket #3. http://trac.cython.org/cython_trac/ticket/3 And this is totally orthogonal to the whole final/inline discussion. It only relies on the "missing __new__ support" assumption that we already make. > and then whether INLINE is declared or not C-side is an orthogonal issue. > > I.e. "cdef inline foo(self)" could mean "allow overriding the method, > but make it inline in situations where the vtable can be skipped". > > Not that I think there's a real use for this... There almost certainly isn't. Unless the method is declared final, a programmer can't rely on Cython taking the decision to skip the vtable lookup in a specific case, so the result would be more or less unpredictable. Given that the C compiler is free to inline function calls anyway, even if not declared inline, applying the inline modified in this specific case sounds rather useless. Stefan From robertwb at math.washington.edu Mon Jan 11 19:55:05 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 11 Jan 2010 10:55:05 -0800 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B4AE895.6020804@behnel.de> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> Message-ID: <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> On Jan 11, 2010, at 1:00 AM, Stefan Behnel wrote: > Robert Bradshaw, 07.01.2010 22:06: >> On Jan 7, 2010, at 12:34 PM, Neil Ludban wrote: >>> Classes defined in extensions must explicitly indicate that they >>> support subclassing from pure Python. NoneType and bool are core >>> types that don't, there may be others. >> >> Ah, I did not know that. That does make a final modifier more >> attractive. This is a larger change to the language than allowing >> inline methods (with an error if one attempts to override them). > > I'm +0 on 'final' classes. I don't see a major use case, although > there may > be. Some extension types are really just meant as containers for C > types, > and when they are created from module internal code only, there may > not be > much value in subtyping them (although lxml.etree makes a good counter > example). > > Anyway, if it's supported by CPython, it should be ok to support it in > Cython, I guess, as long as users are aware that this really means > that > they put a serious restriction on the usage of their code, and that > this > should be used with caution. > > I certainly have seen enough Java code where 'final' was getting in > the way > at some point (and wasn't easy to remove from externally maintained > code), > or where the keyword was used totally arbitrarily in places where the > compiler can easily infer the semantics anyway, and where it just adds > noise to the code. I always expect an important intention when I see > this > keyword in code, and when there isn't one, it tends to confuse me, > because > it doesn't give me the feeling that I have understood the code. > > >>> The intended point was that the semantics of making a class "final" >>> are understood and would imply the requested optimizations, while >>> the "inline" keyword wouldn't fit at all in this context. > > IMHO, "inline" fits exactly the intended optimisation, whereas "final" > doesn't imply (and certainly not request) any optimisations at all. It just makes it safely possible. I am also of the mind that declaring something final so that it can be optimized is less obvious and straightforward than just declaring it inline in the first place. >> The inline keyword is nice because we already use it for functions >> (where the final keyword makes no sense). I think it's probably worth >> introducing both keywords, where inline would control emission of the >> C inline directive. (Of course for non-inline final methods, the C >> compiler could decide whether or not to actually inline.) A class >> could be final, with the same semantics as java. One question is >> would >> inline imply final, or require it? > > It makes no sense without final, Well, given that it's technically only a hint that the compiler C can ignore, and nearly always would for virtual methods, it would make sense but not be very useful. > so it should imply it (and certainly not That's what I was leaning towards as well. So, it sounds like +1 to inline methods (with the implication that they have final semantics), and lets hold off on adding the final keyword to the language until a CEP with compelling justification as it is a more significant change to the language. - Robert From sturla at molden.no Tue Jan 12 14:04:47 2010 From: sturla at molden.no (Sturla Molden) Date: Tue, 12 Jan 2010 14:04:47 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> Message-ID: I'd like to make a different suggestion. What about a keyword to pass after cdef in the declaration of the object to indicate the vtab can be skipped? I e an assertion that the object will be of the declared type. This will maintain better compatibility with Python. cdef Foo bar = Foo() cdef inline Foo fastbar = bar bar.method() # through vtab fastbar.method() # direct call SM Den 11. jan. 2010 kl. 19.55 skrev Robert Bradshaw : > On Jan 11, 2010, at 1:00 AM, Stefan Behnel wrote: > >> Robert Bradshaw, 07.01.2010 22:06: >>> On Jan 7, 2010, at 12:34 PM, Neil Ludban wrote: >>>> Classes defined in extensions must explicitly indicate that they >>>> support subclassing from pure Python. NoneType and bool are core >>>> types that don't, there may be others. >>> >>> Ah, I did not know that. That does make a final modifier more >>> attractive. This is a larger change to the language than allowing >>> inline methods (with an error if one attempts to override them). >> >> I'm +0 on 'final' classes. I don't see a major use case, although >> there may >> be. Some extension types are really just meant as containers for C >> types, >> and when they are created from module internal code only, there may >> not be >> much value in subtyping them (although lxml.etree makes a good >> counter >> example). >> >> Anyway, if it's supported by CPython, it should be ok to support it >> in >> Cython, I guess, as long as users are aware that this really means >> that >> they put a serious restriction on the usage of their code, and that >> this >> should be used with caution. >> >> I certainly have seen enough Java code where 'final' was getting in >> the way >> at some point (and wasn't easy to remove from externally maintained >> code), >> or where the keyword was used totally arbitrarily in places where the >> compiler can easily infer the semantics anyway, and where it just >> adds >> noise to the code. I always expect an important intention when I see >> this >> keyword in code, and when there isn't one, it tends to confuse me, >> because >> it doesn't give me the feeling that I have understood the code. >> >> >>>> The intended point was that the semantics of making a class "final" >>>> are understood and would imply the requested optimizations, while >>>> the "inline" keyword wouldn't fit at all in this context. >> >> IMHO, "inline" fits exactly the intended optimisation, whereas >> "final" >> doesn't imply (and certainly not request) any optimisations at all. > > It just makes it safely possible. I am also of the mind that declaring > something final so that it can be optimized is less obvious and > straightforward than just declaring it inline in the first place. > >>> The inline keyword is nice because we already use it for functions >>> (where the final keyword makes no sense). I think it's probably >>> worth >>> introducing both keywords, where inline would control emission of >>> the >>> C inline directive. (Of course for non-inline final methods, the C >>> compiler could decide whether or not to actually inline.) A class >>> could be final, with the same semantics as java. One question is >>> would >>> inline imply final, or require it? >> >> It makes no sense without final, > > Well, given that it's technically only a hint that the compiler C can > ignore, and nearly always would for virtual methods, it would make > sense but not be very useful. > >> so it should imply it (and certainly not > > That's what I was leaning towards as well. > > So, it sounds like +1 to inline methods (with the implication that > they have final semantics), and lets hold off on adding the final > keyword to the language until a CEP with compelling justification as > it is a more significant change to the language. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Tue Jan 12 15:44:11 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 12 Jan 2010 15:44:11 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> Message-ID: <4B4C8ABB.9000409@behnel.de> Sturla Molden, 12.01.2010 14:04: > I'd like to make a different suggestion. What about a keyword to pass > after cdef in the declaration of the object to indicate the vtab can > be skipped? I e an assertion that the object will be of the declared > type. This will maintain better compatibility with Python. > > cdef Foo bar = Foo() > cdef inline Foo fastbar = bar > > bar.method() # through vtab > fastbar.method() # direct call Why would you want to control this from user code? Stefan From nludban at osc.edu Tue Jan 12 16:11:59 2010 From: nludban at osc.edu (Neil Ludban) Date: Tue, 12 Jan 2010 10:11:59 -0500 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B4C8ABB.9000409@behnel.de> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> <4B4C8ABB.9000409@behnel.de> Message-ID: <866EC610-98A0-45D9-8914-A31B696071D0@osc.edu> On Jan 12, 2010, at 9:44 AM, Stefan Behnel wrote: > > Sturla Molden, 12.01.2010 14:04: >> I'd like to make a different suggestion. What about a keyword to pass >> after cdef in the declaration of the object to indicate the vtab can >> be skipped? I e an assertion that the object will be of the declared >> type. This will maintain better compatibility with Python. >> >> cdef Foo bar = Foo() >> cdef inline Foo fastbar = bar >> >> bar.method() # through vtab >> fastbar.method() # direct call > > Why would you want to control this from user code? > > Stefan It's explicit and selectively breaks Python's semantics as needed for optimization. Much nicer than implicit and global breakage. From stefan_ml at behnel.de Tue Jan 12 16:17:58 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 12 Jan 2010 16:17:58 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <866EC610-98A0-45D9-8914-A31B696071D0@osc.edu> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> <4B4C8ABB.9000409@behnel.de> <866EC610-98A0-45D9-8914-A31B696071D0@osc.edu> Message-ID: <4B4C92A6.8020507@behnel.de> Neil Ludban, 12.01.2010 16:11: > On Jan 12, 2010, at 9:44 AM, Stefan Behnel wrote: >> Sturla Molden, 12.01.2010 14:04: >>> I'd like to make a different suggestion. What about a keyword to pass >>> after cdef in the declaration of the object to indicate the vtab can >>> be skipped? I e an assertion that the object will be of the declared >>> type. This will maintain better compatibility with Python. >>> >>> cdef Foo bar = Foo() >>> cdef inline Foo fastbar = bar >>> >>> bar.method() # through vtab >>> fastbar.method() # direct call >> Why would you want to control this from user code? > > It's explicit and selectively breaks Python's semantics as needed > for optimization. Much nicer than implicit and global breakage. You assume that inlining would break things. It can't, given the current type instantiation semantics. That's why I don't see a reason to let users ask for inlining when the compiler can do it automatically without any user interaction. Stefan From stefan_ml at behnel.de Tue Jan 12 16:25:57 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 12 Jan 2010 16:25:57 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B4C92A6.8020507@behnel.de> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> <4B4C8ABB.9000409@behnel.de> <866EC610-98A0-45D9-8914-A31B696071D0@osc.edu> <4B4C92A6.8020507@behnel.de> Message-ID: <4B4C9485.5050704@behnel.de> Stefan Behnel, 12.01.2010 16:17: > Neil Ludban, 12.01.2010 16:11: >> On Jan 12, 2010, at 9:44 AM, Stefan Behnel wrote: >>> Sturla Molden, 12.01.2010 14:04: >>>> I'd like to make a different suggestion. What about a keyword to pass >>>> after cdef in the declaration of the object to indicate the vtab can >>>> be skipped? I e an assertion that the object will be of the declared >>>> type. This will maintain better compatibility with Python. >>>> >>>> cdef Foo bar = Foo() >>>> cdef inline Foo fastbar = bar >>>> >>>> bar.method() # through vtab >>>> fastbar.method() # direct call >>> Why would you want to control this from user code? >> It's explicit and selectively breaks Python's semantics as needed >> for optimization. Much nicer than implicit and global breakage. > > You assume that inlining would break things. It can't, given the current > type instantiation semantics. Sorry, ignore what I just said. It can, just not in the above example. The case we are talking about here is this: cdef MyType x = something_that_returns_some_object() cdef inline MyType fast_x = x in which case 'x' may refer to any subtype of MyType, which can obviously override the methods in MyType. I'm not sure I'm comfortable with this idea, though. This would allow users to break into the hierarchy of an existing type by calling a specific method sort-of half-explicitly. That sounds rather error prone to me. It's better to keep the control inside of the class definition IMHO. Stefan From robertwb at math.washington.edu Tue Jan 12 21:31:55 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 12 Jan 2010 12:31:55 -0800 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> Message-ID: <8D44B9BB-CAB8-49CF-BB18-1A1399B6FC42@math.washington.edu> On Jan 12, 2010, at 5:04 AM, Sturla Molden wrote: > I'd like to make a different suggestion. What about a keyword to pass > after cdef in the declaration of the object to indicate the vtab can > be skipped? I e an assertion that the object will be of the declared > type. This will maintain better compatibility with Python. > > cdef Foo bar = Foo() > cdef inline Foo fastbar = bar Hmm... The notion of an "inline object" to me evokes allocating on the stack, or passing by value, or something like that--not ignoring type information when doing method dispatching. This isn't too important-- another keyword could be found. > bar.method() # through vtab > fastbar.method() # direct call There is already a more explicit way to get this behavior: cdef class A: cdef identify(self): print "I'm an A" cdef class B(A): cdef identify(self): print "I'm a B" cdef A a = A() cdef B b = B() a.identify() # prints "I'm an A" b.identify() # prints "I'm a B" A.identify(b) # prints "I'm an A" It could probably be optimized to a direct call if desired by making the type vtab store const function pointers. In any case, a non-overridable inline cdef method doesn't break Python semantics. (Well, you can't override it, but that's caught at compile time, rather than potentially surprising an end user at runtime.) - Robert From sturla at molden.no Wed Jan 13 11:29:59 2010 From: sturla at molden.no (Sturla Molden) Date: Wed, 13 Jan 2010 11:29:59 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <4B4C8ABB.9000409@behnel.de> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> <4B4C8ABB.9000409@behnel.de> Message-ID: <42CC7582-6656-44A7-B7F3-284BEEE920F2@molden.no> Den 12. jan. 2010 kl. 15.44 skrev Stefan Behnel : > >> cdef Foo bar = Foo() >> cdef inline Foo fastbar = bar >> >> bar.method() # through vtab >> fastbar.method() # direct call > > Why would you want to control this from user code? > Because I feel the common usecase for skipping the vtab indirection is calling a method repeatedly from user code. This will not break Python semantics globally, but be a local optimization when the user knows the type of an object. Yes, we can already do this by declaring a const function pointer to the method; but this approach is less messy, and requires less knowledge of C. Another way of defining this could be by inheritance. For example: cdef class FastFoo(Foo, cython.finalobject): pass cdef Foo bar = Foo() cdef FastFoo fastbar = bar Sturla From stefan_ml at behnel.de Wed Jan 13 13:00:01 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 13 Jan 2010 13:00:01 +0100 Subject: [Cython] Make 'inline' methods in cdef-class non-overridable? In-Reply-To: <42CC7582-6656-44A7-B7F3-284BEEE920F2@molden.no> References: <4B44CA57.2010201@behnel.de> <4B44E1FE.7000900@student.matnat.uio.no> <9E9EADE9-3F87-486B-9C04-1544EE903B23@math.washington.edu> <5BD2E015-331D-450C-B84F-F1AD6D06F4C2@osc.edu> <7e494c94956dd69540e20f98a8d54560.squirrel@webmail.uio.no> <4B45F4BF.7060408@behnel.de> <7418DE25-52BE-42B3-803F-C4E60D3D830A@osc.edu> <739477D6-7422-4FCD-84A8-64A662735EE7@math.washington.edu> <50D0DA70-8249-439C-99D5-F677C98E7288@osc.edu> <92E11115-DCCD-4DF4-8C28-97D4CA914B3E@math.washington.edu> <4B4AE895.6020804@behnel.de> <9EFABF87-F234-40ED-8524-70DE78E9F198@math.washington.edu> <4B4C8ABB.9000409@behnel.de> <42CC7582-6656-44A7-B7F3-284BEEE920F2@molden.no> Message-ID: <4B4DB5C1.6080902@behnel.de> Sturla Molden, 13.01.2010 11:29: > Den 12. jan. 2010 kl. 15.44 skrev Stefan Behnel : > >>> cdef Foo bar = Foo() >>> cdef inline Foo fastbar = bar >>> >>> bar.method() # through vtab >>> fastbar.method() # direct call >> >> Why would you want to control this from user code? > > Because I feel the common usecase for skipping the vtab indirection is > calling a method repeatedly from user code. > > This will not break Python semantics globally, but be a local > optimization when the user knows the type of an object. I'm not so much worried about Python semantics here. Extension types are different anyway. My main concern is with the user knowing "the type of an object". The optimisation you enforce here is based on an explicit type that you ask Cython to use. It's not based on the actual type of the object. If anywhere in the evolution of the code base it happens that the object in question is replaced with a subtype, the code above will break without warning and potentially in a subtle, non-obvious way. The only way to make this safe would be to enforce exact type checks on assignments to a "cdef inline" variable. That breaks other common code semantics, and it also wouldn't work for the case where a basetype defines a method that is *not* overridden by subtypes. But it would at least raise an exception if the code is known to be incorrect. Given that such a restriction would be required, I fail to see a real advantage over a 'final' class. > Another way of defining this could be by inheritance. For example: > > cdef class FastFoo(Foo, cython.finalobject): > pass > > cdef Foo bar = Foo() > cdef FastFoo fastbar = bar That can't work because Foo isn't a FastFoo type. If we add a 'final' modifier to classes, a decorator is the only sensible syntax, IMHO. Stefan From berthold at despammed.com Thu Jan 14 09:28:52 2010 From: berthold at despammed.com (Berthold Hoellmann) Date: Thu, 14 Jan 2010 09:28:52 +0100 Subject: [Cython] Status of fwrap? Message-ID: What is the of fwrap? The latest entry in http://fortrancython.wordpress.com/ from last August promises a first release before 2010 which obviously did not happen. Kind regards Berthold H?llmann -- __ Address: G / \ L Germanischer Lloyd phone: +49-40-36149-7374 -+----+- Vorsetzen 35 P.O.Box 111606 fax : +49-40-36149-7320 \__/ D-20459 Hamburg D-20416 Hamburg From kwmsmith at gmail.com Fri Jan 15 00:07:14 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Thu, 14 Jan 2010 17:07:14 -0600 Subject: [Cython] Status of fwrap? In-Reply-To: References: Message-ID: Sickness, death in the family, trying to finish my dissertation -- all cut into my discretionary time. I'll put an fwrap status post on the blog for those who are interested. If you'd like to lend a hand, fparser could use some work. Kurt On Thu, Jan 14, 2010 at 2:28 AM, Berthold Hoellmann wrote: > > What is the of fwrap? The latest entry in > > ? ? http://fortrancython.wordpress.com/ > > from last August promises a first release before 2010 which obviously > did not happen. > > Kind regards > Berthold H?llmann > -- > ? ? ? ? ? ? ? ? ? ? ? ? ? ?__ ? Address: > ? ? ? ? ? ? ? ? ? ? ? ? G / ?\ L Germanischer Lloyd > phone: +49-40-36149-7374 -+----+- Vorsetzen 35 ? ? ? P.O.Box 111606 > fax ?: +49-40-36149-7320 ? \__/ ? D-20459 Hamburg ? ?D-20416 Hamburg > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Fri Jan 15 09:27:33 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 15 Jan 2010 09:27:33 +0100 Subject: [Cython] Status of fwrap? In-Reply-To: References: Message-ID: <4B5026F5.8030101@student.matnat.uio.no> Kurt Smith wrote: > Sickness, death in the family, trying to finish my dissertation -- all > cut into my discretionary time. I'll put an fwrap status post on the > blog for those who are interested. > > If you'd like to lend a hand, fparser could use some work. > I thought I'd answer the part about build systems here (see http://fortrancython.wordpress.com/ for original post). First off, my take is just "do whatever will get any results quickest" and take it from there. As it was said in SciPy 09: The trick is to get things 90% working so that people can start playing with it -- perhaps somebody else will find it easy to step in and create the perfect build solution once they start using fwrap. Get to the point where people can start to scratch their itches. Also be careful that the issue of building Python code and native code together is a problem that's not yet solved -- i.e. there's not yet a standard solution with a significant mindshare. 1. Cython/NumPy distutils: I'd advise against this. David Cournapeau has been the primary maintainer of NumPy distutils, but he's very actively pushing for getting off distutils for scientific use altogether. (And the reason "scientific use" is a category itself is because we have mixed C/Python/Fortran, a land fwrap very much live in.) To start with this David launched "toydist", so that one can use any tool for building and still make Python packages: http://thread.gmane.org/gmane.comp.python.numeric.general/35276 2. Scons or numscons -- those would be my options. Perhaps scons, because presumably numscons users should be able to use scons and numscons plugins? For compiling Cython code there's some work in the Cython repo: Tools/site_scons. In the end, I think it is unavoidable that fwrap will support many build systems, and perhaps that some users want to craft their build systems manually. Dag Sverre From robertwb at math.washington.edu Fri Jan 22 08:24:38 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Jan 2010 23:24:38 -0800 Subject: [Cython] Cython 0.12.1 and 0.13 release plans Message-ID: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> It seems we're due for another release...there haven't been any major changes since November, but there are a number of small improvements and bugfixes, including bugfixes for compiling on Windows (I hope, still waiting for feedback), Solaris, and less string type checking for the SciPy folks. Does anyone have anything else they'd like to see in sooner rather than later? Any blockers? Otherwise I'll start rolling alphas and testing on the freshly-minted Sage 4.3.1. I'd like to see an 0.13 soon thereafter. I'm imagining a big release where we pull in a bunch of stuff that's been "almost ready" for a while: 1. Safe type inference (on by default) This seems to be basically ready, but is too big of a change for an x.y.z release. 2. Closures Yes, this has been "almost ready" for a long time, but the only bug now is a refnanny complaint. (The refcounts are correct, it's a question of miss-placed give/got refs on the scope object). Craig Citro wants to run another battery of tests on it as well, it's just a question of him or I finding a couple of hours (at most I think) to sit down and do it. 3. C++ support This has been lingering way to long. Danilo got some stuff done over the summer, but there's still a huge amount to go. However, in the spirit of release early, release often, the stuff that is there should get out and get used. Last month I cleaned up a lot of stuff, and synced up with the 0.12 release. I also wrote several tests, fixed all the bugs they exposed, finished operator overloading support, and improved error reporting so unimplemented stuff will give appropriate errors rather than cause compiler crashes. Danilo also submitted a small patch over the break. It's not to the point that one can easily use all of STL (that was the goal, but there's no clean way to declare, for example, vector::iterator, and references are a work in progress, let alone stack-allocated objects (or emulations thereof)), but it is still a huge step forward in the way C++ classes are declared and used. Thoughts? - Robert From stefan_ml at behnel.de Fri Jan 22 09:40:04 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jan 2010 09:40:04 +0100 Subject: [Cython] Cython 0.12.1 and 0.13 release plans In-Reply-To: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> Message-ID: <4B596464.1030804@behnel.de> Robert Bradshaw, 22.01.2010 08:24: > It seems we're due for another release... Very much +1. > Does anyone have anything else they'd like to see > in sooner rather than later? Any blockers? According to trac, there are no real blockers for 0.12.1. The only real bugs targeted for that release are #473 (MSVC problem) and #467. The latter one can be bad in rare cases but shouldn't do much harm normally (and isn't exactly trivial to get right). > Otherwise I'll start > rolling alphas and testing on the freshly-minted Sage 4.3.1. Please do. > I'd like to see an 0.13 soon thereafter. Big +1 again. I'm fine with 0.13.0 being "not 100% stable" but would love to have it out there so that people can use the new features and find bugs. > I'm imagining a big release > where we pull in a bunch of stuff that's been "almost ready" for a > while: > > 1. Safe type inference (on by default) > > This seems to be basically ready, but is too big of a change for an > x.y.z release. Ok with me, but that's in cython-devel already. Would you want to roll back the implementation or just disable the feature by default? The latter should basically revert to the original behaviour already, AFAICT. > 2. Closures Sure, let's get that one out and then take a look at where that gets us. There hasn't been such a major new feature in Cython for a while, and this will enable tons of new use cases and forms the basis for quite a lot of other Python features (e.g. lambda and generators). > Craig Citro wants to run another battery of tests on it Can we add that to the regular test suite somehow? > 3. C++ support I can't say I'm anywhere near up-to-date with the status of that, but if you think it's at a somewhat stable point, so that existing features won't have to get rolled back and replaced, I'm fine with adding it (or parts of it) in 0.13. It's always good to enable users to exercise new features early, and especially the signature overloading support should be very useful to Cython's optimiser by itself. What about the Fortran integration? Is there anything we can add from that already? If we get a part of that in, we might be able to attract other Fortran users. Some of them may even end up participating in the further development. Putting this on more shoulders is the best way to push it further. Stefan From robertwb at math.washington.edu Fri Jan 22 10:08:28 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Jan 2010 01:08:28 -0800 Subject: [Cython] Cython 0.12.1 and 0.13 release plans In-Reply-To: <4B596464.1030804@behnel.de> References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> <4B596464.1030804@behnel.de> Message-ID: <78A33703-F6BF-410F-B602-CE37077464AC@math.washington.edu> On Jan 22, 2010, at 12:40 AM, Stefan Behnel wrote: > > Robert Bradshaw, 22.01.2010 08:24: >> It seems we're due for another release... > > Very much +1. > > >> Does anyone have anything else they'd like to see >> in sooner rather than later? Any blockers? > > According to trac, there are no real blockers for 0.12.1. The only > real > bugs targeted for that release are #473 (MSVC problem) Fix posted, waiting for feedback. > and #467. The latter one can be bad in rare cases but shouldn't do > much harm normally (and isn't > exactly trivial to get right). Not quite a blocker, but it'd be really good to get that fixed. > >> Otherwise I'll start >> rolling alphas and testing on the freshly-minted Sage 4.3.1. > > Please do. > > >> I'd like to see an 0.13 soon thereafter. > > Big +1 again. I'm fine with 0.13.0 being "not 100% stable" but would > love > to have it out there so that people can use the new features and > find bugs. Also, most of the big changes are new features, so even if they're not 100% polished it should be pretty safe with existing code. >> I'm imagining a big release >> where we pull in a bunch of stuff that's been "almost ready" for a >> while: >> >> 1. Safe type inference (on by default) >> >> This seems to be basically ready, but is too big of a change for an >> x.y.z release. > > Ok with me, but that's in cython-devel already. Would you want to > roll back > the implementation or just disable the feature by default? The latter > should basically revert to the original behaviour already, AFAICT. I see at http://hg.cython.org/cython-devel/file/fce98451e2dd/Cython/Compiler/Options.py#l51 that it's False. None is the "safe" value, right? >> 2. Closures > > Sure, let's get that one out and then take a look at where that gets > us. > There hasn't been such a major new feature in Cython for a while, > and this > will enable tons of new use cases and forms the basis for quite a > lot of > other Python features (e.g. lambda and generators). lambda is pretty much there. (IIRC, you did the parse tree transform for that, right?) > > >> Craig Citro wants to run another battery of tests on it > > Can we add that to the regular test suite somehow? Yes, that's the plan. > > >> 3. C++ support > > I can't say I'm anywhere near up-to-date with the status of that, > but if > you think it's at a somewhat stable point, so that existing features > won't > have to get rolled back and replaced, I'm fine with adding it (or > parts of > it) in 0.13. It's always good to enable users to exercise new features > early, and especially the signature overloading support should be very > useful to Cython's optimiser by itself. Yes, I tried to make sure everything exposed is stable and compatible moving forward. There are some design decisions that would be good to confirm before we release (e.g. I added operators cython.preincrement, cython.dereference, etc. which might be suited for shorter names, and we would be fixing on "operator+" rather than "__add__" declaration notation.) I'd like to add a wiki at least, and hopefully a chapter in the user manual, about what can be done. For now, the best is to look at the cpp_* tests, e.g. http://hg.cython.org/gsoc-danilo/file/tip/tests/run/cpp_templates.pyx . A big step forward is that now it understands new, del, and inheritance of types. (Before one had to do all sorts of pointer casting which was really messy.) On top of that, the rest is mostly how many features we support, not a whole new way of wrapping libraries. > What about the Fortran integration? Is there anything we can add > from that > already? If we get a part of that in, we might be able to attract > other > Fortran users. Some of them may even end up participating in the > further > development. Putting this on more shoulders is the best way to push > it further. That'd be nice as well, but it sounds like most of the focus of the project right now is on build systems, and I'm not sure if it's set in stone enough for people to start building codebases around. (I'd be very glad to be wrong, if some of this is ready to go in, now would be great time!) - Robert From stefan_ml at behnel.de Fri Jan 22 13:43:07 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jan 2010 13:43:07 +0100 Subject: [Cython] Cython 0.12.1 and 0.13 release plans In-Reply-To: <78A33703-F6BF-410F-B602-CE37077464AC@math.washington.edu> References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> <4B596464.1030804@behnel.de> <78A33703-F6BF-410F-B602-CE37077464AC@math.washington.edu> Message-ID: <4B599D5B.6090604@behnel.de> Robert Bradshaw, 22.01.2010 10:08: > On Jan 22, 2010, at 12:40 AM, Stefan Behnel wrote: >> and #467. The latter one can be bad in rare cases but shouldn't do >> much harm normally (and isn't >> exactly trivial to get right). > > Not quite a blocker, but it'd be really good to get that fixed. Why did I expect that answer? ;) >>> I'm imagining a big release >>> where we pull in a bunch of stuff that's been "almost ready" for a >>> while: >>> >>> 1. Safe type inference (on by default) >>> >>> This seems to be basically ready, but is too big of a change for an >>> x.y.z release. >> Ok with me, but that's in cython-devel already. Would you want to >> roll back >> the implementation or just disable the feature by default? The latter >> should basically revert to the original behaviour already, AFAICT. > > I see at http://hg.cython.org/cython-devel/file/fce98451e2dd/Cython/Compiler/Options.py#l51 > that it's False. None is the "safe" value, right? Right, so we'd just leave it as False for 0.12.x then. >>> 2. Closures >> Sure, let's get that one out and then take a look at where that gets >> us. >> There hasn't been such a major new feature in Cython for a while, >> and this >> will enable tons of new use cases and forms the basis for quite a >> lot of >> other Python features (e.g. lambda and generators). > > lambda is pretty much there. (IIRC, you did the parse tree transform > for that, right?) :) I remembered that I had started working on that, but I didn't remember that I had actually finished it up. But I just checked and there's a working test case, so I'm happy to agree that that's done. I guess generators are next, then. >>> 3. C++ support >> I can't say I'm anywhere near up-to-date with the status of that, >> but if >> you think it's at a somewhat stable point, so that existing features >> won't >> have to get rolled back and replaced, I'm fine with adding it (or >> parts of >> it) in 0.13. It's always good to enable users to exercise new features >> early, and especially the signature overloading support should be very >> useful to Cython's optimiser by itself. > > Yes, I tried to make sure everything exposed is stable and compatible > moving forward. There are some design decisions that would be good to > confirm before we release (e.g. I added operators cython.preincrement, > cython.dereference, etc. which might be suited for shorter names I'm fine with "preincrement", assuming that it's only really required in a couple of cases. The "dereference" could be shortened to "deref", I expect that to be used much more often. We also shouldn't forget that users can always do the abbreviation themselves during the import. BTW, should we use a separate namespace module? E.g. "cythoncpp"? I think that would make it clearer what the usage scope is, and the imports would still be as broadly simplifiable as before. And Cython could warn if you import the C++ module while generating plain C code. :) >, and > we would be fixing on "operator+" rather than "__add__" declaration > notation.) Yes, I still think that makes sense. Stefan From ndbecker2 at gmail.com Fri Jan 22 15:48:04 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 22 Jan 2010 09:48:04 -0500 Subject: [Cython] Cython 0.12.1 and 0.13 release plans References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> Message-ID: Robert Bradshaw wrote: > 3. C++ support > > This has been lingering way to long. Danilo got some stuff done over > the summer, but there's still a huge amount to go. However, in the > spirit of release early, release often, the stuff that is there should > get out and get used. Last month I cleaned up a lot of stuff, and > synced up with the 0.12 release. I also wrote several tests, fixed all > the bugs they exposed, finished operator overloading support, and > improved error reporting so unimplemented stuff will give appropriate > errors rather than cause compiler crashes. Danilo also submitted a > small patch over the break. It's not to the point that one can easily > use all of STL (that was the goal, but there's no clean way to > declare, for example, vector::iterator, and references are a work > in progress, let alone stack-allocated objects (or emulations > thereof)), but it is still a huge step forward in the way C++ classes > are declared and used. > > Thoughts? > > - Robert Your mention of overloading got me thinking, there is recent work in boost::python to do a more careful job of exposing overload c++ functions to python. I think this was discussed on Development of Python/C++ integration From kwmsmith at gmail.com Fri Jan 22 18:51:36 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 22 Jan 2010 11:51:36 -0600 Subject: [Cython] Cython 0.12.1 and 0.13 release plans In-Reply-To: <78A33703-F6BF-410F-B602-CE37077464AC@math.washington.edu> References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> <4B596464.1030804@behnel.de> <78A33703-F6BF-410F-B602-CE37077464AC@math.washington.edu> Message-ID: On Fri, Jan 22, 2010 at 3:08 AM, Robert Bradshaw wrote: > On Jan 22, 2010, at 12:40 AM, Stefan Behnel wrote: >> Robert Bradshaw, 22.01.2010 08:24: >> What about the Fortran integration? Is there anything we can add >> from that >> already? If we get a part of that in, we might be able to attract >> other >> Fortran users. Some of them may even end up participating in the >> further >> development. Putting this on more shoulders is the best way to push >> it further. > > > That'd be nice as well, but it sounds like most of the focus of the > project right now is on build systems, and I'm not sure if it's set in > stone enough for people to start building codebases around. (I'd be > very glad to be wrong, if some of this is ready to go in, now would be > great time!) I'll pipe up here; sorry, been away from the mailing list for a while. For the record, the fortran project wasn't about providing explicit fortran support in Cython, but rather to use Cython as a way to help wrap fortran codes. As a necessary part of the project, Cython will gain support for memoryviews with a nice declaration syntax and the ability to pass these memoryviews as arguments to cdef functions, and some other features (declaring them as fields in extension types is another). These memoryviews are PEP 3118 buffers that are very flexible and can work with fortran arrays. They're updates and improvements to the existing buffer support in Cython. I'd very much like the Cython-side of the GSoC project to be merged into 0.13, but I'll have to coordinate with Dag and see what remains to be done. The main features are in place, but it's been a while since I've focused on this part of the project. I'm happy to devote my fwrap time to this to get it in for the 0.13 release. I'll be in touch with Dag and see what he has to say (or he might jump in here). Kurt From dagss at student.matnat.uio.no Fri Jan 22 19:51:02 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Jan 2010 19:51:02 +0100 Subject: [Cython] Cython 0.12.1 and 0.13 release plans In-Reply-To: References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> <4B596464.1030804@behnel.de> <78A33703-F6BF-410F-B602-CE37077464AC@math.washington.edu> Message-ID: <4B59F396.2010104@student.matnat.uio.no> Kurt Smith wrote: > On Fri, Jan 22, 2010 at 3:08 AM, Robert Bradshaw > wrote: >> On Jan 22, 2010, at 12:40 AM, Stefan Behnel wrote: >>> Robert Bradshaw, 22.01.2010 08:24: > >>> What about the Fortran integration? Is there anything we can add >>> from that >>> already? If we get a part of that in, we might be able to attract >>> other >>> Fortran users. Some of them may even end up participating in the >>> further >>> development. Putting this on more shoulders is the best way to push >>> it further. >> >> That'd be nice as well, but it sounds like most of the focus of the >> project right now is on build systems, and I'm not sure if it's set in >> stone enough for people to start building codebases around. (I'd be >> very glad to be wrong, if some of this is ready to go in, now would be >> great time!) > > I'll pipe up here; sorry, been away from the mailing list for a while. > > For the record, the fortran project wasn't about providing explicit > fortran support in Cython, but rather to use Cython as a way to help > wrap fortran codes. As a necessary part of the project, Cython will > gain support for memoryviews with a nice declaration syntax and the > ability to pass these memoryviews as arguments to cdef functions, and > some other features (declaring them as fields in extension types is > another). These memoryviews are PEP 3118 buffers that are very > flexible and can work with fortran arrays. They're updates and > improvements to the existing buffer support in Cython. > > I'd very much like the Cython-side of the GSoC project to be merged > into 0.13, but I'll have to coordinate with Dag and see what remains > to be done. The main features are in place, but it's been a while > since I've focused on this part of the project. I'm happy to devote > my fwrap time to this to get it in for the 0.13 release. > > I'll be in touch with Dag and see what he has to say (or he might jump in here). If you're asking me what I think you should work on, then I think the memoryviews should be rather low on the list; there's still a chance I'll make some time for finishing those, and they're not strictly needed, just convenient. So I'd have this priority: 1) Rock stable Fortran parsing and wrapper generation (where you have the advantage of knowing the internals better than anybody else that might help) 2) Build system (which somebody else might help with if 1) was just working) 3) Memory views (which I might do at some point) Did you form any conclusions on the waf vs. scons vs. distutils side? -- Dag Sverre From robertwb at math.washington.edu Fri Jan 22 20:37:28 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Jan 2010 11:37:28 -0800 Subject: [Cython] Cython 0.12.1 and 0.13 release plans In-Reply-To: <4B599D5B.6090604@behnel.de> References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> <4B596464.1030804@behnel.de> <78A33703-F6BF-410F-B602-CE37077464AC@math.washington.edu> <4B599D5B.6090604@behnel.de> Message-ID: <20E8FF47-5C18-489E-8269-F24706BAF949@math.washington.edu> On Jan 22, 2010, at 4:43 AM, Stefan Behnel wrote: >>>> 3. C++ support >>> I can't say I'm anywhere near up-to-date with the status of that, >>> but if >>> you think it's at a somewhat stable point, so that existing features >>> won't >>> have to get rolled back and replaced, I'm fine with adding it (or >>> parts of >>> it) in 0.13. It's always good to enable users to exercise new >>> features >>> early, and especially the signature overloading support should be >>> very >>> useful to Cython's optimiser by itself. >> >> Yes, I tried to make sure everything exposed is stable and compatible >> moving forward. There are some design decisions that would be good to >> confirm before we release (e.g. I added operators >> cython.preincrement, >> cython.dereference, etc. which might be suited for shorter names > > I'm fine with "preincrement", assuming that it's only really > required in a > couple of cases. The "dereference" could be shortened to "deref", I > expect > that to be used much more often. We also shouldn't forget that users > can > always do the abbreviation themselves during the import. Right now we have: - preincrement - postincrement - predecrement - postdecrement - dereference - address # was added for pure mode I'm assuming any user will alias them to shorter names, but maybe we should too. For example, "inc" is nice and short, but ambiguous. > BTW, should we use a separate namespace module? E.g. "cythoncpp"? I > think > that would make it clearer what the usage scope is, and the imports > would > still be as broadly simplifiable as before. And Cython could warn if > you > import the C++ module while generating plain C code. :) Those operators all make sense in C as well, no need to limit them. If we're going with a separate namespace, I'd prefer cython.operators. - Robert From robertwb at math.washington.edu Fri Jan 22 20:38:45 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Jan 2010 11:38:45 -0800 Subject: [Cython] Cython 0.12.1.alpha0 Message-ID: Up at http://sage.math.washington.edu/home/robertwb/cython/Cython-0.12.1.alpha0.tar.gz . I just packaged up the current cython-devel tip, will start a Sage building with it later today. - Robert From kwmsmith at gmail.com Fri Jan 22 21:14:43 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 22 Jan 2010 14:14:43 -0600 Subject: [Cython] Cython 0.12.1 and 0.13 release plans In-Reply-To: <4B59F396.2010104@student.matnat.uio.no> References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> <4B596464.1030804@behnel.de> <78A33703-F6BF-410F-B602-CE37077464AC@math.washington.edu> <4B59F396.2010104@student.matnat.uio.no> Message-ID: On Fri, Jan 22, 2010 at 12:51 PM, Dag Sverre Seljebotn wrote: > Kurt Smith wrote: >> On Fri, Jan 22, 2010 at 3:08 AM, Robert Bradshaw >> wrote: >>> On Jan 22, 2010, at 12:40 AM, Stefan Behnel wrote: >>>> Robert Bradshaw, 22.01.2010 08:24: >> >>>> What about the Fortran integration? Is there anything we can add >>>> from that >>>> already? If we get a part of that in, we might be able to attract >>>> other >>>> Fortran users. Some of them may even end up participating in the >>>> further >>>> development. Putting this on more shoulders is the best way to push >>>> it further. >>> >>> That'd be nice as well, but it sounds like most of the focus of the >>> project right now is on build systems, and I'm not sure if it's set in >>> stone enough for people to start building codebases around. (I'd be >>> very glad to be wrong, if some of this is ready to go in, now would be >>> great time!) >> >> I'll pipe up here; sorry, been away from the mailing list for a while. >> >> For the record, the fortran project wasn't about providing explicit >> fortran support in Cython, but rather to use Cython as a way to help >> wrap fortran codes. ?As a necessary part of the project, Cython will >> gain support for memoryviews with a nice declaration syntax and the >> ability to pass these memoryviews as arguments to cdef functions, and >> some other features (declaring them as fields in extension types is >> another). ?These memoryviews are PEP 3118 buffers that are very >> flexible and can work with fortran arrays. ?They're updates and >> improvements to the existing buffer support in Cython. >> >> I'd very much like the Cython-side of the GSoC project to be merged >> into 0.13, but I'll have to coordinate with Dag and see what remains >> to be done. ?The main features are in place, but it's been a while >> since I've focused on this part of the project. ?I'm happy to devote >> my fwrap time to this to get it in for the 0.13 release. >> >> I'll be in touch with Dag and see what he has to say (or he might jump in here). > > If you're asking me what I think you should work on, then I think the > memoryviews should be rather low on the list; there's still a chance > I'll make some time for finishing those, and they're not strictly > needed, just convenient. > > So I'd have this priority: > ?1) Rock stable Fortran parsing and wrapper generation (where you have > the advantage of knowing the internals better than anybody else that > might help) > ?2) Build system (which somebody else might help with if 1) was just > working) Yes, these two have been my priority. Since fwrap will be shipped separately from Cython, these can evolve on their own, and ideally (1) will be ready to go in a few months. (2) already works, using numpy.distutils and Cython.distutils, but I'd much prefer something else. Re: memoryviews: My preference would be to get the core functionality of memoryviews working and included in Cython 0.13, since this is a natural sub-project and would be nice to have in a distributed version. (By 'core functionality' I mean -- somewhat selfishly -- the memoryview functionality needed for fwrap :-) But it sounds like you would like to take the memoryviews further than what was done last summer, which is fine. How much of the memoryview support do you want implemented before merging with cython-dev? It would be good to have fwrap be able to work with either the existing buffers in Cython and the new memoryviews, so it ultimately doesn't matter if memoryviews are merged into 0.13. > ?3) Memory views (which I might do at some point) > > Did you form any conclusions on the waf vs. scons vs. distutils side? As I mentioned -- distutils already work, but isn't nice. I'm fine with it for now. I haven't been able to test waf vs. scons yet but hope to soon. From robertwb at math.washington.edu Sun Jan 24 12:29:47 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 24 Jan 2010 03:29:47 -0800 Subject: [Cython] Cython 0.12.1.alpha0 In-Reply-To: References: Message-ID: <8B9A9C4B-AACA-4CD0-AF61-B8D767201AEE@math.washington.edu> On Jan 22, 2010, at 11:38 AM, Robert Bradshaw wrote: > Up at http://sage.math.washington.edu/home/robertwb/cython/Cython-0.12.1.alpha0.tar.gz > . I just packaged up the current cython-devel tip, will start a Sage > building with it later today. Sage does not compile: http://trac.cython.org/cython_trac/ticket/480 From cournape at gmail.com Sun Jan 24 14:16:54 2010 From: cournape at gmail.com (David Cournapeau) Date: Sun, 24 Jan 2010 22:16:54 +0900 Subject: [Cython] Cython 0.12.1.alpha0 In-Reply-To: References: Message-ID: <5b8d13221001240516w7a3f9e62l85bb309635fae34@mail.gmail.com> On Sat, Jan 23, 2010 at 4:38 AM, Robert Bradshaw wrote: > Up at http://sage.math.washington.edu/home/robertwb/cython/Cython-0.12.1.alpha0.tar.gz > ?. I just packaged up the current cython-devel tip, will start a Sage > building with it later today. It seems that it is being taken care of, but I would like to confirm: will the type checking done by cython be relaxed for this release ? I would like to fix this ASAP for scipy and numpy binaries, cheers, David From stefan_ml at behnel.de Sun Jan 24 14:55:32 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 24 Jan 2010 14:55:32 +0100 Subject: [Cython] Cython 0.12.1.alpha0 In-Reply-To: <5b8d13221001240516w7a3f9e62l85bb309635fae34@mail.gmail.com> References: <5b8d13221001240516w7a3f9e62l85bb309635fae34@mail.gmail.com> Message-ID: <4B5C5154.7040605@behnel.de> David Cournapeau, 24.01.2010 14:16: > On Sat, Jan 23, 2010 at 4:38 AM, Robert Bradshaw > wrote: >> Up at http://sage.math.washington.edu/home/robertwb/cython/Cython-0.12.1.alpha0.tar.gz >> . I just packaged up the current cython-devel tip, will start a Sage >> building with it later today. > > It seems that it is being taken care of, but I would like to confirm: > will the type checking done by cython be relaxed for this release ? I assume you are referring to the extension type size checking when cimporting between modules? Stefan From cournape at gmail.com Mon Jan 25 01:58:28 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 25 Jan 2010 09:58:28 +0900 Subject: [Cython] Cython 0.12.1.alpha0 In-Reply-To: <4B5C5154.7040605@behnel.de> References: <5b8d13221001240516w7a3f9e62l85bb309635fae34@mail.gmail.com> <4B5C5154.7040605@behnel.de> Message-ID: <5b8d13221001241658j1ff82a8ama9aa433c2c8146ad@mail.gmail.com> On Sun, Jan 24, 2010 at 10:55 PM, Stefan Behnel wrote: > > David Cournapeau, 24.01.2010 14:16: >> On Sat, Jan 23, 2010 at 4:38 AM, Robert Bradshaw >> wrote: >>> Up at http://sage.math.washington.edu/home/robertwb/cython/Cython-0.12.1.alpha0.tar.gz >>> ?. I just packaged up the current cython-devel tip, will start a Sage >>> building with it later today. >> >> It seems that it is being taken care of, but I would like to confirm: >> will the type checking done by cython be relaxed for this release ? > > I assume you are referring to the extension type size checking when > cimporting between modules? Yes - currently, numpy 1.4.0 is not compatible with scipy 0.7.1 binaries (at least) because of this. I would like to release 1.4.1 and scipy 0.7.1.1 ASAP, as it is a rather major inconvenience to people relying on binaries, Thanks for the help, cheers, David From stefan_ml at behnel.de Mon Jan 25 08:20:57 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 25 Jan 2010 08:20:57 +0100 Subject: [Cython] Support for borrowed references In-Reply-To: <4B4A0D26.7070807@behnel.de> References: <4B48E1BA.7000304@behnel.de> <4B4A0D26.7070807@behnel.de> Message-ID: <4B5D4659.9070206@behnel.de> [bump] Stefan Behnel, 10.01.2010 18:23: > [...] why not just represent a borrowed reference as a pointer? > So you could write > > cdef list some_list = [] > cdef list* borrowed_ref = some_list > > and borrowed_ref would be a non-refcounted pointer to a Python list. > Assignments back to a normal reference would be allowed: > > cdef list my_list = borrowed_ref # increfs the pointer > > After all, a non-refcounted reference to a Python object is not more than a > bare pointer to a well-defined Python builtin/extension type (including > "object*"). What do the others think about this? Note that this does not relate to the "stolen reference" case, for which Robert already proposed a "steal()" function (and which I expect to be a rare use case anyway). It only deals with borrowed references, where e.g. PyTuple_GET_ITEM() would be declared as cdef object* PyTuple_GET_ITEM(object t, Py_ssize_t index) and Cython would automatically do the right thing when you write cdef object x = PyTuple_GET_ITEM(t, 0) or cdef object* x = PyTuple_GET_ITEM(t, 0) respectively. The only drawback I see with this syntax is that you can't write cdef object x = PyTuple_GET_ITEM(t, 0)[0] which you'd normally do with a pointer... Stefan From dagss at student.matnat.uio.no Mon Jan 25 10:17:50 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 25 Jan 2010 10:17:50 +0100 Subject: [Cython] Support for borrowed references In-Reply-To: <4B5D4659.9070206@behnel.de> References: <4B48E1BA.7000304@behnel.de> <4B4A0D26.7070807@behnel.de> <4B5D4659.9070206@behnel.de> Message-ID: <4B5D61BE.3030901@student.matnat.uio.no> Stefan Behnel wrote: > [bump] > > Stefan Behnel, 10.01.2010 18:23: > >> [...] why not just represent a borrowed reference as a pointer? >> So you could write >> >> cdef list some_list = [] >> cdef list* borrowed_ref = some_list >> >> and borrowed_ref would be a non-refcounted pointer to a Python list. >> Assignments back to a normal reference would be allowed: >> >> cdef list my_list = borrowed_ref # increfs the pointer >> >> After all, a non-refcounted reference to a Python object is not more than a >> bare pointer to a well-defined Python builtin/extension type (including >> "object*"). >> > > What do the others think about this? > > Note that this does not relate to the "stolen reference" case, for which > Robert already proposed a "steal()" function (and which I expect to be a > rare use case anyway). It only deals with borrowed references, where e.g. > PyTuple_GET_ITEM() would be declared as > > cdef object* PyTuple_GET_ITEM(object t, Py_ssize_t index) > > and Cython would automatically do the right thing when you write > > cdef object x = PyTuple_GET_ITEM(t, 0) > > or > > cdef object* x = PyTuple_GET_ITEM(t, 0) > > respectively. The only drawback I see with this syntax is that you can't write > > cdef object x = PyTuple_GET_ITEM(t, 0)[0] > > which you'd normally do with a pointer... > Well, there's the drawback of making the language more complicated, and also I think the notation is confusing -- plain "object" is not by value, and "object*" is not a pointer to an "object", so to speak. How about a set of explicit functions like cdef object x = skip_incref(PyTuple_GET_ITEM(t, 0)) ? I don't care that much though, so I'm +0 at any rate. Dag Sverre From dagss at student.matnat.uio.no Mon Jan 25 10:29:20 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 25 Jan 2010 10:29:20 +0100 Subject: [Cython] Cython 0.12.1.alpha0 In-Reply-To: <5b8d13221001241658j1ff82a8ama9aa433c2c8146ad@mail.gmail.com> References: <5b8d13221001240516w7a3f9e62l85bb309635fae34@mail.gmail.com> <4B5C5154.7040605@behnel.de> <5b8d13221001241658j1ff82a8ama9aa433c2c8146ad@mail.gmail.com> Message-ID: <4B5D6470.7030000@student.matnat.uio.no> David Cournapeau wrote: > On Sun, Jan 24, 2010 at 10:55 PM, Stefan Behnel wrote: > >> David Cournapeau, 24.01.2010 14:16: >> >>> On Sat, Jan 23, 2010 at 4:38 AM, Robert Bradshaw >>> wrote: >>> >>>> Up at http://sage.math.washington.edu/home/robertwb/cython/Cython-0.12.1.alpha0.tar.gz >>>> . I just packaged up the current cython-devel tip, will start a Sage >>>> building with it later today. >>>> >>> It seems that it is being taken care of, but I would like to confirm: >>> will the type checking done by cython be relaxed for this release ? >>> >> I assume you are referring to the extension type size checking when >> cimporting between modules? >> > > Yes - currently, numpy 1.4.0 is not compatible with scipy 0.7.1 > binaries (at least) because of this. I would like to release 1.4.1 and > scipy 0.7.1.1 ASAP, as it is a rather major inconvenience to people > relying on binaries, > I don't think there's anything you can do in NumPy or SciPy (unless those contained Cython sources?) Basically people's Cython extensions must be recompiled with Cython 0.12.1; the NumPy/SciPy release has little to do with it. (Unless you actually refrained from breaking binary compatability in 1.4.0 because of Cython?) It certainly looks like http://hg.cython.org/cython-devel/rev/9d8b2ecef24a should have taken care of it; of course it would be good to have somebody verify that it indeed solves the exact issue with NumPy (if Robert didn't already). This certainly can be made more friendly -- I'd like to have a decorator that could be used on the NumPy classes (so that there's no warning either), and introduce an __onpxdimport__ function which could be used in numpy.pxd to make sure import_array is always called (so that NumPy will always use its own versioning scheme). I may very well not get time for before the release though. Dag Sverre From cournape at gmail.com Mon Jan 25 12:41:05 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 25 Jan 2010 20:41:05 +0900 Subject: [Cython] Cython 0.12.1.alpha0 In-Reply-To: <4B5D6470.7030000@student.matnat.uio.no> References: <5b8d13221001240516w7a3f9e62l85bb309635fae34@mail.gmail.com> <4B5C5154.7040605@behnel.de> <5b8d13221001241658j1ff82a8ama9aa433c2c8146ad@mail.gmail.com> <4B5D6470.7030000@student.matnat.uio.no> Message-ID: <5b8d13221001250341g4be47eb0lcca6ea7fe81c1a7a@mail.gmail.com> On Mon, Jan 25, 2010 at 6:29 PM, Dag Sverre Seljebotn wrote: > I don't think there's anything you can do in NumPy or SciPy (unless > those contained Cython sources?) Basically people's Cython extensions > must be recompiled with Cython 0.12.1; the NumPy/SciPy release has > little to do with it. I think there is a misunderstanding: I am only concerned with binary releases here. ATM, if you have scipy 0.7.1 installed from our installer, and you then install numpy 1.4.0 from our installer, it does not work, partly because of this cython check. Obviously, we cannot fix anything w.r.t. 3rd party extensions, cheers, David From robertwb at math.washington.edu Mon Jan 25 19:12:32 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 25 Jan 2010 10:12:32 -0800 Subject: [Cython] Cython 0.12.1.alpha0 In-Reply-To: <5b8d13221001241658j1ff82a8ama9aa433c2c8146ad@mail.gmail.com> References: <5b8d13221001240516w7a3f9e62l85bb309635fae34@mail.gmail.com> <4B5C5154.7040605@behnel.de> <5b8d13221001241658j1ff82a8ama9aa433c2c8146ad@mail.gmail.com> Message-ID: On Jan 24, 2010, at 4:58 PM, David Cournapeau wrote: > On Sun, Jan 24, 2010 at 10:55 PM, Stefan Behnel > wrote: >> >> David Cournapeau, 24.01.2010 14:16: >>> On Sat, Jan 23, 2010 at 4:38 AM, Robert Bradshaw >>> wrote: >>>> Up at http://sage.math.washington.edu/home/robertwb/cython/Cython-0.12.1.alpha0.tar.gz >>>> . I just packaged up the current cython-devel tip, will start a >>>> Sage >>>> building with it later today. >>> >>> It seems that it is being taken care of, but I would like to >>> confirm: >>> will the type checking done by cython be relaxed for this release ? >> >> I assume you are referring to the extension type size checking when >> cimporting between modules? > > Yes - currently, numpy 1.4.0 is not compatible with scipy 0.7.1 > binaries (at least) because of this. I would like to release 1.4.1 and > scipy 0.7.1.1 ASAP, as it is a rather major inconvenience to people > relying on binaries, Yes, this should be resolved (but I would recommend trying it out yourself with the alpha just to be sure it does what you want). - Robert From robertwb at math.washington.edu Mon Jan 25 19:29:47 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 25 Jan 2010 10:29:47 -0800 Subject: [Cython] Support for borrowed references In-Reply-To: <4B5D4659.9070206@behnel.de> References: <4B48E1BA.7000304@behnel.de> <4B4A0D26.7070807@behnel.de> <4B5D4659.9070206@behnel.de> Message-ID: On Jan 24, 2010, at 11:20 PM, Stefan Behnel wrote: > [bump] > > Stefan Behnel, 10.01.2010 18:23: >> [...] why not just represent a borrowed reference as a pointer? >> So you could write >> >> cdef list some_list = [] >> cdef list* borrowed_ref = some_list >> >> and borrowed_ref would be a non-refcounted pointer to a Python list. >> Assignments back to a normal reference would be allowed: >> >> cdef list my_list = borrowed_ref # increfs the pointer >> >> After all, a non-refcounted reference to a Python object is not >> more than a >> bare pointer to a well-defined Python builtin/extension type >> (including >> "object*"). > > What do the others think about this? I think the idea is good, but I'm not a fan of the syntax. - Robert From dsurviver at gmail.com Mon Jan 25 20:26:07 2010 From: dsurviver at gmail.com (Danilo Freitas) Date: Mon, 25 Jan 2010 16:26:07 -0300 Subject: [Cython] Cython 0.12.1 and 0.13 release plans In-Reply-To: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> References: <7663118D-EA53-45DA-99CE-A60D2E9F25CA@math.washington.edu> Message-ID: <45239151001251126xf9244b3q20c058dcd5fa1b19@mail.gmail.com> 2010/1/22 Robert Bradshaw : > 3. C++ support > > This has been lingering way to long. Danilo got some stuff done over > the summer, but there's still a huge amount to go. However, in the > spirit of release early, release often, the stuff that is there should > get out and get used. Last month I cleaned up a lot of stuff, and > synced up with the 0.12 release. I also wrote several tests, fixed all > the bugs they exposed, finished operator overloading support, and > improved error reporting so unimplemented stuff will give appropriate > errors rather than cause compiler crashes. Danilo also submitted a > small patch over the break. It's not to the point that one can easily > use all of STL (that was the goal, but there's no clean way to > declare, for example, vector::iterator, and references are a work > in progress, let alone stack-allocated objects (or emulations > thereof)), but it is still a huge step forward in the way C++ classes > are declared and used. > It has been a long time from the begin. I think many stuff could be done earlier, but unfortunally it wasn't. But I think we already got many stuff that can be used. -- - Danilo Freitas From dwf at cs.toronto.edu Tue Jan 26 00:20:31 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Mon, 25 Jan 2010 18:20:31 -0500 Subject: [Cython] Status of fwrap? In-Reply-To: References: Message-ID: <4B61EA75-70C8-496A-8DA0-D4AB0B2F0DB0@cs.toronto.edu> On 14-Jan-10, at 6:07 PM, Kurt Smith wrote: > Sickness, death in the family, trying to finish my dissertation -- all > cut into my discretionary time. I'll put an fwrap status post on the > blog for those who are interested. My condolences Kurt -- I had to deal with pretty much that precise combination last semester and know how taxing it can be. Good to hear that the project isn't dead, though! Good luck with the dissertation, David From robertwb at math.washington.edu Tue Jan 26 19:44:46 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 26 Jan 2010 10:44:46 -0800 Subject: [Cython] Cython 0.12.1.beta0 In-Reply-To: References: Message-ID: <59402BDB-767B-48C6-BABC-F12EDE5AF7A1@math.washington.edu> I've posted http://cython.org/release/Cython-0.12.1.beta0.tar.gz . Sage now compiles and passes all doctests. At this point it'd be good to have some widespread testing--if all goes well we'll release in a week. - Robert From robertwb at math.washington.edu Tue Jan 26 19:49:59 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 26 Jan 2010 10:49:59 -0800 Subject: [Cython] Cython release notes Message-ID: <2F66D91E-8D50-4D06-A0B6-36DD0893FB3D@math.washington.edu> If there's anything of special note that you think should be in the release notes, please submit a blurb to this thread, especially if it doesn't show up at http://trac.cython.org/cython_trac/query?group=component&milestone=0.12.1 (and in that case making a ticket about it would probably be worthwhile as well). - Robert From dagss at student.matnat.uio.no Tue Jan 26 19:57:51 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 26 Jan 2010 19:57:51 +0100 Subject: [Cython] Cython release notes In-Reply-To: <2F66D91E-8D50-4D06-A0B6-36DD0893FB3D@math.washington.edu> References: <2F66D91E-8D50-4D06-A0B6-36DD0893FB3D@math.washington.edu> Message-ID: <1a7fbda6cc5b07d69f466eb36b7e9ce5.squirrel@webmail.uio.no> Robert B. wrote: > If there's anything of special note that you think should be in the > release notes, please submit a blurb to this thread, especially if it > doesn't show up at > http://trac.cython.org/cython_trac/query?group=component&milestone=0.12.1 > (and in that case making a ticket about it would probably be > worthwhile as well). Well, I feel I'm mostly in the dark as to what exactly is in 0.12.1 with respect to type inference, so I expect some others are too - that should definitely be documented in the release notes (as a preparation to the 0.13 release as well). As I'm in the dark myself I can't write them... Dag Sverre From stefan_ml at behnel.de Wed Jan 27 08:44:10 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 27 Jan 2010 08:44:10 +0100 Subject: [Cython] Cython release notes In-Reply-To: <1a7fbda6cc5b07d69f466eb36b7e9ce5.squirrel@webmail.uio.no> References: <2F66D91E-8D50-4D06-A0B6-36DD0893FB3D@math.washington.edu> <1a7fbda6cc5b07d69f466eb36b7e9ce5.squirrel@webmail.uio.no> Message-ID: <4B5FEECA.2060802@behnel.de> Dag Sverre Seljebotn, 26.01.2010 19:57: > Robert B. wrote: >> If there's anything of special note that you think should be in the >> release notes, please submit a blurb to this thread, especially if it >> doesn't show up at >> http://trac.cython.org/cython_trac/query?group=component&milestone=0.12.1 >> (and in that case making a ticket about it would probably be >> worthwhile as well). > > Well, I feel I'm mostly in the dark as to what exactly is in 0.12.1 with > respect to type inference, so I expect some others are too Hmm, I guess the problem here is that the 0.12.x interface to type inference is a little stupid. It's off by default, you can write # cython: infer_types=None to enable safe type inference, and # cython: infer_types=True to enable complete (potentially unsafe, including ints etc.) inference. It only really makes sense in 0.13, where you use True/False to enable or disable complete type inference, and safe type inference is otherwise enabled by default. Apart from that, it's all there in 0.12.1, and especially safe type inference is worth testing (if only to make sure 0.13 won't break your code when it comes out). As discussed before, the safe mode enables type inference for all Python object types and prefers C double values over Python's float. Stefan From hoytak at gmail.com Thu Jan 28 06:51:46 2010 From: hoytak at gmail.com (Hoyt Koepke) Date: Wed, 27 Jan 2010 21:51:46 -0800 Subject: [Cython] weird error with gcc 4.4, gomp, cython, and matplotlib Message-ID: <4db580fd1001272151k54a4767ga119860c047aa636@mail.gmail.com> I'm really not sure which mailing list to report this on -- it has got to rank up there with one of the most obscure errors of all times. I suspect it's an error in gcc's new openmp implementation, gomp, but not sure; I can report it there if people think I should. In gcc 4.4.1, when I compile a completely empty .pyx module in (1) c++ mode and (2) pass -lgomp to the linker, simply (3) importing that empty module causes any subsequent use of matplotlib to segfault the program. Changing either (1), (2) or (3) makes the error go away. I'm using the latest 32bit ubuntu (9.10), python 2.6.4, with the latest cython-devel (2820:105c661599c9) and the latest matplotlib from svn (8097). In matplotlib, I'm using the qt4agg backend. Everything else appears to be working save for this error. If my main file is simply: import tests.emptymodule import tests.plottest where emptymodules is a completely empty cython module and plottest is from matplotlib.pylab import * figure() plot([0,1], [0,1], '-b') show() The program segfaults on the plot() call, with backtrace (gdb) bt 8 #0 0x00e03bc7 in __cxa_allocate_exception () from /usr/lib/libstdc++.so.6 #1 0x0093f282 in py_to_agg_transformation_matrix (obj=0x8223f58, errors=false) at src/agg_py_transforms.cpp:20 #2 0x00946ff3 in _path_module::update_path_extents (this=0x8856098, args=...) at src/path.cpp:346 #3 0x0094e2fd in Py::ExtensionModule<_path_module>::invoke_method_varargs (this=, method_def=0x8476e00, args=...) at ./CXX/Python2/ExtensionModule.hxx:184 #4 0x0093ae26 in method_varargs_call_handler (_self_and_name_tuple=0x888448c, _args=0x8f052fc) at CXX/Python2/cxx_extensions.cxx:1714 #5 0x080dc0d0 in PyEval_EvalFrameEx () #6 0x080dddf2 in PyEval_EvalCodeEx () #7 0x080dc1b4 in PyEval_EvalFrameEx () Other commands like imshow also cause a segfault. Anyway, I can get by for now without openmp, but I'm curious if anyone has ideas about this bug. I've attached a tarball with the code that reproduces it for me; simply run build.sh and run. Thanks! -- hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ -------------- next part -------------- A non-text attachment was scrubbed... Name: cython-gomp-matplotlib-error.tar.bz2 Type: application/x-bzip2 Size: 2797 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20100127/c056c38c/attachment.bin From robertwb at math.washington.edu Thu Jan 28 11:00:32 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 28 Jan 2010 02:00:32 -0800 Subject: [Cython] weird error with gcc 4.4, gomp, cython, and matplotlib In-Reply-To: <4db580fd1001272151k54a4767ga119860c047aa636@mail.gmail.com> References: <4db580fd1001272151k54a4767ga119860c047aa636@mail.gmail.com> Message-ID: <89BD7BD4-B790-4D5A-87ED-1E44580478B0@math.washington.edu> On Jan 27, 2010, at 9:51 PM, Hoyt Koepke wrote: > I'm really not sure which mailing list to report this on -- it has got > to rank up there with one of the most obscure errors of all times. I > suspect it's an error in gcc's new openmp implementation, gomp, but > not sure; I can report it there if people think I should. I'm guessing that's the issue, but just to be sure, try using the bare ----------- empty.cpp ----------------- #include "Python.h" static struct PyMethodDef methods[] = { {0, 0, 0, 0} }; PyMODINIT_FUNC initemptymodule(void) { Py_InitModule4("emptymodule", methods, 0, 0, PYTHON_API_VERSION); } ------------------------------------------- to make your empty module. > In gcc 4.4.1, when I compile a completely empty .pyx module in (1) c++ > mode and (2) pass -lgomp to the linker, simply (3) importing that > empty module causes any subsequent use of matplotlib to segfault the > program. Changing either (1), (2) or (3) makes the error go away. > > I'm using the latest 32bit ubuntu (9.10), python 2.6.4, with the > latest cython-devel (2820:105c661599c9) and the latest matplotlib from > svn (8097). In matplotlib, I'm using the qt4agg backend. Everything > else appears to be working save for this error. > > If my main file is simply: > > import tests.emptymodule > import tests.plottest > > where emptymodules is a completely empty cython module and plottest is > > from matplotlib.pylab import * > > figure() > plot([0,1], [0,1], '-b') > show() > > The program segfaults on the plot() call, with backtrace > > (gdb) bt 8 > #0 0x00e03bc7 in __cxa_allocate_exception () from /usr/lib/libstdc+ > +.so.6 > #1 0x0093f282 in py_to_agg_transformation_matrix (obj=0x8223f58, > errors=false) at src/agg_py_transforms.cpp:20 > #2 0x00946ff3 in _path_module::update_path_extents (this=0x8856098, > args=...) at src/path.cpp:346 > #3 0x0094e2fd in > Py::ExtensionModule<_path_module>::invoke_method_varargs (this= optimized out>, method_def=0x8476e00, args=...) > at ./CXX/Python2/ExtensionModule.hxx:184 > #4 0x0093ae26 in method_varargs_call_handler > (_self_and_name_tuple=0x888448c, _args=0x8f052fc) at > CXX/Python2/cxx_extensions.cxx:1714 > #5 0x080dc0d0 in PyEval_EvalFrameEx () > #6 0x080dddf2 in PyEval_EvalCodeEx () > #7 0x080dc1b4 in PyEval_EvalFrameEx () > > Other commands like imshow also cause a segfault. > > Anyway, I can get by for now without openmp, but I'm curious if anyone > has ideas about this bug. I've attached a tarball with the code that > reproduces it for me; simply run build.sh and run. > > Thanks! > -- hoyt > > ++++++++++++++++++++++++++++++++++++++++++++++++ > + Hoyt Koepke > + University of Washington Department of Statistics > + http://www.stat.washington.edu/~hoytak/ > + hoytak at gmail.com > ++++++++++++++++++++++++++++++++++++++++++ > error.tar.bz2>_______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Thu Jan 28 15:05:37 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 28 Jan 2010 15:05:37 +0100 Subject: [Cython] weird error with gcc 4.4, gomp, cython, and matplotlib In-Reply-To: <89BD7BD4-B790-4D5A-87ED-1E44580478B0@math.washington.edu> References: <4db580fd1001272151k54a4767ga119860c047aa636@mail.gmail.com> <89BD7BD4-B790-4D5A-87ED-1E44580478B0@math.washington.edu> Message-ID: <4B6199B1.2030701@student.matnat.uio.no> Robert Bradshaw wrote: > On Jan 27, 2010, at 9:51 PM, Hoyt Koepke wrote: > > >> I'm really not sure which mailing list to report this on -- it has got >> to rank up there with one of the most obscure errors of all times. I >> suspect it's an error in gcc's new openmp implementation, gomp, but >> not sure; I can report it there if people think I should. >> > > You didn't recently upgrade NumPy, did you? Appears a binary incompatability with SciPy cropped up causing a segfault in some situations. See NumPy list. I.e. if you did upgrade NumPy, rebuild *everything* of compiled code which depends on it. Dag Sverre From hoytak at gmail.com Thu Jan 28 18:30:48 2010 From: hoytak at gmail.com (Hoyt Koepke) Date: Thu, 28 Jan 2010 09:30:48 -0800 Subject: [Cython] [matplotlib-devel] weird error with gcc 4.4, gomp, cython, and matplotlib In-Reply-To: <4B619E7F.9010108@stsci.edu> References: <4db580fd1001272151k54a4767ga119860c047aa636@mail.gmail.com> <4B619E7F.9010108@stsci.edu> Message-ID: <4db580fd1001280930h445fdc3bk324fc16e1fa46b8a@mail.gmail.com> On Thu, Jan 28, 2010 at 2:00 AM, Robert Bradshaw wrote: [snip] > I'm guessing that's the issue, but just to be sure, try using the bare I still get the error with the super-bare extension module, so cython's off the hook :-). > Have you verified that the same C++ compiler and version of libstdc++ is > being used for everything? I've tried it two ways; first with everything -- numpy, scipy, matplotlib -- installed and compiled from svn, and one with them installed from the latest ubuntu repositories. I also tried it on both machines I have access to; a 64bit server with gcc 4.3.3 and python 2.6.2, and a 32bit one with gcc 4.4.1 and python 2.6.4. The error is the same on both of those. The python version and the gcc compiler are the two things in common. > If you did upgrade NumPy, rebuild *everything* of compiled code > which depends on it. Yes I did do that, and recompiled everything, but still good to check. gcc is smelling fishier and fishier (or maybe python 2.6?). -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From robertwb at math.washington.edu Thu Jan 28 20:31:21 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 28 Jan 2010 11:31:21 -0800 Subject: [Cython] Cython 0.12.1.beta0 In-Reply-To: <59402BDB-767B-48C6-BABC-F12EDE5AF7A1@math.washington.edu> References: <59402BDB-767B-48C6-BABC-F12EDE5AF7A1@math.washington.edu> Message-ID: In particular, It'd be nice if someone with MSVC to verify that http://trac.cython.org/cython_trac/ticket/473 has been killed. On Jan 26, 2010, at 10:44 AM, Robert Bradshaw wrote: > I've posted http://cython.org/release/Cython-0.12.1.beta0.tar.gz . > Sage now compiles and passes all doctests. At this point it'd be > good to have some widespread testing--if all goes well we'll release > in a week. > > - Robert From lists at cheimes.de Fri Jan 29 02:58:43 2010 From: lists at cheimes.de (Christian Heimes) Date: Fri, 29 Jan 2010 02:58:43 +0100 Subject: [Cython] [matplotlib-devel] weird error with gcc 4.4, gomp, cython, and matplotlib In-Reply-To: <4db580fd1001280930h445fdc3bk324fc16e1fa46b8a@mail.gmail.com> References: <4db580fd1001272151k54a4767ga119860c047aa636@mail.gmail.com> <4B619E7F.9010108@stsci.edu> <4db580fd1001280930h445fdc3bk324fc16e1fa46b8a@mail.gmail.com> Message-ID: Hoyt Koepke wrote: > Yes I did do that, and recompiled everything, but still good to check. > > gcc is smelling fishier and fishier (or maybe python 2.6?). Please post a simple example on the Python developer list that can reproduce the error with GCC 4.4. We have had issue with new GCC releases in the past. Sometimes new optimizations are the culprit of strange faults. You may want to play around with different -O levels, too. Sometimes the error doesn't occur with less optimizations. Please make sure you are using the correct compiler flags, especially -fno-strict-aliasing and -fwrapv. python2.6-config should give you the correct compilers flags. Christian From hoytak at gmail.com Fri Jan 29 04:27:25 2010 From: hoytak at gmail.com (Hoyt Koepke) Date: Thu, 28 Jan 2010 19:27:25 -0800 Subject: [Cython] [matplotlib-devel] weird error with gcc 4.4, gomp, cython, and matplotlib In-Reply-To: References: <4db580fd1001272151k54a4767ga119860c047aa636@mail.gmail.com> <4B619E7F.9010108@stsci.edu> <4db580fd1001280930h445fdc3bk324fc16e1fa46b8a@mail.gmail.com> Message-ID: <4db580fd1001281927l32762da0s75600baf7717fd0b@mail.gmail.com> > Please post a simple example on the Python developer list that can > reproduce the error with GCC 4.4. We have had issue with new GCC > releases in the past. Sometimes new optimizations are the culprit of > strange faults. You may want to play around with different -O levels, > too. Sometimes the error doesn't occur with less optimizations. Please > make sure you are using the correct compiler flags, especially > -fno-strict-aliasing and -fwrapv. python2.6-config should give you the > correct compilers flags. Yes, everything is compiled through python's dist utils, so I don't think I'm missing something that should be there. However, I'm no longer using some of the custom optimization flags, such as -ftree-parallelize-loops=N (and a couple of others), which require gomp to be linked into the program. Without those, the error goes away. Note that it affects both gcc 4.3 and 4.4, though I haven't tested it against 4.4.[234], just gcc 4.4.1. I'll get together a small example to post there; I have a pretty crazy academic deadline coming up on Monday, so I'll do it first thing after that. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dsurviver at gmail.com Fri Jan 29 04:34:26 2010 From: dsurviver at gmail.com (Danilo Freitas) Date: Fri, 29 Jan 2010 00:34:26 -0300 Subject: [Cython] C++ support - A simple tutorial Message-ID: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> As you know, Robert is planning to put the work about C++ Support in next release. So, we have some new stuff, and people need to learn it. I wrote a very simple tutorial on wiki [0], with some examples. I think it's easy to learn (and use) it. So, if you have any questions about it, just ask here. [0] http://wiki.cython.org/gsoc09/daniloaf/progress -- - Danilo Freitas From stefan_ml at behnel.de Fri Jan 29 11:25:13 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jan 2010 11:25:13 +0100 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> Message-ID: <4B62B789.3060803@behnel.de> Danilo Freitas, 29.01.2010 04:34: > As you know, Robert is planning to put the work about C++ Support in > next release. So, we have some new stuff, and people need to learn it. > > I wrote a very simple tutorial on wiki [0], with some examples. I > think it's easy to learn (and use) it. Thanks, there seems to be a lot in there (feature-wise). > So, if you have any questions about it, just ask here. > > [0] http://wiki.cython.org/gsoc09/daniloaf/progress One thing that strikes me: Is "new" a keyword now? I don't find that a good idea - it will break existing code at best. Why was this chosen over something like "cython.new()"? Or is it only enabled when compiling C++ code? That would be bad enough, but would at least make it less likely to break code. Reusing 'del' seems sensible, OTOH. Stefan From stefan_ml at behnel.de Fri Jan 29 11:44:35 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jan 2010 11:44:35 +0100 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> Message-ID: <4B62BC13.3020304@behnel.de> Danilo Freitas, 29.01.2010 04:34: > I wrote a very simple tutorial on wiki [0], with some examples. I > think it's easy to learn (and use) it. > > So, if you have any questions about it, just ask here. > > [0] http://wiki.cython.org/gsoc09/daniloaf/progress Regarding the comment about function overloading: """ Note that the exact function resolution depends on the notion of conversion and promotion, which for integral types may not be known until C compile time. This is ongoing work, a ticket about this will be created soon. """ I don't think it's worth investing too much time here. There is certainly a "good-enough" level somewhere, and everything else can be clarified by the users with explicit casts. I wouldn't expect 'tight' function overloading (e.g. for different int types) such a common phenomenon that Cython should go into great length to make this overly easy. Stefan From chrisde88 at yahoo.de Fri Jan 29 18:44:10 2010 From: chrisde88 at yahoo.de (Christian) Date: Fri, 29 Jan 2010 18:44:10 +0100 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> Message-ID: <4B631E6A.9040803@yahoo.de> Hi, pretty good overview. But as implied in the comment of polymorphism the example doesn't demonstrate polymorphism. Maybe you should add some lines like: cdef A *c = new B() That would be polymorphism, or not? In "Overloading Operators" there is a comment "Please avoid declaring C++ objects this way. Use 'cdef Foo *f = new Foo()'" But why should it be avoided, that doesn't make sense to me. Christian Danilo Freitas schrieb: > As you know, Robert is planning to put the work about C++ Support in > next release. So, we have some new stuff, and people need to learn it. > > I wrote a very simple tutorial on wiki [0], with some examples. I > think it's easy to learn (and use) it. > > So, if you have any questions about it, just ask here. > > [0] http://wiki.cython.org/gsoc09/daniloaf/progress > From dsurviver at gmail.com Fri Jan 29 19:05:01 2010 From: dsurviver at gmail.com (Danilo Freitas) Date: Fri, 29 Jan 2010 15:05:01 -0300 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <4B62B789.3060803@behnel.de> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62B789.3060803@behnel.de> Message-ID: <45239151001291005j2b52a3edq166ba132d2a4228f@mail.gmail.com> 2010/1/29 Stefan Behnel : > > Danilo Freitas, 29.01.2010 04:34: >> As you know, Robert is planning to put the work about C++ Support in >> next release. So, we have some new stuff, and people need to learn it. >> >> I wrote a very simple tutorial on wiki [0], with some examples. I >> think it's easy to learn (and use) it. > > Thanks, there seems to be a lot in there (feature-wise). > > >> So, if you have any questions about it, just ask here. >> >> [0] http://wiki.cython.org/gsoc09/daniloaf/progress > > One thing that strikes me: Is "new" a keyword now? I don't find that a good > idea - it will break existing code at best. Why was this chosen over > something like "cython.new()"? Or is it only enabled when compiling C++ > code? That would be bad enough, but would at least make it less likely to > break code. I didn't think about that. I just thought using 'new' like C++ would be very simple. I don't see how this keyword can break the code. But I think we can change it, if necessary. -- - Danilo Freitas From dsurviver at gmail.com Fri Jan 29 19:15:02 2010 From: dsurviver at gmail.com (Danilo Freitas) Date: Fri, 29 Jan 2010 15:15:02 -0300 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <4B631E6A.9040803@yahoo.de> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B631E6A.9040803@yahoo.de> Message-ID: <45239151001291015m67d8f8e9r7c663d50b05cfa0@mail.gmail.com> 2010/1/29 Christian : > Hi, > pretty good overview. > > But as implied in the comment of polymorphism the example doesn't > demonstrate polymorphism. Maybe you should add some lines like: Yes, it does. That is an example of 'inclusion polymorphism'. > cdef A *c = new B() > > That would be polymorphism, or not? Don't know exactly if this is polymorphism. This is an example of a cast (upcast, downcast...). But it works. I'll update the wiki with this example. > > In "Overloading Operators" there is a comment > "Please avoid declaring C++ objects this way. Use 'cdef Foo *f = new Foo()'" > > But why should it be avoided, that doesn't make sense to me. > > Christian > > > Danilo Freitas schrieb: >> As you know, Robert is planning to put the work about C++ Support in >> next release. So, we have some new stuff, and people need to learn it. >> >> I wrote a very simple tutorial on wiki [0], with some examples. I >> think it's easy to learn (and use) it. >> >> So, if you have any questions about it, just ask here. >> >> [0] http://wiki.cython.org/gsoc09/daniloaf/progress >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- - Danilo Freitas From stefan_ml at behnel.de Fri Jan 29 19:20:10 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jan 2010 19:20:10 +0100 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <45239151001291005j2b52a3edq166ba132d2a4228f@mail.gmail.com> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62B789.3060803@behnel.de> <45239151001291005j2b52a3edq166ba132d2a4228f@mail.gmail.com> Message-ID: <4B6326DA.4010003@behnel.de> Danilo Freitas, 29.01.2010 19:05: > 2010/1/29 Stefan Behnel : >> Danilo Freitas, 29.01.2010 04:34: >>> As you know, Robert is planning to put the work about C++ Support in >>> next release. So, we have some new stuff, and people need to learn it. >>> >>> I wrote a very simple tutorial on wiki [0], with some examples. I >>> think it's easy to learn (and use) it. >> Thanks, there seems to be a lot in there (feature-wise). >> >> >>> So, if you have any questions about it, just ask here. >>> >>> [0] http://wiki.cython.org/gsoc09/daniloaf/progress >> One thing that strikes me: Is "new" a keyword now? I don't find that a good >> idea - it will break existing code at best. Why was this chosen over >> something like "cython.new()"? Or is it only enabled when compiling C++ >> code? That would be bad enough, but would at least make it less likely to >> break code. > > I didn't think about that. I just thought using 'new' like C++ would > be very simple. It's not very pythonic, though. I think a function makes more sense. I assume there is a reason why calling the constructor isn't enough? > I don't see how this keyword can break the code. def factory(x, new=False): ... Let's see what Robert thinks. Stefan From dagss at student.matnat.uio.no Fri Jan 29 19:31:35 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 29 Jan 2010 19:31:35 +0100 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <4B6326DA.4010003@behnel.de> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62B789.3060803@behnel.de> <45239151001291005j2b52a3edq166ba132d2a4228f@mail.gmail.com> <4B6326DA.4010003@behnel.de> Message-ID: Stefan wrote: > > Danilo Freitas, 29.01.2010 19:05: >> 2010/1/29 Stefan Behnel : >>> Danilo Freitas, 29.01.2010 04:34: >>>> As you know, Robert is planning to put the work about C++ Support in >>>> next release. So, we have some new stuff, and people need to learn it. >>>> >>>> I wrote a very simple tutorial on wiki [0], with some examples. I >>>> think it's easy to learn (and use) it. >>> Thanks, there seems to be a lot in there (feature-wise). >>> >>> >>>> So, if you have any questions about it, just ask here. >>>> >>>> [0] http://wiki.cython.org/gsoc09/daniloaf/progress >>> One thing that strikes me: Is "new" a keyword now? I don't find that a >>> good >>> idea - it will break existing code at best. Why was this chosen over >>> something like "cython.new()"? Or is it only enabled when compiling C++ >>> code? That would be bad enough, but would at least make it less likely >>> to >>> break code. >> >> I didn't think about that. I just thought using 'new' like C++ would >> be very simple. > > It's not very pythonic, though. I think a function makes more sense. > > I assume there is a reason why calling the constructor isn't enough? I'm opposed to just calling the constructor. Everywhere else a constructor is called, a refcounted reference is returned. It is good to be very explicit about the fact that the pointer must be freed, and that this is dissimilar from instantiating Cython classes. Dag Sverre From dsurviver at gmail.com Fri Jan 29 19:39:40 2010 From: dsurviver at gmail.com (Danilo Freitas) Date: Fri, 29 Jan 2010 15:39:40 -0300 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <4B631E6A.9040803@yahoo.de> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B631E6A.9040803@yahoo.de> Message-ID: <45239151001291039g387b6a36nd7e099f9270a1652@mail.gmail.com> > In "Overloading Operators" there is a comment > "Please avoid declaring C++ objects this way. Use 'cdef Foo *f = new Foo()'" > > But why should it be avoided, that doesn't make sense to me. > I forgot answering this question. In Cython, if we wrap a class: cdef extern from "foo.h": cdef cppclass Foo: Foo(int) Cython doesn't allow you do: cdef Foo foo(10) So, at least for now, you can only instanciate non-pointer C++ objects if you have an empty contructor. We're thinking in disallow non-pointer instanciations. But I think we can allow it later. -- - Danilo Freitas From stefan_ml at behnel.de Fri Jan 29 19:43:34 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jan 2010 19:43:34 +0100 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62B789.3060803@behnel.de> <45239151001291005j2b52a3edq166ba132d2a4228f@mail.gmail.com> <4B6326DA.4010003@behnel.de> Message-ID: <4B632C56.1090307@behnel.de> Dag Sverre Seljebotn, 29.01.2010 19:31: > Stefan wrote: >> Danilo Freitas, 29.01.2010 19:05: >>> 2010/1/29 Stefan Behnel : >>>> One thing that strikes me: Is "new" a keyword now? I don't find that a >>>> good >>>> idea - it will break existing code at best. Why was this chosen over >>>> something like "cython.new()"? Or is it only enabled when compiling C++ >>>> code? That would be bad enough, but would at least make it less likely >>>> to break code. >>> I didn't think about that. I just thought using 'new' like C++ would >>> be very simple. >> It's not very pythonic, though. I think a function makes more sense. >> >> I assume there is a reason why calling the constructor isn't enough? > > I'm opposed to just calling the constructor. > > Everywhere else a constructor is called, a refcounted reference is > returned. It is good to be very explicit about the fact that the pointer > must be freed, and that this is dissimilar from instantiating Cython > classes. Fair enough. Back to the keyword question then. 'new' isn't a completely uncommon word to use in Python code: http://www.google.com/codesearch?q=^[^%23]*new[^a-z0-9_]+lang%3Apython It makes perfect sense as name of a factory method name keyword argument or (boolean) variable. I expect that a lot of people use it in their code. Making it a keyword will break all that. Stefan From robertwb at math.washington.edu Fri Jan 29 20:58:44 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 29 Jan 2010 11:58:44 -0800 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <4B62B789.3060803@behnel.de> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62B789.3060803@behnel.de> Message-ID: <81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu> On Jan 29, 2010, at 2:25 AM, Stefan Behnel wrote: > > Danilo Freitas, 29.01.2010 04:34: >> As you know, Robert is planning to put the work about C++ Support in >> next release. So, we have some new stuff, and people need to learn >> it. >> >> I wrote a very simple tutorial on wiki [0], with some examples. I >> think it's easy to learn (and use) it. > > Thanks, there seems to be a lot in there (feature-wise). > > >> So, if you have any questions about it, just ask here. >> >> [0] http://wiki.cython.org/gsoc09/daniloaf/progress > > One thing that strikes me: Is "new" a keyword now? "new" was chosen because of the symmetry between new and del (if one allocates, one needs to delete), and it would be easy for C++ people to understand. Calling the constructor directly returns an object by value, not a heap allocated one. (How to best handle objects returned by value is an open question...) Note that the lack of block-level scoping, and the c89/MSVC requirement of pre-declaring all variables at the top, makes stack allocated C classes a stickier problem, necessitating good support for new. > I don't find that a good > idea - it will break existing code at best. Why was this chosen over > something like "cython.new()"? How would cython.new be used to invoke a constructor? cdef extern from "a.h": cdef cppclass A: A(int, int) cdef *A = cython.new(A(2,3)) #? cdef *A = cython.new(A, 2, 3) #? > Or is it only enabled when compiling C++ > code? That would be bad enough, but would at least make it less > likely to > break code. I think this is an easy issue to work around--the original grammar would prohibit new as an identifier in this context anyway. - Robert From robertwb at math.washington.edu Fri Jan 29 20:59:59 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 29 Jan 2010 11:59:59 -0800 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <45239151001291015m67d8f8e9r7c663d50b05cfa0@mail.gmail.com> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B631E6A.9040803@yahoo.de> <45239151001291015m67d8f8e9r7c663d50b05cfa0@mail.gmail.com> Message-ID: <4FE31FCB-FB26-4737-9BC7-146778F07D9F@math.washington.edu> On Jan 29, 2010, at 10:15 AM, Danilo Freitas wrote: > 2010/1/29 Christian : >> Hi, >> pretty good overview. >> >> But as implied in the comment of polymorphism the example doesn't >> demonstrate polymorphism. Maybe you should add some lines like: > > Yes, it does. That is an example of 'inclusion polymorphism'. > >> cdef A *c = new B() >> >> That would be polymorphism, or not? > > Don't know exactly if this is polymorphism. This is an example of a > cast (upcast, downcast...). But it works. I'll update the wiki with > this example. Polymorphism is (loosely) about letting cdef A *a = new B() a.do_stuff() behave differently than cdef A *a = new A() a.do_stuff() - Robert From robertwb at math.washington.edu Fri Jan 29 21:03:43 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 29 Jan 2010 12:03:43 -0800 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <4B62BC13.3020304@behnel.de> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62BC13.3020304@behnel.de> Message-ID: On Jan 29, 2010, at 2:44 AM, Stefan Behnel wrote: > > Danilo Freitas, 29.01.2010 04:34: >> I wrote a very simple tutorial on wiki [0], with some examples. I >> think it's easy to learn (and use) it. >> >> So, if you have any questions about it, just ask here. >> >> [0] http://wiki.cython.org/gsoc09/daniloaf/progress > > Regarding the comment about function overloading: > > """ > Note that the exact function resolution depends on the notion of > conversion > and promotion, which for integral types may not be known until C > compile > time. This is ongoing work, a ticket about this will be created soon. > """ > > I don't think it's worth investing too much time here. There is > certainly a > "good-enough" level somewhere, and everything else can be clarified > by the > users with explicit casts. I wouldn't expect 'tight' function > overloading > (e.g. for different int types) such a common phenomenon that Cython > should > go into great length to make this overly easy. Of course the goal is to not produce non-compiling output. The caveat is just there because (1) it's undecidable in general whether something is ambiguous at Cython compile time and (2) finding what is "good enough" is probably a matter of people actually trying to use it, so we're allowing that the behavior may change in the future. - Robert From seb.binet at gmail.com Fri Jan 29 23:43:25 2010 From: seb.binet at gmail.com (Sebastien Binet) Date: Fri, 29 Jan 2010 23:43:25 +0100 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62B789.3060803@behnel.de> <81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu> Message-ID: <201001292343.25694.binet@cern.ch> On Friday 29 January 2010 20:58:44 Robert Bradshaw wrote: > > I don't find that a good > > idea - it will break existing code at best. Why was this chosen over > > something like "cython.new()"? > > How would cython.new be used to invoke a constructor? > > cdef extern from "a.h": > cdef cppclass A: > A(int, int) > > cdef *A = cython.new(A(2,3)) #? > cdef *A = cython.new(A, 2, 3) #? what about: cdef *A = cython.new(A)(2,3) ? cheers, sebastien. From njs at vorpus.org Sat Jan 30 00:12:03 2010 From: njs at vorpus.org (Nathaniel Smith) Date: Fri, 29 Jan 2010 15:12:03 -0800 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62B789.3060803@behnel.de> <81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu> Message-ID: <961fa2b41001291512wbc1866as53f25440a718bf66@mail.gmail.com> On Fri, Jan 29, 2010 at 11:58 AM, Robert Bradshaw wrote: > On Jan 29, 2010, at 2:25 AM, Stefan Behnel wrote: >> One thing that strikes me: Is "new" a keyword now? > > "new" was chosen because of the symmetry between new and del (if one > allocates, one needs to delete), and it would be easy for C++ people > to understand. Calling the constructor directly returns an object by > value, not a heap allocated one. (How to best handle objects returned > by value is an open question...) Note that the lack of block-level > scoping, and the c89/MSVC requirement of pre-declaring all variables > at the top, makes stack allocated C classes a stickier problem, > necessitating good support for new. But could someone please clarify explicitly whether the following (trivial, no fancy non-Python features) code will work in the C++ branch? def new(x): return myobj(x) foo = new(1) ? Note that this would fail to compile if the function were named "if" or "cdef", etc., but with the name "new" it is currently legal. > How would cython.new be used to invoke a constructor? > > cdef extern from "a.h": > ? ? cdef cppclass A: > ? ? ? ? A(int, int) > > cdef *A = cython.new(A(2,3)) #? -1 > cdef *A = cython.new(A, 2, 3) #? +0 >> Or is it only enabled when compiling C++ >> code? That would be bad enough, but would at least make it less >> likely to >> break code. > > I think this is an easy issue to work around--the original grammar > would prohibit new as an identifier in this context anyway. Can you give more details on what you mean here? -- Nathaniel From robertwb at math.washington.edu Sat Jan 30 01:31:38 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 29 Jan 2010 16:31:38 -0800 Subject: [Cython] C++ support - A simple tutorial In-Reply-To: <961fa2b41001291512wbc1866as53f25440a718bf66@mail.gmail.com> References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> <4B62B789.3060803@behnel.de> <81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu> <961fa2b41001291512wbc1866as53f25440a718bf66@mail.gmail.com> Message-ID: On Jan 29, 2010, at 3:12 PM, Nathaniel Smith wrote: > On Fri, Jan 29, 2010 at 11:58 AM, Robert Bradshaw > wrote: >> On Jan 29, 2010, at 2:25 AM, Stefan Behnel wrote: >>> One thing that strikes me: Is "new" a keyword now? >> >> "new" was chosen because of the symmetry between new and del (if one >> allocates, one needs to delete), and it would be easy for C++ people >> to understand. Calling the constructor directly returns an object by >> value, not a heap allocated one. (How to best handle objects returned >> by value is an open question...) Note that the lack of block-level >> scoping, and the c89/MSVC requirement of pre-declaring all variables >> at the top, makes stack allocated C classes a stickier problem, >> necessitating good support for new. > > But could someone please clarify explicitly whether the following > (trivial, no fancy non-Python features) code will work in the C++ > branch? > > def new(x): > return myobj(x) > foo = new(1) > > ? Note that this would fail to compile if the function were named "if" > or "cdef", etc., but with the name "new" it is currently legal. This will still be legal--I'll even make sure there's a regression test for it. >> How would cython.new be used to invoke a constructor? >> >> cdef extern from "a.h": >> cdef cppclass A: >> A(int, int) >> >> cdef *A = cython.new(A(2,3)) #? > > -1 > >> cdef *A = cython.new(A, 2, 3) #? > > +0 > >>> Or is it only enabled when compiling C++ >>> code? That would be bad enough, but would at least make it less >>> likely to >>> break code. >> >> I think this is an easy issue to work around--the original grammar >> would prohibit new as an identifier in this context anyway. > > Can you give more details on what you mean here? There's no way to interpret the expression new A(1,10) as valid Python, so there's no ambiguity. - Robert From robertwb at math.washington.edu Sun Jan 31 02:56:48 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 30 Jan 2010 17:56:48 -0800 Subject: [Cython] Cython 0.12.1.beta0 In-Reply-To: <59402BDB-767B-48C6-BABC-F12EDE5AF7A1@math.washington.edu> References: <59402BDB-767B-48C6-BABC-F12EDE5AF7A1@math.washington.edu> Message-ID: <5EAA320C-9340-4B27-8DFA-25BE625157E6@math.washington.edu> On Jan 26, 2010, at 10:44 AM, Robert Bradshaw wrote: > I've posted http://cython.org/release/Cython-0.12.1.beta0.tar.gz . > Sage now compiles and passes all doctests. At this point it'd be > good to have some widespread testing--if all goes well we'll release > in a week. Given the apparent lack of issues with the beta (which is a good thing I hope), I've posted http://cython.org/release/ Cython-0.12.1.rc1.tar.gz which, unless something unexpected comes up, will be the actual release. One last thing we need to resolve is http://trac.cython.org/cython_trac/ticket/387 (the doctesting support via __test__). So far it hasn't seemed to be causing any problems, should we just leave it on and close that ticket? I agree the implementation could be improved. - Robert Preliminary release notes: http://wiki.cython.org/ReleaseNotes-0.12.1 = Cython 0.11.2 = * Release date: February X, 2010 I'm happy to announce the release of Cython 0.12.1. http://cython.org/release/Cython-0.12.1.rc1.tar.gz == New Features == * Type inference improvements. There have been several bug fixes and improvements to the type inferencer. Notably, there is now a "safe" mode enabled by setting the infer_types directive to None. (The None here refers to the "default" mode, which will be the default in 0.13.) This safe mode limits inference to Python object types and C doubles, which should speed up execution without affecting any semantics such as integer overflow behavior like infer_types=True might. There is also a verbose which allows one to see what types are inferred. * The boundscheck directive works for lists and tuples as well as buffers. * len(s) and s.decode("encoding") are efficiently supported for char* s. * Cython's INLINE macro has been renamed to CYTHON_INLINE to reduce conflict and has better support for Windows. * Revision history is now omitted from the source package, resulting in a 85% size reduction. Typing make repo will download the this history and turn the directory into a complete working repository. * Cython modules don't need to be recompiled when the size of an external type grows. (A warning, rather than an error, is produced.) This should be helpful for binary distributions relying on NumPy. Several other bugs and minor improvements have been made. This release should be fully backwards compatible with 0.12. == Contributors to this release == * Arfrever Frehtes Taifersar Arahesis * Stefan Behnel * Robert Bradshaw * Lisandro Dalcin * Julien Danjou * Mark Lodato * Dag Sverre Seljebotn Thanks also to everybody who's helping us out in our discussions on the mailing list. http://trac.cython.org/cython_trac/query?group=component&milestone=0.12.1 From stefan_ml at behnel.de Sun Jan 31 08:05:02 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 31 Jan 2010 08:05:02 +0100 Subject: [Cython] Cython 0.12.1.beta0 In-Reply-To: <5EAA320C-9340-4B27-8DFA-25BE625157E6@math.washington.edu> References: <59402BDB-767B-48C6-BABC-F12EDE5AF7A1@math.washington.edu> <5EAA320C-9340-4B27-8DFA-25BE625157E6@math.washington.edu> Message-ID: <4B652B9E.3080905@behnel.de> Robert Bradshaw, 31.01.2010 02:56: > On Jan 26, 2010, at 10:44 AM, Robert Bradshaw wrote: > >> I've posted http://cython.org/release/Cython-0.12.1.beta0.tar.gz . >> Sage now compiles and passes all doctests. At this point it'd be >> good to have some widespread testing--if all goes well we'll release >> in a week. > > Given the apparent lack of issues with the beta (which is a good thing > I hope), I've posted http://cython.org/release/ > Cython-0.12.1.rc1.tar.gz which, unless something unexpected comes up, > will be the actual release. Latest cython-devel, it seems. Fine with me. > One last thing we need to resolve is http://trac.cython.org/cython_trac/ticket/387 > (the doctesting support via __test__). So far it hasn't seemed to be > causing any problems, should we just leave it on and close that > ticket? I agree the implementation could be improved. Dag should comment on this, but my take is, if it works, it works. We can always revisit the implementation once the infrastructure is there. (The ticket mentions CythonUtilityCode, I assume that's a way to implement utility functions in Cython rather than C?) > Preliminary release notes: http://wiki.cython.org/ReleaseNotes-0.12.1 > > > = Cython 0.11.2 = > * Release date: February X, 2010 Agreed that January 31 is a bit too close by now, but early next week should be fine (assuming the week starts on Monday, which it doesn't do everywhere). We've waited long enough during the alpha/beta cycle. > * Type inference improvements. > > There have been several bug fixes and improvements to the type > inferencer. > Notably, there is now a "safe" mode enabled by setting the > infer_types directive to None. > (The None here refers to the "default" mode, which will be the > default in 0.13.) I think that explanation is needed, but also sufficient. I added some fixes in the Wiki. Stefan From dagss at student.matnat.uio.no Sun Jan 31 10:49:09 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 31 Jan 2010 10:49:09 +0100 Subject: [Cython] Cython 0.12.1.beta0 In-Reply-To: <4B652B9E.3080905@behnel.de> References: <59402BDB-767B-48C6-BABC-F12EDE5AF7A1@math.washington.edu> <5EAA320C-9340-4B27-8DFA-25BE625157E6@math.washington.edu> <4B652B9E.3080905@behnel.de> Message-ID: Stefan Behnel wrote: > Robert Bradshaw, 31.01.2010 02:56: >> One last thing we need to resolve is >> http://trac.cython.org/cython_trac/ticket/387 >> (the doctesting support via __test__). So far it hasn't seemed to be >> causing any problems, should we just leave it on and close that >> ticket? I agree the implementation could be improved. > > Dag should comment on this, but my take is, if it works, it works. We can > always revisit the implementation once the infrastructure is there. (The > ticket mentions CythonUtilityCode, I assume that's a way to implement > utility functions in Cython rather than C?) OK, I closed the ticket and made a new one for implementation improvement on the wishlist milestone. http://trac.cython.org/cython_trac/ticket/497 Dag Sverre