#jython IRC Log (v0.9)

Index

IRC Log for 2015-03-04

Timestamps are in GMT/BST.

[0:22] <peke> jimbaker: yes, jline 2.12.1 seems to have fixed the problem with backslash and other characters that require altgr on finnish locale.
[0:22] <peke> thanks also for submitting the issue about runtime instrumentation.
[0:23] <peke> i doubt i can easily help implementing that issue (http://bugs.jython.org/issue2278)
[0:23] <peke> but i'll definitely try this functionality if it is available!
[0:46] * fwierzbicki (~Adium@99-106-169-5.lightspeed.sntcca.sbcglobal.net) has joined #jython
[0:49] * fwierzbicki2 (~Adium@99-106-169-5.lightspeed.sntcca.sbcglobal.net) Quit (Ping timeout: 244 seconds)
[0:53] * fwierzbicki (~Adium@99-106-169-5.lightspeed.sntcca.sbcglobal.net) Quit (Quit: Leaving.)
[1:03] * fwierzbicki (~Adium@99-106-169-5.lightspeed.sntcca.sbcglobal.net) has joined #jython
[1:05] * fwierzbicki (~Adium@99-106-169-5.lightspeed.sntcca.sbcglobal.net) Quit (Client Quit)
[1:06] * fwierzbicki (~Adium@99-106-169-5.lightspeed.sntcca.sbcglobal.net) has joined #jython
[1:09] * fwierzbicki1 (~Adium@99-106-169-5.lightspeed.sntcca.sbcglobal.net) has joined #jython
[1:11] * fwierzbicki (~Adium@99-106-169-5.lightspeed.sntcca.sbcglobal.net) Quit (Ping timeout: 272 seconds)
[1:20] * FunkyELF (~FunkyELF@2606:a000:d140:6000:8c5:eeb8:4e1c:3b22) has joined #jython
[1:21] <FunkyELF> jimbaker: you still there? This is ELFrederich but now at home
[1:21] <FunkyELF> I monkeypatched the __tojava__ method of the class I need to get coerced into a java array.
[1:21] <FunkyELF> the __tojava__ method is getting called with these params:
[1:21] <FunkyELF> (<netref class '__builtin__.list'>, '[Foo@22802495, Foo@59e8d980, Foo@424f5762]') and (<type 'java.lang.Class'>, "<type '[LFoo;'>")
[1:22] <FunkyELF> thats the type(x) and repr(x) for self and java_type
[1:38] <FunkyELF> jimbaker: ... got it working: http://paste.pound-python.org/show/59FWaCeqiXLO8LZwigYV/
[1:38] <FunkyELF> I'm sure it could be better. I should be using the java_type parameter somewhere right?
[2:14] <FunkyELF> jimbaker: trying to look at the source code for Jython. Where does coersion take place for plain Python lists? When I call a method that is supposed to take a Java array and I pass it a Jython list, where is that coercion?
[2:22] <jimbaker> FunkyELF, so that conversion is managed by https://github.com/jythontools/jython/blob/master/src/org/python/core/PySequence.java#L432
[2:23] <jimbaker> PyList extends PySequenceList, which extends PySequence
[2:24] <FunkyELF> jimbaker: great thanks. I'll probably need to do something like that __tojava__ but from Jython
[2:24] <jimbaker> should be quite doable
[2:25] <jimbaker> FunkyELF, as you can see, it's pretty straightforward code
[2:25] <FunkyELF> jimbaker: it is... but I appreciate your help. Been a while since I've actually done any Java coding
[2:26] <FunkyELF> its a lot of cognitive work to get Java and Python in your head at the same time
[2:26] <jimbaker> FunkyELF, no problem. i agree about the cognitive load, but i guess i'm somewhat used to it by now ;)
[2:28] <FunkyELF> jimbaker: I wrote a Python library that exposes Python classes / objects to Tcl
[2:28] <FunkyELF> lots of overhead there. I've also wrote code that parses .h files and generates Cython code to create bindings. A lot of stuff to keep in your head
[2:28] <jimbaker> hmmm... i guess tcl is some ways is easier, but it is annoyingly limited as well. but it's been a long time since i used it
[2:29] <FunkyELF> jimbaker: it wasn't much work at all... I have a pretty sophisticated system in about 300 lines of Python code. There is already Tkinter which allows you to bind Tcl commands to Python callables
[2:29] <jimbaker> FunkyELF, so i think the one good thing re java when working with jython is that we have done a lot of work on creating semantic equivalence. the tough thing with java is the huge library, and at times mapping it to c equivalents
[2:30] <FunkyELF> all I did really was the object introspection to expose things at the class / object level rather than just commands
[2:31] <jimbaker> FunkyELF, makes sense, and that's how you can create that magic of semantic equivalence. that is, working with java objects looks like working with their python equivalents
[2:32] <FunkyELF> this line number you pointed me at... too bad there isn't a way for me to call this on my netref object
[2:33] <FunkyELF> jimbaker: I'm sure now that I can handle this by monkeypatching __tojava__ on my netref for __builtin__.list ... but I have to wonder what a better solution would be
[2:34] <FunkyELF> would it be on RPyC to implement __tojava__ directly, or would it be for Jython to try to dynamically see if an object is a sequence
[2:35] <FunkyELF> this netref to the list builtin surely implements __iter__, if that is the case couldn't Jython do some magic there? It only coerces PySequence to arrays, not any iterable Python object
[2:36] <jimbaker> FunkyELF, perhaps we could expand this coercion scope
[2:36] <jimbaker> it's a pretty frequent thing for us to do
[2:37] <FunkyELF> jimbaker: yeah... Maybe I'll look at doing it
[2:37] <jimbaker> re PySequence#__tojava__, looks like we don't expose it. i suppose that's generally true for __tojava__ methods that are actually written in java, so you cannot directly get at it
[2:38] <FunkyELF> jimbaker: there must be some giant test suite that I could run to see if it breaks anything
[2:38] <jimbaker> FunkyELF, we always appreciate a patch. and even better, a test to go with it
[2:38] <jimbaker> yes, it's the standard python regrtest.py
[2:38] <jimbaker> you can run it with ant regrtest
[2:39] <jimbaker> or jython Lib/test/regrtest.py
[2:39] <jimbaker> it's of course somewhat modified for jython, but we try to use as much of cpython's tests unmodified as possible
[2:39] <jimbaker> again, i think the latest commit on datetime shows this pretty well
[2:40] <jimbaker> eg https://hg.python.org/jython/rev/67210f355576#l2.14
[2:40] <jimbaker> generally we prefer test cases written in python, although stuff that's testing java integration occasionally should be written in java junit
[2:40] <FunkyELF> possible to import PyArray from Jython?
[2:41] <jimbaker> absolutely. i do that sort all of the time
[2:41] <FunkyELF> from org.python.core import PyArray
[2:41] <FunkyELF> I'm just trying to do a 1:1 from this PySequence.__tojava__
[2:43] <jimbaker> you might look at https://github.com/jythontools/jython/blob/master/Lib/jythonlib.py
[2:43] <jimbaker> and https://github.com/jythontools/jython/blob/master/Lib/weakref.py
[2:43] <jimbaker> for some examples
[2:45] <FunkyELF> array = PyArray(component, len(self))
[2:45] <jimbaker> if you grep for "from org.python.core import" in Lib, you will see other examples of this in practice
[2:45] <FunkyELF> TypeError: 'int' object is not iterable
[2:45] <FunkyELF> seems likt it is trying to call into the wrong constructor of PyArray
[2:46] <jimbaker> you can control this by boxing. from java.lang import Integer
[2:47] <jimbaker> then something like array = PyArray(component, Integer(len(self))) should work
[2:47] <jimbaker> but should be able to use the fact that PyArray equals array.array
[2:47] <jimbaker> so just use the regular array module
[2:47] <jimbaker> https://docs.python.org/2/library/array.html
[2:48] <jimbaker> unless there's a constructor that we are not exposing for some reason
[2:48] <jimbaker> but usually the constructors in something like PyArray are to handle issues like subclassing
[2:49] <FunkyELF> TypeError: 'java.lang.Integer' object is not iterable
[2:49] <FunkyELF> hmm... I'll look at the regular array module
[2:49] <jimbaker> so we have this whole expose mechanism to wrap java classes so they work as python classes
[2:49] <jimbaker> yeah, i think you will have better luck
[2:51] <jimbaker> so some interesting complexity. it all fits together however. but the fact that python code in jython can directly access the jython runtime is rather powerful, and we certainly use its leverage
[2:51] <FunkyELF> http://paste.pound-python.org/show/hKRdemCGC7iysODYNsKK/
[2:51] <FunkyELF> thats where I'm getting that Integer object is not iterable error... but I'll look at the array module
[2:52] <FunkyELF> jimbaker: where do you work?
[2:52] <jimbaker> FunkyELF, i'm at rackspace
[2:53] <FunkyELF> cool
[2:54] <FunkyELF> Don't think I can use the regular array module. I don't see a type code for object
[2:55] <jimbaker> FunkyELF, ahh, right. you can use jarray instead then. there's a reason we still have it
[2:56] <jimbaker> internally it's the same thing, just exposed to python somewhat differently
[2:56] <jimbaker> although i suspect that array.array does take a typecode for object. don't recall off the top of my head of course
[2:57] <FunkyELF> well... I'd like to use PyArray, wonder why I can't call the correct constructor
[2:57] <FunkyELF> I guess even when I call it with an Integer the constructor that takes "Object" picks up the Integer
[2:58] <FunkyELF> ValueError: bad typecode (must be c, b, B, u, h, H, i, I, l, L, f or d)
[2:59] <FunkyELF> nope... all accounted for
[2:59] <jimbaker> supply it the class, not a typecode letter
[2:59] <jimbaker> sorry, undocumented stuff here ;)
[2:59] <jimbaker> eg, x = array.array(BigDecimal)
[3:01] <jimbaker> again, it's because the array module and the jarray module are effectively the same thing, just slightly different wrappers of PyArray
[3:01] <FunkyELF> okay
[3:01] <jimbaker> we should have better docs. but i rather get the RC out first
[3:03] <FunkyELF> ;-)
[3:03] <jimbaker> a little bit here, http://www.jython.org/jythonbook/en/1.0/DataTypes.html#jython-specific-collections, but still not advanced enough
[3:04] <FunkyELF> jimbaker: looks like I'm set except I don't know what to raise if I can't do the conversion. http://paste.pound-python.org/show/BRhD4EflToReRXh76R6k/
[3:04] <jimbaker> Py.NoConversion
[3:05] <jimbaker> so that's used here - https://github.com/jythontools/jython/blob/master/Lib/datetime.py#L1075
[3:05] <jimbaker> and just imported from org.python.core
[3:05] <FunkyELF> okay... was wondering where it was from. now I think I'm all set
[3:05] <jimbaker> maybe you can write a blog post to capture this knowledge :)
[3:06] <jimbaker> awesome, glad i could help!
[3:06] <FunkyELF> ahh... I return it, not raise it
[3:07] <jimbaker> exactly. exceptions are cheap. checking return values is even cheaper
[3:07] <FunkyELF> jimbaker: this was a hurdle, I can relax a little bit now, but I'd like to maybe try to get any iterable coerced properly to arrays so that things like this wouldn't be an issue
[3:07] <FunkyELF> jimbaker: don't get me started on that ;-)
[3:08] <jimbaker> i think that's very reasonable to support iterator/iterable conversion to array
[3:09] <FunkyELF> can you see this image? https://bitbucket.org/ericfrederich/exception_problems
[3:10] <jimbaker> FunkyELF, yes, looks good
[3:10] <jimbaker> or complicated :)
[3:10] <jimbaker> FunkyELF, so here's what we need to change - https://github.com/jythontools/jython/blob/master/src/org/python/core/PyIterator.java#L70
[3:10] <jimbaker> just need to support coercing to array
[3:11] <jimbaker> simple enough java patch
[3:12] <FunkyELF> yeah... when I created Cython bindings to some C library. The library used return codes and I convert them to Python exceptions... but then need to convert them back into return codes when it goes back to C.
[3:12] <FunkyELF> when we have code that triggers other code via callbacks and other stuff we weren't getting the full stack trace, just the most recent one... not all the way back. I had to create that diagram to see what the hell was going on
[3:14] <jimbaker> FunkyELF, http://bugs.jython.org/issue2279
[3:14] <FunkyELF> jimbaker: so handling it at PyIterator level would work because when these __tojava__ methods can't do the conversion they call super.__tojava__ ?
[3:15] <jimbaker> FunkyELF, the inheritance hierarchy is not so simple unfortunately
[3:16] <jimbaker> but we can generalize so that the user experience is like this
[3:18] <FunkyELF> jimbaker: if this particular netref class isn't a subclass of PyIterator I'm not sure why that __tojava__ method would be called
[3:19] <FunkyELF> but then again... I've only looked at Jython source code for a couple hours
[3:20] <jimbaker> FunkyELF, i think you would want it for PySequence, and other iterables. PyObject is too root unfortunately
[3:21] <FunkyELF> jimbaker: I think the problem is that whats coming over the wire gets created as just a netref object which is just a subclass of object
[3:21] <FunkyELF> ... it just happens to be an object that supports the iteration protocol
[3:22] <jimbaker> FunkyELF, maybe that suggests we should teach RPyC about this support?
[3:22] <jimbaker> i haven't looked at its codebase however
[3:23] <FunkyELF> jimbaker: where does it stop though?... you can't have every project out there add a __tojava__ method
[3:24] <jimbaker> well, RPyC is sort of unique in this way
[3:24] <jimbaker> anyway, i think the thing is, figure out a point solution, as you have done; then see what can be generalized
[3:24] <FunkyELF> I guess most other code would subclass list or something rather than implement __len__, __iter__, etc
[3:25] <FunkyELF> jimbaker: I agree. now that I understand a little more I will look into it
[3:25] <jimbaker> FunkyELF, here's another thought - to me this suggests we could leverage abstract base class support
[3:28] <FunkyELF> I wonder if this netref class could be defined in a different way that Jython would know to create it as a list subclass
[3:29] <FunkyELF> https://github.com/tomerfiliba/rpyc/blob/master/rpyc/core/netref.py
[3:29] <FunkyELF> its created using metaprogramming
[3:29] <FunkyELF> created at the bottom from class_factory
[3:35] <jimbaker> FunkyELF, right, it's some nice magic. at the very least, you could inject __tojava__ into the netref itself
[3:35] <FunkyELF> why is PyObject too root? Where does the __tojava__ stuff start getting called? is it top down or bottom up?
[3:35] <jimbaker> PyObject would be too root for converting in general to array
[3:36] <FunkyELF> jimbaker: but you'd only be selectively converting things that have __iter__, __len__, and __getitem__
[3:36] <jimbaker> FunkyELF, so i think one thing that's possibly missing here is that the netref stuff doesn't see __tojava__ for almost all classes
[3:36] <FunkyELF> I just threw a System.out.println in the __tojava__ of PyObject. Its being called all over the place as it is
[3:37] <jimbaker> eg, it would work for datetime.datetime or threading.Thread because they __tojava__ defined in python
[3:37] <jimbaker> but not for list, etc
[3:37] <jimbaker> FunkyELF, it certainly is
[3:37] <jimbaker> just a question of what it converts to
[3:38] <jimbaker> FunkyELF, so maybe the answer is to expose __tojava__
[3:38] <jimbaker> then proxies could have access to it
[3:41] <jimbaker> that's also an easy experiment
[3:43] <FunkyELF> but then the work would have to be done on each proxy
[3:43] <FunkyELF> http://paste.pound-python.org/show/sbrPap9mS2VDdqaV2CGU/
[3:44] <FunkyELF> for example... that is pure Jython code that doesn't have any proxies and the call to b.blabla doesn't work
[3:45] <jimbaker> FunkyELF, my point is that it would not have to be done for proxies that do this sort of introspection, https://github.com/tomerfiliba/rpyc/blob/master/rpyc/core/netref.py#L215
[3:46] <FunkyELF> not sure I follow
[3:47] <FunkyELF> I would favor any solution where Jython doesn't specifically know about RPyC and RPyC doesn't specifically know about Jython
[3:48] <jimbaker> FunkyELF, that's my thought here. the composition pattern used here means __tojava__ is not called through
[3:48] <jimbaker> but the proxy does introspect for methods
[3:48] <jimbaker> so maybe we should help it find them
[3:48] <jimbaker> anyway, it's just an experiment. i don't know if this might break things. hopefully not
[3:49] <jimbaker> presumably most python code doesn't care what __xyz__ methods are defined
[3:49] <FunkyELF> but I don't think that it would have the side effect of having that code I pasted above start working
[3:54] <jimbaker> FunkyELF, true - this would require abstract base class support as well
[3:55] <jimbaker> if it looks like a sequence, treat it as one from the purposes of java conversion
[3:55] <jimbaker> semantic equivalence is still a work in progress :)
[3:56] <FunkyELF> copied / pasted the "if (c.isArray())" from PySequece.java to the bottom of PyObject.java. Just one change (had to call __getitem__ rather than pyget). Not it works.
[3:56] <FunkyELF> ... but its up to you if there is a better design than handling this at the PyObject level. I just wanted to test it
[3:59] <FunkyELF> *now it works
[4:00] <jimbaker> FunkyELF, worth trying, but i am skeptical it can be done in general. it
[4:00] <jimbaker> is certainly a big change
[4:01] <jimbaker> but i believe in doing experiments to assess scope, that's for sure!
[4:01] <FunkyELF> but the change is essentially how Python works... if it looks like a duck and acts like a duck, treat it like a duck
[4:01] <FunkyELF> how would you do it with abstract base classes?
[4:02] <jimbaker> ack. and maybe it works because special cases are already handled specially in their respective __tojava__. eg PyString
[4:02] <jimbaker> and even then, it still can make sense
[4:03] <jimbaker> so first, let's try this with ant regrtest. see what fails. if nothing does, that's very interesting
[4:03] <FunkyELF> [exec] test_isinstance
[4:04] <FunkyELF> seems alphabetical
[4:04] <FunkyELF> I'm on "i"
[4:04] <FunkyELF> ...just hit "j" which is probably a big one because of "J"ava
[4:04] <jimbaker> yes, it's lexicographic
[4:08] <FunkyELF> jimbaker: got an AssertionError but I'm not sure if I'd be getting it without my change
[4:09] <jimbaker> FunkyELF, there are a few failures we are seeing on linux now because of netty upgrade
[4:09] <FunkyELF> http://paste.pound-python.org/show/tj1O7MBVGuMTPEJtB6oV/
[4:09] <jimbaker> i need to fix the ssl handshake under certain states
[4:09] <jimbaker> yep, that's known
[4:10] <jimbaker> sort of uninteresting
[4:10] <FunkyELF> and now the ssl one just came up
[4:10] <jimbaker> hard to write good tests for sockets. but the ssl failure is valid, and should be fixed
[4:10] <jimbaker> per how handshaking is supposed to work
[4:11] <jimbaker> basically we don't wrap the exception properly iirc
[4:11] <jimbaker> it still is not terribly interesting - it happens because of a close on one side during handshaking
[4:12] <FunkyELF> jimbaker: about time for me to go to bed... I created a pull request on GitHub in case you wanted to play with regrtest
[4:12] <jimbaker> FunkyELF, thanks. will take a look. fun talking with you about this topic!
[4:13] <FunkyELF> jimbaker: yes... thanks again. I learned a lot
[4:13] <FunkyELF> I'll probably be back tomorrow at work as ELFrederich
[4:14] * FunkyELF (~FunkyELF@2606:a000:d140:6000:8c5:eeb8:4e1c:3b22) has left #jython
[9:25] * mbooth (~mbooth@cpc11-shef10-2-0-cust659.barn.cable.virginm.net) has joined #jython
[9:25] * mbooth (~mbooth@cpc11-shef10-2-0-cust659.barn.cable.virginm.net) Quit (Changing host)
[9:25] * mbooth (~mbooth@redhat/mbooth) has joined #jython
[10:04] * Trundle (~andy@python/site-packages/trundle) has joined #jython
[12:41] <ELFrederich> ... and nobody has been here since ;-)
[13:59] * xemdetia (xemdetia@nat/ibm/x-olzpueudulwmbxpa) has joined #jython
[14:34] * enebo (~enebo@c-75-73-8-169.hsd1.mn.comcast.net) has joined #jython
[15:19] * dustinm (~dustinm@2607:5300:100:200::160d) Quit (Ping timeout: 250 seconds)
[15:25] * dustinm (~dustinm@105.ip-167-114-152.net) has joined #jython
[15:47] * peke (peke@kapsi.fi) Quit (Ping timeout: 265 seconds)
[15:47] * grey_ (~foo@kapsi.fi) Quit (Ping timeout: 265 seconds)
[15:54] * grey_ (eaaltone@kapsi.fi) has joined #jython
[16:04] * peke (peke@kapsi.fi) has joined #jython
[17:18] * viablepanic (~showboat@c-83-233-149-82.cust.bredband2.com) has joined #jython
[17:18] * viablepanic (~showboat@c-83-233-149-82.cust.bredband2.com) Quit (Read error: Connection reset by peer)
[17:18] * viablepanic (~showboat@c-83-233-149-82.cust.bredband2.com) has joined #jython
[17:27] * viablepanic (~showboat@c-83-233-149-82.cust.bredband2.com) Quit (Quit: viablepanic)
[17:27] * viablepanic (~showboat@c-83-233-149-82.cust.bredband2.com) has joined #jython
[17:29] * viablepanic (~showboat@c-83-233-149-82.cust.bredband2.com) Quit (Client Quit)
[17:47] * nemopanic (~slu@code.vatten.slu.se) has joined #jython
[17:53] * srcerer (~chatzilla@dns2.klsairexpress.com) Quit (Quit: ChatZilla 0.9.91.1 [Firefox 35.0.1/20150122214805])
[18:00] * nemopanic (~slu@code.vatten.slu.se) Quit (Quit: That's it for today)
[18:12] * srcerer (~chatzilla@dns2.klsairexpress.com) has joined #jython
[19:39] * Trundle (~andy@python/site-packages/trundle) Quit (Remote host closed the connection)
[20:37] * ELFrederich (~ELFrederi@12.54.94.28) Quit (Remote host closed the connection)
[20:40] * Trundle (~andy@python/site-packages/trundle) has joined #jython
[22:25] * Trundle (~andy@python/site-packages/trundle) Quit (Remote host closed the connection)
[23:21] * xemdetia (xemdetia@nat/ibm/x-olzpueudulwmbxpa) Quit (Ping timeout: 265 seconds)
[23:54] * enebo (~enebo@c-75-73-8-169.hsd1.mn.comcast.net) Quit (Quit: enebo)

Index

These logs were automatically created by JythonLogBot_ on irc.freenode.net using a slightly modified version of the Java IRC LogBot (github).