Timestamps are in GMT/BST.
[0:48] * robbyoconnor (~wakawaka@guifications/user/r0bby) Quit (Ping timeout: 240 seconds)
[1:26] * clajo04 (~clajo04@pool-108-21-222-14.nycmny.fios.verizon.net) Quit (Quit: clajo04)
[2:56] * lheuer1 (~Adium@f048164205.adsl.alicedsl.de) has joined #jython
[2:58] * lheuer (~Adium@unaffiliated/lheuer) Quit (Ping timeout: 252 seconds)
[4:11] * robbyoconnor (~wakawaka@guifications/user/r0bby) has joined #jython
[4:11] <topi`> jimbaker: I remember having an odd issue with jyjdbc at some point. It was requiring some "iijdbc" jar or something like that.
[4:12] <topi`> damn, woke up way too early and cannot get sleep any more
[4:13] <topi`> I might just as well try to get jyjdbc in place of zxjdbc :)
[4:24] * robbyoconnor (~wakawaka@guifications/user/r0bby) Quit (Read error: Operation timed out)
[4:28] * robbyoconnor (~wakawaka@guifications/user/r0bby) has joined #jython
[4:36] * lheuer1 (~Adium@f048164205.adsl.alicedsl.de) Quit (Quit: Leaving.)
[4:55] <jimbaker> topi`, good choice, my best hacking sometimes happens under that circumstances ;)
[5:07] <jimbaker> topi`, also thanks for the further twisted investigation - just looked at https://twistedmatrix.com/trac/ticket/3413#comment:39
[5:11] * r0bby (~wakawaka@guifications/user/r0bby) has joined #jython
[5:12] * robbyoconnor (~wakawaka@guifications/user/r0bby) Quit (Ping timeout: 264 seconds)
[5:23] * r0bby (~wakawaka@guifications/user/r0bby) Quit (Quit: Konversation terminated!)
[7:32] * agronholm (~agronholm@2001:1bc8:102:6f29:1c2:7dae:d253:37b) Quit (Ping timeout: 264 seconds)
[7:35] * agronholm (~agronholm@2001:1bc8:102:6f29:dc32:32a9:59b:c7ad) has joined #jython
[7:35] * ChanServ sets mode +o agronholm
[7:38] * lheuer (~Adium@f048164205.adsl.alicedsl.de) has joined #jython
[7:38] * lheuer (~Adium@f048164205.adsl.alicedsl.de) Quit (Changing host)
[7:38] * lheuer (~Adium@unaffiliated/lheuer) has joined #jython
[13:52] * zz_whg_away is now known as whg
[14:08] * enebo (~enebo@c-75-73-8-169.hsd1.mn.comcast.net) has joined #jython
[16:20] * Arfrever (~Arfrever@apache/committer/Arfrever) Quit (Quit: Ex??re)
[17:11] * enebo (~enebo@c-75-73-8-169.hsd1.mn.comcast.net) Quit (Quit: enebo)
[17:12] * Arfrever (~Arfrever@apache/committer/Arfrever) has joined #jython
[17:32] * bogush (~bogush@193.105.245.50) has joined #jython
[17:53] * bogush (~bogush@193.105.245.50) Quit (Remote host closed the connection)
[18:09] <jimbaker> so i think fireside, the clamp-compatible wsgi-servlet bridge, is going to be much faster, even before i rewrite it into java. certainly simpler at ~200 lines vs ~1500 lines in modjy (mostly python)
[18:10] * srcerer (~chatzilla@dns2.klsairexpress.com) has joined #jython
[18:27] * enebo (~enebo@166.137.110.13) has joined #jython
[18:43] <jimbaker> and it now works on my simple example! i need to test with ab, vs timing curl, but numbers are very nice already (at most few milliseconds, but again for a simple_app type example). but of course what i care about is fireside overhead
[20:18] * lopex (uid4272@gateway/web/irccloud.com/x-zoitsgkrxpqrmvcx) Quit (Ping timeout: 265 seconds)
[20:20] * Guest17399 (uid4272@gateway/web/irccloud.com/x-zxudfucxjrxdqrmo) has joined #jython
[20:21] * Guest17399 is now known as lopex
[20:24] <topi`> jimbaker: which patches are still blocking jython 2.7 beta 2?
[20:24] <jimbaker> so in testing with ab, i get more like less than < 1ms
[20:24] <jimbaker> for fireside
[20:25] <jimbaker> topi`, it's applying socket-reboot
[20:25] <topi`> have you contacted Frank, can he spare his time on the cause?
[20:25] <topi`> jimbaker: so beta 2 wouldn't have any of the current hacks in socket.py?
[20:25] <jimbaker> frank and i are in close contact, but he's pretty busy on a django proj this moment
[20:26] <jimbaker> but it's possible that beta 2 will free him up :)
[20:26] <topi`> django *could* run nicely on jython, as a servlet on tomcat or glassfish, *if* Modjy can be speeded up!
[20:26] <jimbaker> topi`, it will be significantly cleaned up, that's for sure - certainly none of the corner cases in current socket support
[20:27] <jimbaker> topi`, curiously enough, i have a new solution for you
[20:27] <topi`> I think the current version of Modjy is adding several milliseconds of latency
[20:27] <jimbaker> fireside - https://github.com/jythontools/fireside
[20:27] <topi`> yeah, you were just rambling about it :)
[20:27] <jimbaker> exactly. i don't think modjy will get any love
[20:27] <topi`> do you intend to make it a "direct" replacement for modjy?
[20:28] <topi`> so that one doesn't have to muck around existing setups et
[20:28] <jimbaker> no. if i did that, i would have refactored modjy
[20:28] <topi`> setting up modjy is just setting some variables in a nasty XML file. (web.xml?)
[20:28] <jimbaker> it's possible that someone could pull in the fireside work into modjy. but just like we saw an evolution of wsgi containers in cpython space, i expect the same in jython
[20:29] <topi`> and of course, setting up the correct libs in correct directories
[20:29] <jimbaker> topi`, because it's based on clamp, the integration is far easier
[20:29] <topi`> I don't have any attachment or other feelings towards modjy ;) it's just bundled in when I make a WAR out of my django proj
[20:30] <jimbaker> topi`, exactly. i expect similar tooling. right now it's painfully trivial. let me paste a gist
[20:30] <topi`> jimbaker: I can start working on the django-on-jython package ;) it needs some patching anyways.
[20:32] <topi`> IIRC django-on-jython is the package that contains modjy, and scripts for setting modjy up for the .war
[20:33] <jimbaker> https://gist.github.com/jimbaker/9399013
[20:33] <jimbaker> topi`, yeah, i'm actually the person who started the project... but i don't really recall much details ;)
[20:34] <topi`> hehe
[20:34] <topi`> the last commits there were from somebody else
[20:35] <jimbaker> it's been probably been 5 years since i worked on DoJ
[20:35] <topi`> ok, what does the fireside-single.jar contain? netty4 and friends?
[20:35] <topi`> hm, probably async sockets are not needed for WSGI
[20:35] <jimbaker> no netty. it depends on a container like jetty
[20:36] <jimbaker> although there will be netty packed in jython runtime at some near future. for this stuff, i just use the jython-ssl branch
[20:37] <topi`> that fireside-singlejar probably has the clamp stuff builtin?
[20:37] <topi`> I mean baked in :)
[20:37] <jimbaker> yes, very much so
[20:37] <jimbaker> clamp is general goodness
[20:38] <topi`> so hellowsgi is the package/module where simple_app gets invoked?
[20:38] <jimbaker> so https://github.com/jythontools/fireside/blob/master/fireside/servlet.py#L25 shows the clamp point
[20:38] <jimbaker> i haven't pushed up hellowsgi, but it's just regular python code
[20:39] <jimbaker> reminds me of one nice convention we had on launchpad, junk branches for stuff like hellowsgi
[20:39] <jimbaker> i will create a repo just so you can see
[20:40] <topi`> the WSGIservlet class really looks like a vanilla WSGI impl
[20:42] <topi`> oh, you subclass ToolBase which is clamped at org.python.tools
[20:42] <topi`> perhaps i'm too tired, after only sleepng a few hours last night :/
[20:43] <topi`> normally i have good sleeping skills. Give me a pillow and I'll start snoring ;)
[20:44] <jimbaker> https://github.com/jimbaker/hellowsgi
[20:44] <topi`> "This spike demonstrates with working code that for Jython 2.7 ..." what is a spike?
[20:44] <jimbaker> topi`, yes, that's pretty much the point of WSGIServlet. but give it a metaclass base of ToolBase and clamp magic happens
[20:44] <topi`> except a web traffic spike
[20:45] <jimbaker> a spike simply means i haven't bothered to do certain things, like write unit tests
[20:45] <topi`> hehe
[20:45] <jimbaker> so basically the idea is, you are playing with an api, trying to figure out how to use it
[20:45] <topi`> unit testing sometimes speeds up dev, especially if you have complex logic and/or magic
[20:46] <topi`> hacking away at some api's is entirely different :)
[20:46] <jimbaker> right. i love TDD unit tests in many circumstances
[20:46] <jimbaker> instead what i did was accumulate a bunch of functional tests
[20:46] <jimbaker> from a variety of sources. some of which i should just add to that branch
[20:47] <topi`> yeah that's a simple wsgi app :)
[20:47] <jimbaker> right. it doesn't even bother testing out the brunt of that code, around reading the input stream
[20:47] <topi`> but django's wsgi calls aren't really much more than that
[20:48] <topi`> it would definitely nice to re-benchmark django, especially against CPython :)
[20:48] <topi`> I was slightly frustrated at the high latency of django's responses behind modjy
[20:48] <topi`> at first I blamed jython
[20:49] <jimbaker> the point of fireside is to avoid any unnecessary expense. modjy supports many modes of operations. the code is complex. fireside instead has an opinionated way - it uses clamp to load up the jython runtime, etc
[20:50] <topi`> "KISS - keep it simple stupid"
[20:50] <jimbaker> so now, instead of fighting that complexity, i was looking at gc tuning parameters
[20:50] <topi`> what kinds of objects are filling up the garbage?
[20:51] <jimbaker> i'm sure some of it is involved in moving between python and java space, such as the need to copy a string into a buffer before writing it
[20:51] <topi`> so, this is how buffers work now in jython?
[20:52] <jimbaker> that will go away once fireside is rewritten in java
[20:52] <jimbaker> oh, right, you also see that. yes, buffer, bytearray, all there
[20:52] <topi`> well, byte/string copies ought to be fast in java, right?
[20:52] <jimbaker> they are fast
[20:53] <topi`> about that socket-reboot, is there any work I can do to take that forward? towards a 2.7 beta 2 release :)
[20:54] <jimbaker> it's just the gc pauses that now matter. so w/ whatever java and jetty runner defaults i'm running with, i can get to ~16000 requests at about < 0.8 ms per request (not counting queuing)
[20:54] <jimbaker> then a long long pause
[20:54] <jimbaker> then the cycle repeats. i think that's pretty good
[20:54] <jimbaker> it can just be better w/ less garbage in the mix + a bit of option tuning
[20:55] <topi`> yeah, but long pauses aren't that uncommon. for example, our node.js service sometimes takes long naps, evidently the google V8 engine is stuck with GC
[20:55] <jimbaker> sure, and good load balancing can make it not so bad for actual usage
[20:55] <topi`> two servers running in different phases of the GC cycle ;)
[20:56] <jimbaker> N servers running, you don't have to worry then about phasing ;)
[20:57] <topi`> we have now deployed our rev.B system onto some tomcat servers... it's using the spring framework, I guess you know about it
[20:57] <topi`> but its .WAR is just as bit as my django project's (when cast into a .war with jython baked in)
[20:58] <topi`> just as big
[20:58] <topi`> I don't really think too highly about the spring tools, the annotations are a bit heavy and complex and difficult to learn
[20:59] <topi`> despite its flaws, I prefer django ;)
[20:59] <jimbaker> so you can either use annotations OR use a dynamic language. they get to the same place
[20:59] <jimbaker> i suppose even better is when clamp supports annotations. coming soon!
[21:00] <topi`> then one would write a site using spring tools entirely written in python ;)
[21:01] <jimbaker> something like that. just imagine the productivity! ;)
[21:01] <topi`> btw, I bought the Scala book you recommended. it's interesting, but sometimes really tedious to try to learn about the type inference stuff
[21:02] <topi`> and from my point of view, a big problem with scala is, that in order to be productive with scala, you'd need to know java and the java world well first
[21:02] <topi`> because that's where scala lives, really
[21:02] <jimbaker> topi`, agreed. scala has a huge learning curve. btw, i'm trying to think of the exact book i recommended
[21:02] <topi`> jython is different, you just import your favourite python libraries, and of you go
[21:02] <jimbaker> atomic scala is a decent intro
[21:03] <topi`> jimbaker: it was the Manning book on Scala
[21:03] <topi`> FP with Scala
[21:03] <jimbaker> yes, that's a book i paired w/ atomic scala for my students
[21:03] <jimbaker> and this is the obligatory youtube about "just imagine the productivity" - http://www.youtube.com/watch?v=XxyB29bDbBA
[21:04] <topi`> I guess the reason I'm still writing all stuff in python is that everything in python is - simple
[21:04] <topi`> it's a language you can literally learn in 24 hours
[21:05] <jimbaker> so i really like scala for writing parsers, interpreters, and compilers in a classroom setting, plus libraries and DSLs for big data
[21:05] <jimbaker> but that's a small if interesting subset of systems
[21:07] <topi`> you can write those in scala, and then use the .class'es from within jython :)
[21:07] <topi`> in today's world, 95% of the work a oder does is just gluing stuff together ... that's where python excels
[21:07] <topi`> a coder does
[21:11] <topi`> I'll clone the fireside repo and try it out. Will give then feedback
[21:13] <topi`> btw what went wrong with my jython-ssl install? I used the installer jar. Problem is, the CLASSPATH is not set correctly for finding things like the Bzip2 jar and ICU4J stuff
[21:14] <topi`> things that exist in jython-ssl/extlibs
[21:16] <topi`> ok, cloned it, will just running setup.py do all the clamp magic for me?
[21:17] <topi`> d'oh, from "distutils import log" fails
[21:17] <topi`> that was with the older django virtualenv
[21:18] * xymox (lechuck@unaffiliated/contempt) Quit (Ping timeout: 264 seconds)
[21:24] * xymox (lechuck@unaffiliated/contempt) has joined #jython
[21:26] <topi`> oh, clamp is already at version 0.4 , with a singlejar script that was installed to my venv's bin :)
[21:31] * xymox (lechuck@unaffiliated/contempt) Quit (Ping timeout: 244 seconds)
[21:34] * xymox (lechuck@unaffiliated/contempt) has joined #jython
[21:34] <jimbaker> topi`, sounds good. do tell me how it works for you
[21:35] <topi`> single jar fails:
[21:35] <topi`> AttributeError: 'module' object has no attribute 'getsitepackages'
[21:35] <topi`> is this "module" perhaps the distutils module?
[21:36] <jimbaker> something is missing, that's from the site module
[21:36] <topi`> site_path = site.getsitepackages()[0]
[21:36] <topi`> "import site" succeeds in interpreter, at least
[21:37] <topi`> if I take a dir(), there is 'addsitepackages' but no getsitepackages...
[21:37] <topi`> what is this site module, anyways?
[21:39] <jimbaker> the site module is the key piece for supporting site-packages
[21:39] <jimbaker> in particular, it supports such things as pth files
[21:39] <topi`> yeah
[21:40] <jimbaker> when you get past that issue - might just mean blowing away and starting over, don't know
[21:40] <jimbaker> (or at least a fresh checkout)
[21:41] <jimbaker> the basic clamp approach is to just install, install, ... - where install usually is bound to the underlying clamp command
[21:41] <jimbaker> that is setup.py clamp
[21:41] <jimbaker> this builds out jars, putting them into the jars/ directory and pointing at them w/ jar.pth
[21:42] <jimbaker> singlejar then does the final assembly
[21:43] <topi`> yeah and the clamp command is doing the singlejar stuff
[21:43] <topi`> which is where i'm stuck now :)
[21:44] <topi`> I diff'd the site.py from my old jython virtualenv back in october, and it's older (no _is_64bit there, for example)
[21:44] <topi`> but the required getsitepackages() is nowhere to be seen
[21:44] <jimbaker> for war support, i think we should do something similar but maybe it can pick up all the registered wsgi handlers as well. not certain. this is the sort of thing that the django on jython management command for war files supports
[21:45] <jimbaker> hmmm, that's very strange
[21:45] <topi`> /home/topi/jython-ssl/Lib/site.py:def getsitepackages():
[21:45] <topi`> here it is
[21:45] <topi`> but it seems some *other* version ended up in the jython install which I created with the jython-installer.jar
[21:45] <topi`> since jython-ssl is the tree where I ran "ant all-jars"
[21:46] <jimbaker> interesting. i haven't tried this with jython-installer.jar
[21:46] <topi`> could it be that virtualenv is mucking around with site.py?
[21:47] <jimbaker> ahh, another interesting combo to look at. so again, i haven't played w/ virtualenv, given that it really relies on pip for full functionality
[21:48] <topi`> that's true... but at least for any django use, venv is a must
[21:48] <jimbaker> i like to not consider all possibilities of how things can be combined together when i'm trying to focus on certain specifics like netty 4. it helps keep me sane :)
[21:48] <topi`> I'll try a global install of jython-ssl instead of venv
[21:48] <jimbaker> oh for sure. i just haven't looked at it yet, since it's further down on the list, after pip
[21:49] <jimbaker> in terms of 2.7 support
[21:49] <topi`> I admit I don't know in detail what virtualenv is really doing...
[21:49] <topi`> other than add to the existing complexity ;)
[21:53] <topi`> ok, got past it now :)
[21:53] <topi`> from javax.servlet.http import HttpServlet
[21:53] <topi`> ImportError: No module named servlet
[21:53] <topi`> I need something in my CLASSPATH... hmm
[21:57] <topi`> I can, however, import e.g. java.net.URL
[21:58] <topi`> ah.. now I remember, that comes from the servlet-api.jar which is *somewhere* on my root fs
[21:59] <topi`> /usr/share/tomcat7/lib/servlet-api.jar
[22:02] <jimbaker> sure. i think we also have it in jython itself
[22:03] <jimbaker> yes
[22:03] <jimbaker> servlet-api-2.5.jar
[22:03] <jimbaker> should upgrade to 3.x i suppose
[22:04] <topi`> what's the purpose of "jython-single.jar" which was created when I ran the singlejar command in fireside dir?
[22:05] <topi`> is there all of jython bundled there so that I can just drop that onto the WAR lib/ and nothing else?
[22:06] <topi`> the name of that .jar is not specified in web.xml is it?
[22:11] <jimbaker> the single jar contains everything in your jython env, including the jython runtime
[22:11] <jimbaker> so you just need that, plus a web.xml, in your war file
[22:12] <jimbaker> note this is unlike modjy which supports a mixed mode of operations. of course we could move the jars around a bit, in case you want them to be included by the container
[22:12] <jimbaker> directly, instead of being placed in the war
[22:13] <topi`> ok there is the hello world!
[22:13] <topi`> now, some benchmarking...
[22:14] <topi`> jimbaker: expecting something to be provided by the container somewhat deflates the idea of self-sufficient .WARs
[22:14] <topi`> (which you can just deploy on any random container at some crappy ISP)
[22:14] <jimbaker> yeah, let's just assume self suffiency for now
[22:15] <topi`> I don't know if there are potential memory savings coming from sharing .jars at the container
[22:15] <topi`> disk space is not really a problem
[22:15] <topi`> but RAM is, usually
[22:16] <jimbaker> agreed. so it depends on one deploys. for now, doing it self contained is much easier to reason about. but in theory, it could be shared
[22:16] <jimbaker> i'm glad to see you're getting hello, world! however
[22:17] <jimbaker> also your django app should be sufficiently punishing of the remaining details :)
[22:20] <topi`> I did 100 requests (using python's urllib2) and it took 12 secs
[22:20] <topi`> granted, I run tomcat7 on virtualbox, which brings it to a crawl
[22:21] * whg is now known as zz_whg
[22:21] <jimbaker> topi`, definitely not in fireside
[22:22] <topi`> I'll try to move the .war to our quad core box at work, which runs tomcat7
[22:22] <topi`> this is a real machine, not a paravirtualized snail
[22:22] <jimbaker> so right virtualization can be tough on running servers. i've been using it with jetty, probably not much of a difference in the virtualbox env
[22:22] * clajo04 (~clajo04@pool-108-21-222-14.nycmny.fios.verizon.net) has joined #jython
[22:23] <jimbaker> so with that, was there any perf diffs from modjy?
[22:23] <topi`> I thought my django was dead slow when I tried it on my vbox, then moved it to the "real" server box, and lo behold, it was usable :)
[22:23] <jimbaker> the more interesting question to me, your code ran just fine functionally on fireside?
[22:24] <jimbaker> because that would be rather cool if that's the case, given how little tested some of those code paths are ;)
[22:24] <topi`> I haven't yet slapped it onto the django war, yet
[22:24] <topi`> just tried out a simple wsgi app
[22:24] <topi`> granted, it's easier to set up than modjy
[22:25] <topi`> I remember a long night fighting with modjy
[22:25] <jimbaker> ok, just checking
[22:25] <topi`> especially in conjunction with the singlejar script, it's dead simple to put together your own .war
[22:26] <jimbaker> especially when i add a new command to fireside - setup.py war
[22:26] <topi`> make love, not war!
[22:26] <topi`> >>> t2-t1
[22:26] <topi`> 32.20818591117859
[22:27] <topi`> ok, this was the simplest page my django app was serving. there's the templating overhead
[22:27] <topi`> but no DB queries
[22:27] * sinsnare (~sinisters@pool-108-28-93-153.washdc.fios.verizon.net) Quit (Remote host closed the connection)
[22:27] <topi`> this is a lot of time for serving 100 requets
[22:28] <topi`> hmm
[22:28] <jimbaker> it's definitely something else. i would expect <1ms overhead at wsgi for fireside. let me post my results
[22:29] <topi`> it's just that *everything* is slow on my vbox tomcat
[22:31] <jimbaker> https://gist.github.com/jimbaker/9401155
[22:32] <topi`> yeah, that's a very satisfying requests-per-second figure
[22:32] <jimbaker> so i think that's fairly matched between client and # of threads jetty runner spins up
[22:33] <jimbaker> but i don't know the defaults for jetty runner. it's close regardless
[22:38] <topi`> I installed the test war on our corporate server, and judging by just connecting via curl, it's an order magnitude faster than my vbox
[22:44] * ciziar (~textual@h-135-196.a2.corp.bahnhof.no) has joined #jython
[22:49] * ciziar (~textual@h-135-196.a2.corp.bahnhof.no) Quit (Ping timeout: 264 seconds)
[22:49] <topi`> I tried also with static content from a grails project on the same servers, got about the same num of requests
[22:50] <topi`> I believe the static content is directly served by tomcat
[23:06] <jimbaker> topi`, cool! i would love it if you played w/ fireside some more. again, patches are very much welcome! let me make certain issues are enabled as well
[23:07] <jimbaker> issues are in fact enabled
[23:32] <jimbaker> looks like my ab issue is really about jetty rate limiting. so just need to figure out how to turn off that normally useful feature
[23:35] * ciziar (~textual@h-135-196.a2.corp.bahnhof.no) has joined #jython
[23:38] <jimbaker> in particular, when running with the G1 garbage collector, i'm seeing very good cleanup w/ my periodically paused ab testing - run 5000 tests, then pause 15s, loop. but i would prefer to see a continuous load test as well
These logs were automatically created by JythonLogBot_ on irc.freenode.net using a slightly modified version of the Java IRC LogBot (github).