Quantcast

[Neo4j] Java segfault using paste and Python bindings

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Neo4j] Java segfault using paste and Python bindings

marmida
I am attempting to use Paste to serve a small web app that uses neo4j via
the Python bindings.  I get a segfault whenever I try to create a new
database via 'neo4j.GraphDatabase'.  I have narrowed this down to the
following minimal sample:

http://pastebin.com/MkNRvLR6

The top of the stack trace looks like this:

Stack: [0xb42e7000,0xb4ae8000],  sp=0xb4ae44f0,  free space=8181k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
C  [_jpype.so+0x26497]  JPJavaEnv::NewObjectA(_jclass*, _jmethodID*,
jvalue*)+0x37
C  [_jpype.so+0x3c0e8]  JPMethodOverload::invokeConstructor(_jclass*,
std::vector<HostRef*, std::allocator<HostRef*> >&)+0x178
C  [_jpype.so+0x3a417]  JPMethod::invokeConstructor(std::vector<HostRef*,
std::allocator<HostRef*> >&)+0x47
C  [_jpype.so+0x1beba]  JPClass::newInstance(std::vector<HostRef*,
std::allocator<HostRef*> >&)+0x2a
C  [_jpype.so+0x67b9c]  PyJPClass::newClassInstance(_object*, _object*)+0xfc
C  [python+0x96822]  PyEval_EvalFrameEx+0x4332
C  [python+0x991e7]  PyEval_EvalCodeEx+0x127


I believe that is JPype trying to create an EmbeddedGraphDatabase via
neo4j.GraphDatabase() in python.

I noticed this previous thread:
http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-CPython-Pylons-and-threading-td942435.html

That looks like it might be related, except this problem occurs whenever I
try to create a database, not on shutdown.  The issue definitely has
something to do with using neo4j under Paste; here's a Python shell session
which shows neo4j working:

>>> import webob
>>> import app
>>> debug_app = app.DebugApp()
>>> response = debug_app(webob.Request.blank('/'))
>>> response.body
'it worked'

Relevant version info:
- ubuntu 11.04 (same crash under 32- and 64-bit systems)
- Python 2.7.1-0ubuntu5
- neo4j_embedded-1.5.b2
- jpype 0.5.4.1-3
- paste 1.7.5.1
- OpenJDK  6b22-1.10.2 (same crash under the Sun JRE)


Thanks for your time and any clues you might have.
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Java segfault using paste and Python bindings

marmida
I solved my problem by ensuring that I don't 'import neo4j' until I'm in a
dedicated thread.  So long as all neo4j interaction is handled within that
thread, I don't crash.

The Python neo4j bindings use JPype; I noticed this note from the JPype
docs (http://jpype.sourceforge.net/doc/user-guide/userguide.html#threading):

"For the most part, python threads based on OS level threads (i.e posix
threads), will work without problem. The only thing to remember is to
call jpype.attachThreadToJVM() in the thread body to make the JVM usable
from that thread. For threads that you do not start yourself, you can
call isThreadAttachedToJVM() to check."

I couldn't find any evidence for this, but I think that some of the Java
code in the neo4j bindings may call attachThreadToJVM at import time.

Unfortunately, this means that even though neo4j is thread-safe, it must be
constrained to a single thread when used from Python.  But that's not too
disappointing, considering.

On Sat, Oct 29, 2011 at 4:25 PM, Michael Rene Armida <[hidden email]> wrote:

> I am attempting to use Paste to serve a small web app that uses neo4j via
> the Python bindings.  I get a segfault whenever I try to create a new
> database via 'neo4j.GraphDatabase'.  I have narrowed this down to the
> following minimal sample:
>
> http://pastebin.com/MkNRvLR6
>
> The top of the stack trace looks like this:
>
> Stack: [0xb42e7000,0xb4ae8000],  sp=0xb4ae44f0,  free space=8181k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C  [_jpype.so+0x26497]  JPJavaEnv::NewObjectA(_jclass*, _jmethodID*,
> jvalue*)+0x37
> C  [_jpype.so+0x3c0e8]  JPMethodOverload::invokeConstructor(_jclass*,
> std::vector<HostRef*, std::allocator<HostRef*> >&)+0x178
> C  [_jpype.so+0x3a417]  JPMethod::invokeConstructor(std::vector<HostRef*,
> std::allocator<HostRef*> >&)+0x47
> C  [_jpype.so+0x1beba]  JPClass::newInstance(std::vector<HostRef*,
> std::allocator<HostRef*> >&)+0x2a
> C  [_jpype.so+0x67b9c]  PyJPClass::newClassInstance(_object*,
> _object*)+0xfc
> C  [python+0x96822]  PyEval_EvalFrameEx+0x4332
> C  [python+0x991e7]  PyEval_EvalCodeEx+0x127
>
>
> I believe that is JPype trying to create an EmbeddedGraphDatabase via
> neo4j.GraphDatabase() in python.
>
> I noticed this previous thread:
> http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-CPython-Pylons-and-threading-td942435.html
>
> That looks like it might be related, except this problem occurs whenever I
> try to create a database, not on shutdown.  The issue definitely has
> something to do with using neo4j under Paste; here's a Python shell session
> which shows neo4j working:
>
> >>> import webob
> >>> import app
> >>> debug_app = app.DebugApp()
> >>> response = debug_app(webob.Request.blank('/'))
> >>> response.body
> 'it worked'
>
> Relevant version info:
> - ubuntu 11.04 (same crash under 32- and 64-bit systems)
> - Python 2.7.1-0ubuntu5
> - neo4j_embedded-1.5.b2
> - jpype 0.5.4.1-3
> - paste 1.7.5.1
> - OpenJDK  6b22-1.10.2 (same crash under the Sun JRE)
>
>
> Thanks for your time and any clues you might have.
>
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Java segfault using paste and Python bindings

Jacob Hansson-2
Hey Michael, thanks for the investigative work you put into this, and sorry
for not replying earlier.

I was not aware of the thread limitations of JPype, that's quite
dissapointing.. Overall, it feels like the JPype solution is not a
manageable one in the long term. The embedded use case is cool, because
it's so freaking convenient, but using JPype to embed the JVM is painful.

There are a few other options for JVM->python integration that are starting
to look interesting, perhaps we can look around a little bit and see what
is available. It would be especially cool if we can use something that
works across CPython, Jython as well as PyPy. I know Tobias was thinking
about something that would make that possible.

In the short to medium term though, I'd like to focus on two things (please
note that this is *my opinion*, not a product roadmap):

*Getting cypher and gremlin into the embedded python API*
Eventually, I think we should deprecate the traversal API in the embedded
client in favor of Cypher. Traversals need callbacks and shared state and
so on between the two VMs, and its gonna hurt, I think, if we depend on
that further down the road.

Doing this will also make it simple to keep full 1-1 API compatibility
between the embedded client and REST clients, making it easy for end users
to switch between the two.

*Making the REST API fast enough to make that use case more appealing.*
JPype is really just a cheap way of implementing a form of remoting API.
It's currently worth the pain, since it is a ton faster than our REST API,
but it's a band-aid, not a proper fix. We should focus on speeding up the
REST API.


WDYT?

/Jake

On Tue, Nov 1, 2011 at 5:00 PM, Michael Rene Armida <[hidden email]> wrote:

> I solved my problem by ensuring that I don't 'import neo4j' until I'm in a
> dedicated thread.  So long as all neo4j interaction is handled within that
> thread, I don't crash.
>
> The Python neo4j bindings use JPype; I noticed this note from the JPype
> docs (http://jpype.sourceforge.net/doc/user-guide/userguide.html#threading
> ):
>
> "For the most part, python threads based on OS level threads (i.e posix
> threads), will work without problem. The only thing to remember is to
> call jpype.attachThreadToJVM() in the thread body to make the JVM usable
> from that thread. For threads that you do not start yourself, you can
> call isThreadAttachedToJVM() to check."
>
> I couldn't find any evidence for this, but I think that some of the Java
> code in the neo4j bindings may call attachThreadToJVM at import time.
>
> Unfortunately, this means that even though neo4j is thread-safe, it must be
> constrained to a single thread when used from Python.  But that's not too
> disappointing, considering.
>
> On Sat, Oct 29, 2011 at 4:25 PM, Michael Rene Armida <[hidden email]>
> wrote:
>
> > I am attempting to use Paste to serve a small web app that uses neo4j via
> > the Python bindings.  I get a segfault whenever I try to create a new
> > database via 'neo4j.GraphDatabase'.  I have narrowed this down to the
> > following minimal sample:
> >
> > http://pastebin.com/MkNRvLR6
> >
> > The top of the stack trace looks like this:
> >
> > Stack: [0xb42e7000,0xb4ae8000],  sp=0xb4ae44f0,  free space=8181k
> > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> > code)
> > C  [_jpype.so+0x26497]  JPJavaEnv::NewObjectA(_jclass*, _jmethodID*,
> > jvalue*)+0x37
> > C  [_jpype.so+0x3c0e8]  JPMethodOverload::invokeConstructor(_jclass*,
> > std::vector<HostRef*, std::allocator<HostRef*> >&)+0x178
> > C  [_jpype.so+0x3a417]  JPMethod::invokeConstructor(std::vector<HostRef*,
> > std::allocator<HostRef*> >&)+0x47
> > C  [_jpype.so+0x1beba]  JPClass::newInstance(std::vector<HostRef*,
> > std::allocator<HostRef*> >&)+0x2a
> > C  [_jpype.so+0x67b9c]  PyJPClass::newClassInstance(_object*,
> > _object*)+0xfc
> > C  [python+0x96822]  PyEval_EvalFrameEx+0x4332
> > C  [python+0x991e7]  PyEval_EvalCodeEx+0x127
> >
> >
> > I believe that is JPype trying to create an EmbeddedGraphDatabase via
> > neo4j.GraphDatabase() in python.
> >
> > I noticed this previous thread:
> >
> http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-CPython-Pylons-and-threading-td942435.html
> >
> > That looks like it might be related, except this problem occurs whenever
> I
> > try to create a database, not on shutdown.  The issue definitely has
> > something to do with using neo4j under Paste; here's a Python shell
> session
> > which shows neo4j working:
> >
> > >>> import webob
> > >>> import app
> > >>> debug_app = app.DebugApp()
> > >>> response = debug_app(webob.Request.blank('/'))
> > >>> response.body
> > 'it worked'
> >
> > Relevant version info:
> > - ubuntu 11.04 (same crash under 32- and 64-bit systems)
> > - Python 2.7.1-0ubuntu5
> > - neo4j_embedded-1.5.b2
> > - jpype 0.5.4.1-3
> > - paste 1.7.5.1
> > - OpenJDK  6b22-1.10.2 (same crash under the Sun JRE)
> >
> >
> > Thanks for your time and any clues you might have.
> >
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
>



--
Jacob Hansson
Phone: +46 (0) 763503395
Twitter: @jakewins
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Java segfault using paste and Python bindings

marmida
In reply to this post by marmida
I had no idea that neo4j had a REST interface; I am essentially
implementing my own REST interface in order to use neo4j embedded in a
desktop app.  I will have to look into that more to see if there's an easy
way to embed the server and still communicate to it via the REST API.
 Thanks for the pointer!
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Java segfault using paste and Python bindings

Peter Neubauer
Administrator
Michael,
have you looked at
http://docs.neo4j.org/chunked/snapshot/rest-api.html for
documentation? We probably should promote the manual much more
prominently on the site :/

Cheers,

/peter neubauer

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org              - NOSQL for the Enterprise.
http://startupbootcamp.org/    - Öresund - Innovation happens HERE.



On Wed, Nov 2, 2011 at 7:49 AM, Michael Rene Armida <[hidden email]> wrote:

> I had no idea that neo4j had a REST interface; I am essentially
> implementing my own REST interface in order to use neo4j embedded in a
> desktop app.  I will have to look into that more to see if there's an easy
> way to embed the server and still communicate to it via the REST API.
>  Thanks for the pointer!
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
>
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Loading...