Quantcast

[Neo4j] CPython, Pylons, and threading

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Neo4j] CPython, Pylons, and threading

Aaron Brenzel
Hi everyone!

Just wanted to say I really like neo4j so far and what it's capable of.
It's helping to solve a lot of data modeling problems that were impossible
to handle in a RDBMS system.  I'm using it with the Python bindings which
are mostly intuitive so far, but I've been having an odd issue crop up.
Despite the fact that it's a segmentation fault (I'll print the logs below),
it's not a show stopper because it only occurs on shutting down the Pylons
server (Paste Script 1.7.1) and does not crash the system.  Here are the
relevant pieces of the log:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x08dae990, pid=16123, tid=3085936320
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) Client VM (16.3-b01 mixed mode linux-x86 )
# Problematic frame:
# C  [_jpype.so+0x3b990]  _ZN9JPJavaEnv11NewLocalRefEP8_jobject+0x20
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

Stack: [0xbff41000,0xbff91000],  sp=0xbff8f980,  free space=13abff8f310k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
C  [_jpype.so+0x3b990]  _ZN9JPJavaEnv11NewLocalRefEP8_jobject+0x20
C  [_jpype.so+0x522a1]
_ZN16JPMethodOverload14invokeInstanceERSt6vectorIP7HostRefSaIS2_EE+0x1c1
C  [_jpype.so+0x61c1c]
_ZN8JPMethod6invokeERSt6vectorIP7HostRefSaIS2_EE+0x5c
C  [_jpype.so+0x6bb73]  _ZN15PyJPBoundMethod8__call__EP7_objectS1_S1_+0x313
C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
C  [libpython2.6.so.1.0+0xc1118]  PyEval_EvalFrameEx+0x10d8
C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
C  [libpython2.6.so.1.0+0xc5b24]  PyEval_EvalFrameEx+0x5ae4
C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
C  [libpython2.6.so.1.0+0x54659]
C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
C  [libpython2.6.so.1.0+0xc0da8]  PyEval_EvalFrameEx+0xd68
C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
C  [libpython2.6.so.1.0+0x5456a]
C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
C  [libpython2.6.so.1.0+0xbf304]  PyEval_CallObjectWithKeywords+0x54
C  [libpython2.6.so.1.0+0xea113]  Py_Finalize+0xd3
C  [libpython2.6.so.1.0+0xf7a64]  Py_Main+0x4d4
C  [python2.6+0x592]  main+0x32
C  [libc.so.6+0x15e9c]  __libc_start_main+0xdc

I am almost positive this is a thread issue.  By default, the Pylons HTTP
server starts up with a thread pool of 10 workers.  These are POSIX threads,
which JPype should handle OK.  However, my best guess is in the shutdown
process Pylons is killing the threads before the JVM is finished executing
whatever code is going on in them, maybe some shutdown routines or
something.  Also, Pylons comes with a web-based debugger, which is
essentially a web-based interface to the standard Python debugger.
Attempting to use it for any code using the neo4j bindings causes an
immediate segfault, probably because the debugger traces up and down the
stack and is running into similar thread issues.

Is there a way to ensure the threads detach from the JVM before dying or
something similar to avoid this problem?  Right now it is more annoying than
anything because it fills up the project's main directory with logs (we are
in active development and do a lot of server startup/shutdown) and makes
debugging harder.  It probably would not affect production stability when we
reach that stage because it is only happening while debugging or shutting
down the server, but it is never good to release a project with a known
segfault issue.

Best,
Aaron
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] CPython, Pylons, and threading

Anders Nawroth
Hi Aaron!

It's probably a bug in JPype. At the moment nobody from the Neo4j team
can look deeper into this, but maybe it's possible to check with the
JPype project. Thread issues was recently discussed there:
http://sourceforge.net/mailarchive/message.php?msg_name=92CDD168D1E81F4F9D3839DC45903FC677EAB2E2%40dlee03.ent.ti.com


/anders


On 07/04/2010 05:51 PM, Aaron Brenzel wrote:

> Hi everyone!
>
> Just wanted to say I really like neo4j so far and what it's capable of.
> It's helping to solve a lot of data modeling problems that were impossible
> to handle in a RDBMS system.  I'm using it with the Python bindings which
> are mostly intuitive so far, but I've been having an odd issue crop up.
> Despite the fact that it's a segmentation fault (I'll print the logs below),
> it's not a show stopper because it only occurs on shutting down the Pylons
> server (Paste Script 1.7.1) and does not crash the system.  Here are the
> relevant pieces of the log:
>
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x08dae990, pid=16123, tid=3085936320
> #
> # JRE version: 6.0_20-b02
> # Java VM: Java HotSpot(TM) Client VM (16.3-b01 mixed mode linux-x86 )
> # Problematic frame:
> # C  [_jpype.so+0x3b990]  _ZN9JPJavaEnv11NewLocalRefEP8_jobject+0x20
> #
> # If you would like to submit a bug report, please visit:
> #   http://java.sun.com/webapps/bugreport/crash.jsp
> #
>
> Stack: [0xbff41000,0xbff91000],  sp=0xbff8f980,  free space=13abff8f310k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C  [_jpype.so+0x3b990]  _ZN9JPJavaEnv11NewLocalRefEP8_jobject+0x20
> C  [_jpype.so+0x522a1]
> _ZN16JPMethodOverload14invokeInstanceERSt6vectorIP7HostRefSaIS2_EE+0x1c1
> C  [_jpype.so+0x61c1c]
> _ZN8JPMethod6invokeERSt6vectorIP7HostRefSaIS2_EE+0x5c
> C  [_jpype.so+0x6bb73]  _ZN15PyJPBoundMethod8__call__EP7_objectS1_S1_+0x313
> C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> C  [libpython2.6.so.1.0+0xc1118]  PyEval_EvalFrameEx+0x10d8
> C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> C  [libpython2.6.so.1.0+0xc5b24]  PyEval_EvalFrameEx+0x5ae4
> C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> C  [libpython2.6.so.1.0+0x54659]
> C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> C  [libpython2.6.so.1.0+0xc0da8]  PyEval_EvalFrameEx+0xd68
> C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> C  [libpython2.6.so.1.0+0x5456a]
> C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> C  [libpython2.6.so.1.0+0xbf304]  PyEval_CallObjectWithKeywords+0x54
> C  [libpython2.6.so.1.0+0xea113]  Py_Finalize+0xd3
> C  [libpython2.6.so.1.0+0xf7a64]  Py_Main+0x4d4
> C  [python2.6+0x592]  main+0x32
> C  [libc.so.6+0x15e9c]  __libc_start_main+0xdc
>
> I am almost positive this is a thread issue.  By default, the Pylons HTTP
> server starts up with a thread pool of 10 workers.  These are POSIX threads,
> which JPype should handle OK.  However, my best guess is in the shutdown
> process Pylons is killing the threads before the JVM is finished executing
> whatever code is going on in them, maybe some shutdown routines or
> something.  Also, Pylons comes with a web-based debugger, which is
> essentially a web-based interface to the standard Python debugger.
> Attempting to use it for any code using the neo4j bindings causes an
> immediate segfault, probably because the debugger traces up and down the
> stack and is running into similar thread issues.
>
> Is there a way to ensure the threads detach from the JVM before dying or
> something similar to avoid this problem?  Right now it is more annoying than
> anything because it fills up the project's main directory with logs (we are
> in active development and do a lot of server startup/shutdown) and makes
> debugging harder.  It probably would not affect production stability when we
> reach that stage because it is only happening while debugging or shutting
> down the server, but it is never good to release a project with a known
> segfault issue.
>
> Best,
> Aaron
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] CPython, Pylons, and threading

Aaron Brenzel
In reply to this post by Aaron Brenzel
Hi Anders,

I put in a fix that thankfully didn't involve hacking the JPype source.
Basically, we moved the invocation of the GraphDatabaseService out of the
model and into the app globals.  This ensures it's declared once for the
whole process and all threads use it, rather than each thread potentially
instantiating a new service and wreaking havoc with JPype.  Hopefully that's
a helpful hint to anyone using Pylons or a similar threaded Python web
environment with neo4j.

There is still one outstanding issue, which is that taking down the server
still results in neo4j thinking it didn't shut down cleanly.  Usually not a
huge deal, but if a lot nodes and edges were added during the server session
it can result in a long recovery period (2-3 minutes) when the server boots
back up.  I would guess this is because we're not getting a call into the
shutdown() method before the server process dies.  Unfortunately
reimplementing the __del__ method to call it doesn't work because Python
doesn't guarantee __del__ will be called on a system exit.  When we find a
good solution we'll be sure to post it.

Best,
Aaron

>Date: Tue, 06 Jul 2010 11:08:29 +0200
>From: Anders Nawroth <[hidden email]>
>Subject: Re: [Neo4j] CPython, Pylons, and threading
>To: Neo4j user discussions <[hidden email]>
>Message-ID: <[hidden email]>
>Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
>Hi Aaron!
>
>It's probably a bug in JPype. At the moment nobody from the Neo4j team
>can look deeper into this, but maybe it's possible to check with the
>JPype project. Thread issues was recently discussed there:
>http://sourceforge.net/mailarchive/message.php?msg_name=92CDD168D1E81F4F9D3839DC45903FC677EAB2E2%40dlee03.ent.ti.com<http://sourceforge.net/mailarchive/message.php?msg_name=92CDD168D1E81F4F9D3839DC45903FC677EAB2E2%40dlee03.ent.ti.com>
>
>
>/anders
>
>
>On 07/04/2010 05:51 PM, Aaron Brenzel wrote:
> Hi everyone!
>
> Just wanted to say I really like neo4j so far and what it's capable of.
> It's helping to solve a lot of data modeling problems that were impossible
> to handle in a RDBMS system.  I'm using it with the Python bindings which
> are mostly intuitive so far, but I've been having an odd issue crop up.
> Despite the fact that it's a segmentation fault (I'll print the logs
below),

> it's not a show stopper because it only occurs on shutting down the Pylons
> server (Paste Script 1.7.1) and does not crash the system.  Here are the
> relevant pieces of the log:
>
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x08dae990, pid=16123, tid=3085936320
> #
> # JRE version: 6.0_20-b02
> # Java VM: Java HotSpot(TM) Client VM (16.3-b01 mixed mode linux-x86 )
> # Problematic frame:
> # C  [_jpype.so+0x3b990]  _ZN9JPJavaEnv11NewLocalRefEP8_
jobject+0x20

> #
> # If you would like to submit a bug report, please visit:
> #   http://java.sun.com/webapps/bugreport/crash.jsp
> #
>
> Stack: [0xbff41000,0xbff91000],  sp=0xbff8f980,  free space=13abff8f310k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C  [_jpype.so+0x3b990]  _ZN9JPJavaEnv11NewLocalRefEP8_jobject+0x20
> C  [_jpype.so+0x522a1]
> _ZN16JPMethodOverload14invokeInstanceERSt6vectorIP7HostRefSaIS2_EE+0x1c1
> C  [_jpype.so+0x61c1c]
> _ZN8JPMethod6invokeERSt6vectorIP7HostRefSaIS2_EE+0x5c
> C  [_jpype.so+0x6bb73]
 _ZN15PyJPBoundMethod8__call__EP7_objectS1_S1_+0x313

> C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> C  [libpython2.6.so.1.0+0xc1118]  PyEval_EvalFrameEx+0x10d8
> C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> C  [libpython2.6.so.1.0+0xc5b24]  PyEval_EvalFrameEx+0x5ae4
> C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> C  [libpython2.6.so.1.0+0x54659]
> C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> C  [libpython2.6.so.1.0+0xc0da8]  PyEval_EvalFrameEx+0xd68
> C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> C  [libpython2.6.so.1.0+0x5456a]
> C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> C  [libpython2.6.so.1.0+0xbf304]  PyEval_CallObjectWithKeywords+0x54
> C  [libpython2.6.so.1.0+0xea113]  Py_Finalize+0xd3
> C  [libpython2.6.so.1.0+0xf7a64]  Py_Main+0x4d4
> C  [python2.6+0x592]  main+0x32
> C  [libc.so.6+0x15e9c]  __libc_start_main+0xdc
>
> I am almost positive this is a thread issue.  By default, the Pylons HTTP
> server starts up with a thread pool of 10 workers.  These are POSIX
threads,

> which JPype should handle OK.  However, my best guess is in the shutdown
> process Pylons is killing the threads before the JVM is finished executing
> whatever code is going on in them, maybe some shutdown routines or
> something.  Also, Pylons comes with a web-based debugger, which is
> essentially a web-based interface to the standard Python debugger.
> Attempting to use it for any code using the neo4j bindings causes an
> immediate segfault, probably because the debugger traces up and down the
> stack and is running into similar thread issues.
>
> Is there a way to ensure the threads detach from the JVM before dying or
> something similar to avoid this problem?  Right now it is more annoying
than
> anything because it fills up the project's main directory with logs (we
are
> in active development and do a lot of server startup/shutdown) and makes
> debugging harder.  It probably would not affect production stability when
we

> reach that stage because it is only happening while debugging or shutting
> down the server, but it is never good to release a project with a known
> segfault issue.
>
> Best,
> Aaron
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] CPython, Pylons, and threading

Tobias Ivarsson
You could use the python atexit module.

On Wed, Jul 7, 2010 at 3:21 AM, Aaron Brenzel <[hidden email]>wrote:

> Hi Anders,
>
> I put in a fix that thankfully didn't involve hacking the JPype source.
> Basically, we moved the invocation of the GraphDatabaseService out of the
> model and into the app globals.  This ensures it's declared once for the
> whole process and all threads use it, rather than each thread potentially
> instantiating a new service and wreaking havoc with JPype.  Hopefully
> that's
> a helpful hint to anyone using Pylons or a similar threaded Python web
> environment with neo4j.
>
> There is still one outstanding issue, which is that taking down the server
> still results in neo4j thinking it didn't shut down cleanly.  Usually not a
> huge deal, but if a lot nodes and edges were added during the server
> session
> it can result in a long recovery period (2-3 minutes) when the server boots
> back up.  I would guess this is because we're not getting a call into the
> shutdown() method before the server process dies.  Unfortunately
> reimplementing the __del__ method to call it doesn't work because Python
> doesn't guarantee __del__ will be called on a system exit.  When we find a
> good solution we'll be sure to post it.
>
> Best,
> Aaron
>
> >Date: Tue, 06 Jul 2010 11:08:29 +0200
> >From: Anders Nawroth <[hidden email]>
> >Subject: Re: [Neo4j] CPython, Pylons, and threading
> >To: Neo4j user discussions <[hidden email]>
> >Message-ID: <[hidden email]>
> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> >
> >Hi Aaron!
> >
> >It's probably a bug in JPype. At the moment nobody from the Neo4j team
> >can look deeper into this, but maybe it's possible to check with the
> >JPype project. Thread issues was recently discussed there:
> >
> http://sourceforge.net/mailarchive/message.php?msg_name=92CDD168D1E81F4F9D3839DC45903FC677EAB2E2%40dlee03.ent.ti.com
> <
> http://sourceforge.net/mailarchive/message.php?msg_name=92CDD168D1E81F4F9D3839DC45903FC677EAB2E2%40dlee03.ent.ti.com
> >
> >
> >
> >/anders
> >
> >
> >On 07/04/2010 05:51 PM, Aaron Brenzel wrote:
> > Hi everyone!
> >
> > Just wanted to say I really like neo4j so far and what it's capable of.
> > It's helping to solve a lot of data modeling problems that were
> impossible
> > to handle in a RDBMS system.  I'm using it with the Python bindings which
> > are mostly intuitive so far, but I've been having an odd issue crop up.
> > Despite the fact that it's a segmentation fault (I'll print the logs
> below),
> > it's not a show stopper because it only occurs on shutting down the
> Pylons
> > server (Paste Script 1.7.1) and does not crash the system.  Here are the
> > relevant pieces of the log:
> >
> > #
> > # A fatal error has been detected by the Java Runtime Environment:
> > #
> > #  SIGSEGV (0xb) at pc=0x08dae990, pid=16123, tid=3085936320
> > #
> > # JRE version: 6.0_20-b02
> > # Java VM: Java HotSpot(TM) Client VM (16.3-b01 mixed mode linux-x86 )
> > # Problematic frame:
> > # C  [_jpype.so+0x3b990]  _ZN9JPJavaEnv11NewLocalRefEP8_
> jobject+0x20
> > #
> > # If you would like to submit a bug report, please visit:
> > #   http://java.sun.com/webapps/bugreport/crash.jsp
> > #
> >
> > Stack: [0xbff41000,0xbff91000],  sp=0xbff8f980,  free space=13abff8f310k
> > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> > code)
> > C  [_jpype.so+0x3b990]  _ZN9JPJavaEnv11NewLocalRefEP8_jobject+0x20
> > C  [_jpype.so+0x522a1]
> > _ZN16JPMethodOverload14invokeInstanceERSt6vectorIP7HostRefSaIS2_EE+0x1c1
> > C  [_jpype.so+0x61c1c]
> > _ZN8JPMethod6invokeERSt6vectorIP7HostRefSaIS2_EE+0x5c
> > C  [_jpype.so+0x6bb73]
>  _ZN15PyJPBoundMethod8__call__EP7_objectS1_S1_+0x313
> > C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> > C  [libpython2.6.so.1.0+0xc1118]  PyEval_EvalFrameEx+0x10d8
> > C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> > C  [libpython2.6.so.1.0+0xc5b24]  PyEval_EvalFrameEx+0x5ae4
> > C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> > C  [libpython2.6.so.1.0+0x54659]
> > C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> > C  [libpython2.6.so.1.0+0xc0da8]  PyEval_EvalFrameEx+0xd68
> > C  [libpython2.6.so.1.0+0xc7ada]  PyEval_EvalCodeEx+0x8ba
> > C  [libpython2.6.so.1.0+0x5456a]
> > C  [libpython2.6.so.1.0+0x2704c]  PyObject_Call+0x5c
> > C  [libpython2.6.so.1.0+0xbf304]  PyEval_CallObjectWithKeywords+0x54
> > C  [libpython2.6.so.1.0+0xea113]  Py_Finalize+0xd3
> > C  [libpython2.6.so.1.0+0xf7a64]  Py_Main+0x4d4
> > C  [python2.6+0x592]  main+0x32
> > C  [libc.so.6+0x15e9c]  __libc_start_main+0xdc
> >
> > I am almost positive this is a thread issue.  By default, the Pylons HTTP
> > server starts up with a thread pool of 10 workers.  These are POSIX
> threads,
> > which JPype should handle OK.  However, my best guess is in the shutdown
> > process Pylons is killing the threads before the JVM is finished
> executing
> > whatever code is going on in them, maybe some shutdown routines or
> > something.  Also, Pylons comes with a web-based debugger, which is
> > essentially a web-based interface to the standard Python debugger.
> > Attempting to use it for any code using the neo4j bindings causes an
> > immediate segfault, probably because the debugger traces up and down the
> > stack and is running into similar thread issues.
> >
> > Is there a way to ensure the threads detach from the JVM before dying or
> > something similar to avoid this problem?  Right now it is more annoying
> than
> > anything because it fills up the project's main directory with logs (we
> are
> > in active development and do a lot of server startup/shutdown) and makes
> > debugging harder.  It probably would not affect production stability when
> we
> > reach that stage because it is only happening while debugging or shutting
> > down the server, but it is never good to release a project with a known
> > segfault issue.
> >
> > Best,
> > Aaron
> > _______________________________________________
> > Neo4j mailing list
> > [hidden email]
> > https://lists.neo4j.org/mailman/listinfo/user
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
>



--
Tobias Ivarsson <[hidden email]>
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Loading...