Quantcast

[Neo4j] Python embedding: problems with shutdown and threads

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Neo4j] Python embedding: problems with shutdown and threads

marmida
I'm having some weird problems with threading on shutdown.  As I previously
discovered, using neo4j in a multi-threaded Python app requires complete
segregation of neo4j into a single thread.  That all works fine for
creating a database, however, it appears that shutting one down is a bit of
a mystery.  I wrote a test program to minimally illustrate the difficulty:

http://pastebin.com/gg204kae

Threading is always easy to get wrong, so I added a simple switch to remove
neo4j from the picture.  When I disable neo, the program starts up and
shuts down cleanly.  When I re-enable neo, some of the print statements
never get run, and the app mysteriously exits.

I am testing this by saving it into a script and running it on the command
line.  Any time after the worker thread finishes creating the neo db, I hit
ctrl-c, and the two scenarios play out as above.

It seems that the embedded Python API is not exactly at the forefront of
popular use, and so I understand if nobody knows what's up with this.  I
just thought I should add something, for the record.  I hope my script
doesn't have an obvious bug.

Thanks,
Michael
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Python embedding: problems with shutdown and threads

Tim McNamara
Here are a few notes of my own from using Neo4j with Python embedded.
I know that this isn't the right place .

Ubuntu doesn't set $JAVA_HOME. I ended up hard coding the location of my JVM.

I seem to encounter unclean shutdowns all the time when running with
the iPython REPL. Is there a db.shutdown() command that I've been
neglecting to use?

Tim


On 3 November 2011 17:28, Michael Rene Armida <[hidden email]> wrote:

> I'm having some weird problems with threading on shutdown.  As I previously
> discovered, using neo4j in a multi-threaded Python app requires complete
> segregation of neo4j into a single thread.  That all works fine for
> creating a database, however, it appears that shutting one down is a bit of
> a mystery.  I wrote a test program to minimally illustrate the difficulty:
>
> http://pastebin.com/gg204kae
>
> Threading is always easy to get wrong, so I added a simple switch to remove
> neo4j from the picture.  When I disable neo, the program starts up and
> shuts down cleanly.  When I re-enable neo, some of the print statements
> never get run, and the app mysteriously exits.
>
> I am testing this by saving it into a script and running it on the command
> line.  Any time after the worker thread finishes creating the neo db, I hit
> ctrl-c, and the two scenarios play out as above.
>
> It seems that the embedded Python API is not exactly at the forefront of
> popular use, and so I understand if nobody knows what's up with this.  I
> just thought I should add something, for the record.  I hope my script
> doesn't have an obvious bug.
>
> Thanks,
> Michael
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
>
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Python embedding: problems with shutdown and threads

Jacob Hansson-2
On Thu, Nov 3, 2011 at 6:55 AM, Tim McNamara <[hidden email]>wrote:

> Here are a few notes of my own from using Neo4j with Python embedded.
> I know that this isn't the right place .
>
> Ubuntu doesn't set $JAVA_HOME. I ended up hard coding the location of my
> JVM.
>
> I seem to encounter unclean shutdowns all the time when running with
> the iPython REPL. Is there a db.shutdown() command that I've been
> neglecting to use?
>

Yeah, as long as you call db.shutdown(), things should be ok. See here:

http://docs.neo4j.org/chunked/snapshot/python-embedded-reference-core.html


We should perhaps set up the db to shut itself down properly automatically
if it detects that python is shutting down.. We decided against that
before, since we felt like "we don't own the application lifecycle". WDYT?


>
> Tim
>
>
> On 3 November 2011 17:28, Michael Rene Armida <[hidden email]> wrote:
> > I'm having some weird problems with threading on shutdown.  As I
> previously
> > discovered, using neo4j in a multi-threaded Python app requires complete
> > segregation of neo4j into a single thread.  That all works fine for
> > creating a database, however, it appears that shutting one down is a bit
> of
> > a mystery.  I wrote a test program to minimally illustrate the
> difficulty:
> >
> > http://pastebin.com/gg204kae
> >
> > Threading is always easy to get wrong, so I added a simple switch to
> remove
> > neo4j from the picture.  When I disable neo, the program starts up and
> > shuts down cleanly.  When I re-enable neo, some of the print statements
> > never get run, and the app mysteriously exits.
> >
> > I am testing this by saving it into a script and running it on the
> command
> > line.  Any time after the worker thread finishes creating the neo db, I
> hit
> > ctrl-c, and the two scenarios play out as above.
> >
> > It seems that the embedded Python API is not exactly at the forefront of
> > popular use, and so I understand if nobody knows what's up with this.  I
> > just thought I should add something, for the record.  I hope my script
> > doesn't have an obvious bug.
> >
> > Thanks,
> > Michael
>

I had a discussion with Tobias this morning about threading with JPype, and
am currently writing a fix that will allow full threading support for
neo4j-embedded.

I'll make sure to run this script as well after that fix is implemented, to
make sure it gets fixed as well.

As far as the forefront of use goes, there is a surprising amount of pypy
downloads of neo4j-embedded (4000 new installs in the last 30 days), so the
feedback you are providing here is gonna end up helping a lot of people :)
Thank you so much for taking the time to document the issues you are having.

/jake


> > _______________________________________________
> > Neo4j mailing list
> > [hidden email]
> > https://lists.neo4j.org/mailman/listinfo/user
> >
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
>



--
Jacob Hansson
Phone: +46 (0) 763503395
Twitter: @jakewins
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Python embedding: problems with shutdown and threads

marmida
That's awesome news.  I will check it out when it's ready.

I would suggest wrapping up the lifecycle management in a context manager.
 You're already using the GraphDatabase object's __enter__ and __exit__ to
manage transactions, so you'd probably have to add a second object.

Would you like me to add a bug somewhere?  I could turn that script into a
unit test, which should make it easier to evaluate.

On Thu, Nov 3, 2011 at 3:08 AM, Jacob Hansson <
[hidden email]> wrote:

> On Thu, Nov 3, 2011 at 6:55 AM, Tim McNamara <[hidden email]
> >wrote:
>
> > Here are a few notes of my own from using Neo4j with Python embedded.
> > I know that this isn't the right place .
> >
> > Ubuntu doesn't set $JAVA_HOME. I ended up hard coding the location of my
> > JVM.
> >
> > I seem to encounter unclean shutdowns all the time when running with
> > the iPython REPL. Is there a db.shutdown() command that I've been
> > neglecting to use?
> >
>
> Yeah, as long as you call db.shutdown(), things should be ok. See here:
>
> http://docs.neo4j.org/chunked/snapshot/python-embedded-reference-core.html
>
>
> We should perhaps set up the db to shut itself down properly automatically
> if it detects that python is shutting down.. We decided against that
> before, since we felt like "we don't own the application lifecycle". WDYT?
>
>
> >
> > Tim
> >
> >
> > On 3 November 2011 17:28, Michael Rene Armida <[hidden email]> wrote:
> > > I'm having some weird problems with threading on shutdown.  As I
> > previously
> > > discovered, using neo4j in a multi-threaded Python app requires
> complete
> > > segregation of neo4j into a single thread.  That all works fine for
> > > creating a database, however, it appears that shutting one down is a
> bit
> > of
> > > a mystery.  I wrote a test program to minimally illustrate the
> > difficulty:
> > >
> > > http://pastebin.com/gg204kae
> > >
> > > Threading is always easy to get wrong, so I added a simple switch to
> > remove
> > > neo4j from the picture.  When I disable neo, the program starts up and
> > > shuts down cleanly.  When I re-enable neo, some of the print statements
> > > never get run, and the app mysteriously exits.
> > >
> > > I am testing this by saving it into a script and running it on the
> > command
> > > line.  Any time after the worker thread finishes creating the neo db, I
> > hit
> > > ctrl-c, and the two scenarios play out as above.
> > >
> > > It seems that the embedded Python API is not exactly at the forefront
> of
> > > popular use, and so I understand if nobody knows what's up with this.
>  I
> > > just thought I should add something, for the record.  I hope my script
> > > doesn't have an obvious bug.
> > >
> > > Thanks,
> > > Michael
> >
>
> I had a discussion with Tobias this morning about threading with JPype, and
> am currently writing a fix that will allow full threading support for
> neo4j-embedded.
>
> I'll make sure to run this script as well after that fix is implemented, to
> make sure it gets fixed as well.
>
> As far as the forefront of use goes, there is a surprising amount of pypy
> downloads of neo4j-embedded (4000 new installs in the last 30 days), so the
> feedback you are providing here is gonna end up helping a lot of people :)
> Thank you so much for taking the time to document the issues you are
> having.
>
> /jake
>
>
> > > _______________________________________________
> > > Neo4j mailing list
> > > [hidden email]
> > > https://lists.neo4j.org/mailman/listinfo/user
> > >
> > _______________________________________________
> > Neo4j mailing list
> > [hidden email]
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
>
> --
> Jacob Hansson
> Phone: +46 (0) 763503395
> Twitter: @jakewins
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
>
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Python embedding: problems with shutdown and threads

Jacob Hansson-2
On Thu, Nov 3, 2011 at 3:24 PM, Michael Rene Armida <[hidden email]> wrote:

> That's awesome news.  I will check it out when it's ready.
>
> I would suggest wrapping up the lifecycle management in a context manager.
>  You're already using the GraphDatabase object's __enter__ and __exit__ to
> manage transactions, so you'd probably have to add a second object.
>
> Would you like me to add a bug somewhere?  I could turn that script into a
> unit test, which should make it easier to evaluate.
>

It's fine as it is for now, I've written a unit test that checks that
threading works (which it seems to do, although there are still a few
issues), and I'll add your script to that test suite as well once I get
this to pass.

Unless I hit any snags, should be pushed later today or tomorrow.

There is no need for it now, but if you report bugs in the future, there is
a bug tracker active for the project on github. Reporting bugs there is
probably preferable to reporting them here, since that makes it easy to
keep track of and prioritize work.

https://github.com/neo4j/python-embedded

/jake


>
> On Thu, Nov 3, 2011 at 3:08 AM, Jacob Hansson <
> [hidden email]> wrote:
>
> > On Thu, Nov 3, 2011 at 6:55 AM, Tim McNamara <
> [hidden email]
> > >wrote:
> >
> > > Here are a few notes of my own from using Neo4j with Python embedded.
> > > I know that this isn't the right place .
> > >
> > > Ubuntu doesn't set $JAVA_HOME. I ended up hard coding the location of
> my
> > > JVM.
> > >
> > > I seem to encounter unclean shutdowns all the time when running with
> > > the iPython REPL. Is there a db.shutdown() command that I've been
> > > neglecting to use?
> > >
> >
> > Yeah, as long as you call db.shutdown(), things should be ok. See here:
> >
> >
> http://docs.neo4j.org/chunked/snapshot/python-embedded-reference-core.html
> >
> >
> > We should perhaps set up the db to shut itself down properly
> automatically
> > if it detects that python is shutting down.. We decided against that
> > before, since we felt like "we don't own the application lifecycle".
> WDYT?
> >
> >
> > >
> > > Tim
> > >
> > >
> > > On 3 November 2011 17:28, Michael Rene Armida <[hidden email]> wrote:
> > > > I'm having some weird problems with threading on shutdown.  As I
> > > previously
> > > > discovered, using neo4j in a multi-threaded Python app requires
> > complete
> > > > segregation of neo4j into a single thread.  That all works fine for
> > > > creating a database, however, it appears that shutting one down is a
> > bit
> > > of
> > > > a mystery.  I wrote a test program to minimally illustrate the
> > > difficulty:
> > > >
> > > > http://pastebin.com/gg204kae
> > > >
> > > > Threading is always easy to get wrong, so I added a simple switch to
> > > remove
> > > > neo4j from the picture.  When I disable neo, the program starts up
> and
> > > > shuts down cleanly.  When I re-enable neo, some of the print
> statements
> > > > never get run, and the app mysteriously exits.
> > > >
> > > > I am testing this by saving it into a script and running it on the
> > > command
> > > > line.  Any time after the worker thread finishes creating the neo
> db, I
> > > hit
> > > > ctrl-c, and the two scenarios play out as above.
> > > >
> > > > It seems that the embedded Python API is not exactly at the forefront
> > of
> > > > popular use, and so I understand if nobody knows what's up with this.
> >  I
> > > > just thought I should add something, for the record.  I hope my
> script
> > > > doesn't have an obvious bug.
> > > >
> > > > Thanks,
> > > > Michael
> > >
> >
> > I had a discussion with Tobias this morning about threading with JPype,
> and
> > am currently writing a fix that will allow full threading support for
> > neo4j-embedded.
> >
> > I'll make sure to run this script as well after that fix is implemented,
> to
> > make sure it gets fixed as well.
> >
> > As far as the forefront of use goes, there is a surprising amount of pypy
> > downloads of neo4j-embedded (4000 new installs in the last 30 days), so
> the
> > feedback you are providing here is gonna end up helping a lot of people
> :)
> > Thank you so much for taking the time to document the issues you are
> > having.
> >
> > /jake
> >
> >
> > > > _______________________________________________
> > > > Neo4j mailing list
> > > > [hidden email]
> > > > https://lists.neo4j.org/mailman/listinfo/user
> > > >
> > > _______________________________________________
> > > Neo4j mailing list
> > > [hidden email]
> > > https://lists.neo4j.org/mailman/listinfo/user
> > >
> >
> >
> >
> > --
> > Jacob Hansson
> > Phone: +46 (0) 763503395
> > Twitter: @jakewins
> > _______________________________________________
> > Neo4j mailing list
> > [hidden email]
> > https://lists.neo4j.org/mailman/listinfo/user
> >
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
>



--
Jacob Hansson
Phone: +46 (0) 763503395
Twitter: @jakewins
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Python embedding: problems with shutdown and threads

Lundberg
In reply to this post by Jacob Hansson-2
On 2011-11-03 11:08, Jacob Hansson wrote:

> On Thu, Nov 3, 2011 at 6:55 AM, Tim McNamara <[hidden email]>wrote:
>
>> Here are a few notes of my own from using Neo4j with Python embedded.
>> I know that this isn't the right place .
>>
>> Ubuntu doesn't set $JAVA_HOME. I ended up hard coding the location of my
>> JVM.
>>
>> I seem to encounter unclean shutdowns all the time when running with
>> the iPython REPL. Is there a db.shutdown() command that I've been
>> neglecting to use?
>>
>
> Yeah, as long as you call db.shutdown(), things should be ok. See here:
>
> http://docs.neo4j.org/chunked/snapshot/python-embedded-reference-core.html
>
>
> We should perhaps set up the db to shut itself down properly automatically
> if it detects that python is shutting down.. We decided against that
> before, since we felt like "we don't own the application lifecycle". WDYT?
>

I found this snippet to work great for my modules:
def _close_db():
    try:
        neo4jdb.shutdown()
    except NameError:
        print 'Could not shutdown Neo4j database. Is it open in \
another process?'

import atexit
atexit.register(_close_db)

>
>>
>> Tim
>>
>>
>> On 3 November 2011 17:28, Michael Rene Armida <[hidden email]> wrote:
>>> I'm having some weird problems with threading on shutdown.  As I
>> previously
>>> discovered, using neo4j in a multi-threaded Python app requires complete
>>> segregation of neo4j into a single thread.  That all works fine for
>>> creating a database, however, it appears that shutting one down is a bit
>> of
>>> a mystery.  I wrote a test program to minimally illustrate the
>> difficulty:
>>>
>>> http://pastebin.com/gg204kae
>>>
>>> Threading is always easy to get wrong, so I added a simple switch to
>> remove
>>> neo4j from the picture.  When I disable neo, the program starts up and
>>> shuts down cleanly.  When I re-enable neo, some of the print statements
>>> never get run, and the app mysteriously exits.
>>>
>>> I am testing this by saving it into a script and running it on the
>> command
>>> line.  Any time after the worker thread finishes creating the neo db, I
>> hit
>>> ctrl-c, and the two scenarios play out as above.
>>>
>>> It seems that the embedded Python API is not exactly at the forefront of
>>> popular use, and so I understand if nobody knows what's up with this.  I
>>> just thought I should add something, for the record.  I hope my script
>>> doesn't have an obvious bug.
>>>
>>> Thanks,
>>> Michael
>>
>
> I had a discussion with Tobias this morning about threading with JPype, and
> am currently writing a fix that will allow full threading support for
> neo4j-embedded.
>
> I'll make sure to run this script as well after that fix is implemented, to
> make sure it gets fixed as well.
>
> As far as the forefront of use goes, there is a surprising amount of pypy
> downloads of neo4j-embedded (4000 new installs in the last 30 days), so the
> feedback you are providing here is gonna end up helping a lot of people :)
> Thank you so much for taking the time to document the issues you are having.
>
> /jake
>
>
>>> _______________________________________________
>>> Neo4j mailing list
>>> [hidden email]
>>> https://lists.neo4j.org/mailman/listinfo/user
>>>
>> _______________________________________________
>> Neo4j mailing list
>> [hidden email]
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>


--
Johan Lundberg
NORDUnet A/S
[hidden email]
+46730714375
Tulegatan 11
113 53 Stockholm
Sweden
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Python embedding: problems with shutdown and threads

Jacob Hansson-2
I've just pushed an update to handle the attachThreadToJVM stuff
automatically, I ran your test, Michael, on the DB after that, and it seems
to shut down properly. You may want to double check though.

You can build this now, by checking out
https://github.com/neo4j/python-embedded/ and following the instructions in
the readme.


This will be part of the 1.5 release, which is currently a delayed, but can
probably be expected within a week or two.

/Jake

On Thu, Nov 3, 2011 at 4:48 PM, Johan Lundberg <[hidden email]> wrote:

> On 2011-11-03 11:08, Jacob Hansson wrote:
> > On Thu, Nov 3, 2011 at 6:55 AM, Tim McNamara <
> [hidden email]>wrote:
> >
> >> Here are a few notes of my own from using Neo4j with Python embedded.
> >> I know that this isn't the right place .
> >>
> >> Ubuntu doesn't set $JAVA_HOME. I ended up hard coding the location of my
> >> JVM.
> >>
> >> I seem to encounter unclean shutdowns all the time when running with
> >> the iPython REPL. Is there a db.shutdown() command that I've been
> >> neglecting to use?
> >>
> >
> > Yeah, as long as you call db.shutdown(), things should be ok. See here:
> >
> >
> http://docs.neo4j.org/chunked/snapshot/python-embedded-reference-core.html
> >
> >
> > We should perhaps set up the db to shut itself down properly
> automatically
> > if it detects that python is shutting down.. We decided against that
> > before, since we felt like "we don't own the application lifecycle".
> WDYT?
> >
>
> I found this snippet to work great for my modules:
> def _close_db():
>    try:
>        neo4jdb.shutdown()
>    except NameError:
>        print 'Could not shutdown Neo4j database. Is it open in \
> another process?'
>
> import atexit
> atexit.register(_close_db)
>
>
This looks like a good option, I'll create an issue for adding this.

/Jake


> >
> >>
> >> Tim
> >>
> >>
> >> On 3 November 2011 17:28, Michael Rene Armida <[hidden email]> wrote:
> >>> I'm having some weird problems with threading on shutdown.  As I
> >> previously
> >>> discovered, using neo4j in a multi-threaded Python app requires
> complete
> >>> segregation of neo4j into a single thread.  That all works fine for
> >>> creating a database, however, it appears that shutting one down is a
> bit
> >> of
> >>> a mystery.  I wrote a test program to minimally illustrate the
> >> difficulty:
> >>>
> >>> http://pastebin.com/gg204kae
> >>>
> >>> Threading is always easy to get wrong, so I added a simple switch to
> >> remove
> >>> neo4j from the picture.  When I disable neo, the program starts up and
> >>> shuts down cleanly.  When I re-enable neo, some of the print statements
> >>> never get run, and the app mysteriously exits.
> >>>
> >>> I am testing this by saving it into a script and running it on the
> >> command
> >>> line.  Any time after the worker thread finishes creating the neo db, I
> >> hit
> >>> ctrl-c, and the two scenarios play out as above.
> >>>
> >>> It seems that the embedded Python API is not exactly at the forefront
> of
> >>> popular use, and so I understand if nobody knows what's up with this.
>  I
> >>> just thought I should add something, for the record.  I hope my script
> >>> doesn't have an obvious bug.
> >>>
> >>> Thanks,
> >>> Michael
> >>
> >
> > I had a discussion with Tobias this morning about threading with JPype,
> and
> > am currently writing a fix that will allow full threading support for
> > neo4j-embedded.
> >
> > I'll make sure to run this script as well after that fix is implemented,
> to
> > make sure it gets fixed as well.
> >
> > As far as the forefront of use goes, there is a surprising amount of pypy
> > downloads of neo4j-embedded (4000 new installs in the last 30 days), so
> the
> > feedback you are providing here is gonna end up helping a lot of people
> :)
> > Thank you so much for taking the time to document the issues you are
> having.
> >
> > /jake
> >
> >
> >>> _______________________________________________
> >>> Neo4j mailing list
> >>> [hidden email]
> >>> https://lists.neo4j.org/mailman/listinfo/user
> >>>
> >> _______________________________________________
> >> Neo4j mailing list
> >> [hidden email]
> >> https://lists.neo4j.org/mailman/listinfo/user
> >>
> >
> >
> >
>
>
> --
> Johan Lundberg
> NORDUnet A/S
> [hidden email]
> +46730714375
> Tulegatan 11
> 113 53 Stockholm
> Sweden
> _______________________________________________
> Neo4j mailing list
> [hidden email]
> https://lists.neo4j.org/mailman/listinfo/user
>



--
Jacob Hansson
Phone: +46 (0) 763503395
Twitter: @jakewins
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Neo4j] Python embedding: problems with shutdown and threads

marmida
You are awesome.  I'll check this out soon; I switched my project back to
using sqlite, but I definitely want to use neo, so I'll give it a whirl.

On Fri, Nov 4, 2011 at 4:12 AM, Jacob Hansson <
[hidden email]> wrote:

> I've just pushed an update to handle the attachThreadToJVM stuff
> automatically, I ran your test, Michael, on the DB after that, and it seems
> to shut down properly. You may want to double check though.
>
> You can build this now, by checking out
> https://github.com/neo4j/python-embedded/ and following the instructions
> in
> the readme.
>
>
> This will be part of the 1.5 release, which is currently a delayed, but can
> probably be expected within a week or two.
>
> /Jake
>
>
_______________________________________________
Neo4j mailing list
[hidden email]
https://lists.neo4j.org/mailman/listinfo/user
Loading...