Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-30
17:26:44 openstackgerrit Eric Fried proposed openstack/nova master: Use a static resource tracker in compute manager https://review.openstack.org/620711
17:26:46 efried leakypipes: This all ought to be ready now ^
17:26:58 leakypipes efried: k
17:28:39 openstackgerrit Eric Fried proposed openstack/nova master: Turn off rp association refresh in nova-next https://review.openstack.org/616033
19:35:18 pacharya Hi I ran into weird issue where nova compute received empty port list for an instance during heal instance info cache. This caused the instance info cache for that instance to be set to empty list and now the APIs are not returning any network information. In the subsequent run neutron returned correct port list but nova does not fix the instance info cache so it remains broken. Any ideas on how to fix this?
19:36:06 SteelyDan mriedem: ^
19:40:21 mriedem pacharya: https://review.openstack.org/#/c/591607/
19:41:19 mriedem without ^ the heal_instance_info_cache and network-changed events just refresh the cache from the cache...which is why it doesn't help you
19:41:42 mriedem unfortunately you need to rebuild the cache somehow, and off the top of my head i think that might mean attaching/detaching a port to/from the server,
19:41:43 mriedem if possible,
19:41:55 mriedem could you just attach a dummy port to the server and then immediately detach it?
19:42:11 mriedem otherwise you might have to try migrating the server, but i'm not sure if that will rebuild the network info cache for the instance
19:44:07 mriedem looking at the code, attaching a new port might just put that port into the cache...
19:44:32 mriedem so i'm not really sure what to tell you beside look at https://review.openstack.org/#/c/591607/
19:46:13 mriedem i know we have https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-reset-network but i don't think that will help
19:58:03 pacharya Thanks ! @mriedem. I did try detaching and attaching the port but that did not seem to help. Will try it again just to be sure.
20:01:08 mriedem pacharya: as i said, i think that at most will just refresh the cache for that single port - but if that's the only port attached to the server then that should be enough
20:06:55 pacharya Yeah and you are also correct about reset-network. that does not work either. I will look into the fix that you pointed out
20:20:25 zzzeek cdent: so with https://review.openstack.org/#/c/614350/18, you tried putting the oslo fixtures into placement/tests/fixtures.py and it blew up ?
20:34:43 pacharya mriedem: will this get backported to queens https://review.openstack.org/#/c/591607/ ?
20:34:55 pacharya just curious
20:35:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Use new ``initial_xxx_allocation_ratio`` CONF https://review.openstack.org/602804
20:35:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Add compute_node ratio online data migration script https://review.openstack.org/609995
20:35:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Add ratio online data migration when load compute node https://review.openstack.org/613499
20:36:00 zzzeek cdent: ummmm....it doesnt' seem to be provisioning anonymously named databases....odd
20:36:06 zzzeek wonder how it works
20:36:06 mriedem leakypipes: i have gone through the first 3 of 5 changes in that series, you might want to start warming up to it sooner than later since it's in a runway slot (for the 2nd time) and you are familiar given it was your spec
20:36:40 mriedem pacharya: idk, i haven't looked at it in awhile - maciej has taken it over
20:36:59 mriedem pacharya: i would think so though if we can
20:40:33 mriedem jackding: oh boy the io semaphore unit test changes are causing a race http://logs.openstack.org/22/606122/7/check/openstack-tox-py27/d70a4d5/testr_results.html.gz
20:48:10 mriedem looks like mdbooths new evacuate functional tests are also racy
20:56:11 cdent zzzeek: what's there is the result of much experimentation and it does work: i can witness the databases being created and manipulated (turned on query logging in the database logs)
20:56:24 cdent the names of things are being overridden
20:56:38 zzzeek cdent: OK so it makes anonymously named databases in PG / MySQL for the migration tests ?
20:57:32 cdent zzzeek: no, it skips that part because it was getting in way of managing multiople databases in the same process. also, as far as I could tell the anonymous naming was only working for mysql, not for postgresql
20:58:01 zzzeek cdent: do you run multiple tests concurrently in a single process?
20:58:50 sean-k-mooney zzzeek: in the nova tests we have eventlet so maybe
20:58:54 cdent on the postgres side is was always the same name, so given that worked for postgres, I made it work for mysql and sqlite too, and then locked the tests with an external lock so there aren't concurrency issues between processes. There aren't concurrent tests in the same process but there is concurrency inter-process
20:59:07 zzzeek cdent: yes that is all normal
20:59:13 cdent sean-k-mooney: we're talking about the placement side right now
20:59:20 zzzeek cdent: OK so the oslo_db way here is good to go, so I will see if i can make it work
20:59:37 zzzeek cdent: also there's just one enginefacade in placement, it seems
20:59:52 zzzeek cdent: which also makes this straightforwad, I see normal create_all() for sqlite then dispose on the end, that's all good
21:00:15 sean-k-mooney cdent: ah ok well im caling it a day anyway.
21:00:19 cdent zzzeek: when we talked about this before (on that review) you said that the existing fixtures wouldn't be quite adequate, which is how what's there ended up the way it did. you said it was "probably good enough|
21:01:00 zzzeek cdent: i belive that's because early on I thougth i was looking at nova :)
21:01:02 cdent zzzeek: but if there are ways it can be made more tidy, that would be great
21:01:18 zzzeek cdent: the oslo_db fixtures should work here fully and if they dont thats a bug in oslo.db i have to fix
21:01:26 cdent btw: we've got #openstack-placement now for this sort of thing
21:01:37 zzzeek cdent: I had guessed that but was too lazy to try :)
21:01:44 cdent I know how that can be
21:02:04 cdent If you traverse the several rounds of me trying different things on that review, where it went wrong may be obvious to me
21:02:49 cdent lots of different things were tried so it is entirely possible that multiple factors are making it icky, but it really came down to one issue: trying to have all three of sqlite, postgres and mysql in the same process, in-series
21:02:54 openstack Launchpad bug 1806126 in OpenStack Compute (nova) "LibvirtRbdEvacuateTest and LibvirtFlatEvacuateTest tests race fail" [High,Confirmed]
21:02:54 mriedem https://bugs.launchpad.net/nova/+bug/1806126
21:03:07 mriedem we probably want to revert on that one ^
21:03:40 mriedem https://review.openstack.org/#/c/591733/
21:03:47 mriedem any cores around for revert discussoin?
21:03:53 mriedem friday is the best time for us to fuck the gate over
21:03:57 cdent zzzeek: the in series is what caused the need for the complicated reset business. the situation has probably changed somewhat now that https://review.openstack.org/#/c/619121/ has merged, which changes the Database handling-in-tests
21:04:27 zzzeek cdent: the reset handling here doesn't look that complicated. nova had soemthig much more exotic
21:04:53 cdent i meant the stuff where _started is being manipulated
21:04:58 mriedem zzzeek: unrelated, do you know much about group by and distinct clauses in postgresql?
21:05:06 zzzeek mriedem: sure
21:05:17 mriedem zzzeek: any sort of knowledge on https://review.openstack.org/#/c/619061/ for that would be great
21:05:30 zzzeek mriedem: i would say, it's unusual to be mixing those two things
21:05:35 mriedem my fix is basically based on internet forums...
21:05:43 zzzeek mriedem: also....i dont follow that often, are we supporting postgresql again in openstack ?
21:05:57 mriedem i believe the official stance is "meh"
21:06:04 zzzeek mriedem: OK so no change
21:06:06 mriedem we don't turn away fixes
21:06:16 mriedem but we don't bend over backwards to test and support it either
21:06:23 cdent my understanding is that every now and again mriedem and I try to make it happen
21:06:40 mriedem some neutron and manila people apparently pay attention to pg also
21:07:00 zzzeek mriedem: um, distinct(col_expr) doesnt work on mysql or sqlite unless I'm deeply confused
21:07:05 zzzeek mriedem: how is that working?
21:07:24 mriedem i assumed the underlying sqla plugins were just nooping or something, idk
21:07:30 mriedem i threw up a fix and it made the pg job work again
21:07:35 zzzeek mriedem: also...i dont see why this query uses group_by
21:07:43 cdent zzzeek: it's late and friday for me, so I'm going to watch bad tv and fall asleep. If you've got thoughts on that placement alembic stuff, please let me know, either on that review or on some wip that you invite me to look at
21:07:52 mriedem as noted in my comments, it might not be the best query to start with
21:07:57 cdent everyone have a good weekend
21:07:58 mriedem and simply rewriting the query could fix it for both
21:08:07 mriedem o/
21:10:09 zzzeek mriedem: does InstanceMapping have any column_property() that points to like func.max() or func.count() or an aggregate of some kind ?
21:10:35 zzzeek cdent: my plan is to make this work and put up a review
21:10:41 zzzeek cdent: or if oslo_db is broke, do taht first
21:11:06 cdent zzzeek: great, whichever it is point it out to me, will be very curious
21:12:55 mriedem zzzeek: this is the model https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_models.py#L131
21:13:43 mriedem so i think the answer is "no"
21:14:20 zzzeek mriedem: sure, it would be extremely weird to map an aggregate
21:15:36 openstack bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9)
21:15:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert "Add regression test for bug 1550919" https://review.openstack.org/621289
21:50:44 mriedem SteelyDan: leakypipes: there is some sage advice on here that i wanted to share with you https://www.siouxfallsshoppingnews.com/publishers-message.html
21:51:06 mriedem "Life is simpler when you plow around the stump."
21:51:13 mriedem i can tell you privately what i think that means
21:56:53 SteelyDan um.
21:57:38 openstackgerrit Merged openstack/nova stable/rocky: Don't use private interface in oslo.service https://review.openstack.org/619360
22:20:54 openstackgerrit Eric Fried proposed openstack/nova master: Consolidate inventory refresh https://review.openstack.org/615695
22:20:54 openstackgerrit Eric Fried proposed openstack/nova master: Reduce calls to placement from _ensure https://review.openstack.org/615677

Earlier   Later