Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-15
19:04:19 bbobrov i will reply to the comments now
19:04:56 artom What happened to aspiers?
19:21:41 efried was wondering same
19:35:04 openstackgerrit Eric Fried proposed openstack/nova master: Use Adapter global_request_id kwarg https://review.opendev.org/670907
19:35:15 cdent efried: the semaphore thing is indirectly related to https://bugs.launchpad.net/nova/+bug/1835958 (power states not under the same lock, but in the realm of nova-compute performance)
19:35:15 openstack Launchpad bug 1835958 in OpenStack Compute (nova) "Nova sync power state on large clusters causes poor performance" [Undecided,New]
19:37:04 efried cdent: Only slightly related, I've been talking to alex_xu about moving more stuff under that semaphore.
19:37:16 efried specifically driver-specific claim stuff
19:37:21 efried Not sure if you were around for those conversations.
19:37:33 cdent not that I'm aware of
19:38:33 cdent it's a problem for high throughput nova-computes (like in vmware where the nova-compute is a chokepoint, rather than a member of a nice bit of parallelism like in a big kvm cloud)
19:40:46 efried cdent: well, the plan is to start delegating virt-specific claimage to the virt driver itself. I could see where vmware could take its own semaphore on a specific internal nodeything and background the real claim job, so the next one could come in and get started (but on a different nodeything).
19:40:53 efried and it would be none of RT's business.
19:41:52 efried the more we move claimables into placement, the more we can delegate the claim logic for same to the virt driver.
19:42:05 efried so e.g. PCI devices would eventually become virt driver business.
19:42:23 efried and - as we were discussing this morning - not even stored in the db anymore.
19:44:05 cdent well that would be lovely
19:57:18 slaweq hi, is there any n-meta-api service expert here? Can You take a look at https://bugs.launchpad.net/neutron/+bug/1836642 from Nova PoV? Thx in advance :)
19:57:18 openstack Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New]
20:01:06 efried artom: catching up, are you filing an elastic-recheck profile for bug 1836595 ?
20:01:06 openstack bug 1836595 in neutron "test_server_connectivity_cold_migration_revert failing" [Undecided,New] https://launchpad.net/bugs/1836595
20:02:10 artom efried, I could - should I be? :)
20:02:48 artom efried, I'm kinda hoping https://review.opendev.org/#/c/670848/ would merge though
20:03:42 artom Speaking of which, gmann ^^ :)
20:03:51 artom (If you're around, not sure about your TZ)
20:05:11 efried artom: This is interesting, the failure on that skip patch http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/testr_results.html.gz
20:06:19 artom efried, looks like feral packets to me
20:06:26 efried conflict deleting allocations, looks like it's tempest itself that's being a placement client. Without having looked at the code, one wonders whether it should be retrying
20:06:52 efried feral packets? how so?
20:06:53 artom Yeah, tempest makes a point not to use any Python clients
20:07:09 artom efried, heh, a lazy way of saying "something unrelated I can't be arsed to debug" :P
20:07:22 artom So it'll do the GET requests itself
20:07:27 efried there is no placement client, my point is that this isn't tempest calling something in nova that's talking to placement and getting a conflict.
20:07:39 efried this is tempest talking directly to placement
20:07:49 artom And that's bad?
20:08:14 efried it means there's likely an opportunity to harden the tempest code so this failure doesn't happen anymore.
20:08:59 artom *facepalm*, oh just got what you're saying
20:09:11 artom Tempest shouldn't be deleting allocations itself, Nova should be doing it
20:09:47 artom I think that response is from Nova though
20:11:29 efried mmm, yes, it looks like you're right.
20:11:54 artom That test is really confusing
20:13:16 efried if the error is coming from nova, it means we have a bug in nova.
20:16:08 artom Well, looks like tempest is creating a server, not waiting for it to become ACTIVE, then immediately deleting it
20:16:33 artom So I suspect that makes Nova hit a race between the build process and the delete request
20:16:49 cdent artom: that sounds right
20:17:00 artom Short term that can be "fixed" by making Tempest wait
20:17:22 artom Longer term I guess we'll need to stick a lock in Nova somewhere
20:18:49 cdent placement log entries are near http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-placement-api.txt.gz#_Jul_15_17_27_35_283720
20:20:03 cdent handling for the req id starts here http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_33_968264
20:20:18 artom Actually from the little I know about placement it should handle those kinds of races with the generation thing, right?
20:20:27 artom So maybe Nova just needs to handle Placement error better
20:20:28 efried the generation thing is exactly what's happening here
20:20:55 efried alloc deletion was specifically set up so that you couldn't race deletion with some other consumer op
20:21:24 efried so this guard is doing exactly what it's supposed to, and it's the overarching operation that's broken. As you say, trying to delete while creating.
20:21:30 cdent "instance disappeared during build" http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_34_806925
20:21:49 efried only question is where to fix it.
20:22:24 efried it would be possible to redrive the alloc delete until it works. But that's still potentially racy if the other op is trying to create the alloc at the same time.
20:23:24 efried there's a couple races here actually.
20:23:29 artom @instance_state(ACTIVE) for delete :D
20:23:46 efried The other one is if the deletion goes through before the allocation is even created
20:25:12 efried it will not raise an exception (it returns False, but the caller doesn't check that) so we'll end up with a leaked allocation
20:26:15 cdent efried, artom if you end up creating a bug about this, please let me know what it is so I can follow along, tomorrow
20:27:29 artom efried, mind doing it? You have a better grasp of that stuff, and I'm off in 30 minutes anyways, daycare duty
20:27:46 efried ack
20:48:57 artom *snerk*
20:49:00 artom https://bugs.launchpad.net/nova/+bug/1836204
20:49:00 openstack Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj)
20:49:08 artom I guess it hates blacks
20:49:47 artom *shakes head* I'm really sorry
22:57:32 efried dansmith: Do you feel we need a bp/spec for claim_for_instance?
22:58:09 efried there should be no API, db, object, conf, upgrade, or doc impacts
#openstack-nova - 2019-07-16
00:27:04 openstackgerrit Merged openstack/nova master: doc: Add links to novaclient contributor guide https://review.opendev.org/669028
00:46:31 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Make the '--config-drive' be a boolean option https://review.opendev.org/653683
02:23:22 openstackgerrit Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935
02:25:59 openstackgerrit Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790
03:05:44 openstackgerrit Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935
04:46:47 openstackgerrit Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790
05:09:31 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix a unit test for pre_live_migration method https://review.opendev.org/670948
06:27:21 openstackgerrit Alex Xu proposed openstack/nova master: Add the virt driver interface for claim and unclaim the devices https://review.opendev.org/670782
06:27:21 openstackgerrit Alex Xu proposed openstack/nova master: Move the allocation retrieving early https://review.opendev.org/670783
06:27:22 openstackgerrit Alex Xu proposed openstack/nova master: Claim and unclaim deivces when spawn and destory the instance https://review.opendev.org/670784
06:27:22 openstackgerrit Alex Xu proposed openstack/nova master: Claim/unclaim the deivces when rebuilding the instance https://review.opendev.org/670785
06:27:23 openstackgerrit Alex Xu proposed openstack/nova master: Claim/unclaim the devices when resizing instance https://review.opendev.org/670786
06:27:23 openstackgerrit Alex Xu proposed openstack/nova master: Populate the existing mediated devices in the libvirt device manager https://review.opendev.org/670787
06:38:54 openstackgerrit Yongli He proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765
07:06:34 openstackgerrit melanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.opendev.org/507486
07:51:39 openstackgerrit Lee Yarwood proposed openstack/nova stable/stein: Add 'path' query parameter to console access url https://review.opendev.org/670972
08:28:20 tssurya takashin: are you around ?
08:29:33 takashin tssurya: yes.
08:29:43 tssurya you review comment here: https://review.opendev.org/#/c/666792/2/novaclient/tests/unit/v2/test_shell.py@4277
08:29:51 tssurya can I ask why 74 was added there in the first place ?
08:30:13 tssurya could you respond to the question when you have time ? thanks in advance
08:30:27 tssurya your*
08:30:41 takashin tssurya: okay
08:38:57 tssurya @cores: reviews for https://review.opendev.org/#/c/645611/ would be appreciated, its in the runways. Thanks in advance
09:13:52 openstackgerrit Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/666792
09:25:26 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: doc: Clarify versioned wrapped method https://review.opendev.org/670989
09:37:04 slaweq sean-k-mooney: hi, can You help me with one bug?
09:37:25 slaweq sean-k-mooney: https://bugs.launchpad.net/neutron/+bug/1836642
09:37:25 openstack Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New]

Earlier   Later