Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-15
19:40:53 efried and it would be none of RT's business.
19:41:52 efried the more we move claimables into placement, the more we can delegate the claim logic for same to the virt driver.
19:42:05 efried so e.g. PCI devices would eventually become virt driver business.
19:42:23 efried and - as we were discussing this morning - not even stored in the db anymore.
19:44:05 cdent well that would be lovely
19:57:18 openstack Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New]
19:57:18 slaweq hi, is there any n-meta-api service expert here? Can You take a look at https://bugs.launchpad.net/neutron/+bug/1836642 from Nova PoV? Thx in advance :)
20:01:06 openstack bug 1836595 in neutron "test_server_connectivity_cold_migration_revert failing" [Undecided,New] https://launchpad.net/bugs/1836595
20:01:06 efried artom: catching up, are you filing an elastic-recheck profile for bug 1836595 ?
20:02:10 artom efried, I could - should I be? :)
20:02:48 artom efried, I'm kinda hoping https://review.opendev.org/#/c/670848/ would merge though
20:03:42 artom Speaking of which, gmann ^^ :)
20:03:51 artom (If you're around, not sure about your TZ)
20:05:11 efried artom: This is interesting, the failure on that skip patch http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/testr_results.html.gz
20:06:19 artom efried, looks like feral packets to me
20:06:26 efried conflict deleting allocations, looks like it's tempest itself that's being a placement client. Without having looked at the code, one wonders whether it should be retrying
20:06:52 efried feral packets? how so?
20:06:53 artom Yeah, tempest makes a point not to use any Python clients
20:07:09 artom efried, heh, a lazy way of saying "something unrelated I can't be arsed to debug" :P
20:07:22 artom So it'll do the GET requests itself
20:07:27 efried there is no placement client, my point is that this isn't tempest calling something in nova that's talking to placement and getting a conflict.
20:07:39 efried this is tempest talking directly to placement
20:07:49 artom And that's bad?
20:08:14 efried it means there's likely an opportunity to harden the tempest code so this failure doesn't happen anymore.
20:08:59 artom *facepalm*, oh just got what you're saying
20:09:11 artom Tempest shouldn't be deleting allocations itself, Nova should be doing it
20:09:47 artom I think that response is from Nova though
20:11:29 efried mmm, yes, it looks like you're right.
20:11:54 artom That test is really confusing
20:13:16 efried if the error is coming from nova, it means we have a bug in nova.
20:16:08 artom Well, looks like tempest is creating a server, not waiting for it to become ACTIVE, then immediately deleting it
20:16:33 artom So I suspect that makes Nova hit a race between the build process and the delete request
20:16:49 cdent artom: that sounds right
20:17:00 artom Short term that can be "fixed" by making Tempest wait
20:17:22 artom Longer term I guess we'll need to stick a lock in Nova somewhere
20:18:49 cdent placement log entries are near http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-placement-api.txt.gz#_Jul_15_17_27_35_283720
20:20:03 cdent handling for the req id starts here http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_33_968264
20:20:18 artom Actually from the little I know about placement it should handle those kinds of races with the generation thing, right?
20:20:27 artom So maybe Nova just needs to handle Placement error better
20:20:28 efried the generation thing is exactly what's happening here
20:20:55 efried alloc deletion was specifically set up so that you couldn't race deletion with some other consumer op
20:21:24 efried so this guard is doing exactly what it's supposed to, and it's the overarching operation that's broken. As you say, trying to delete while creating.
20:21:30 cdent "instance disappeared during build" http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_34_806925
20:21:49 efried only question is where to fix it.
20:22:24 efried it would be possible to redrive the alloc delete until it works. But that's still potentially racy if the other op is trying to create the alloc at the same time.
20:23:24 efried there's a couple races here actually.
20:23:29 artom @instance_state(ACTIVE) for delete :D
20:23:46 efried The other one is if the deletion goes through before the allocation is even created
20:25:12 efried it will not raise an exception (it returns False, but the caller doesn't check that) so we'll end up with a leaked allocation
20:26:15 cdent efried, artom if you end up creating a bug about this, please let me know what it is so I can follow along, tomorrow
20:27:29 artom efried, mind doing it? You have a better grasp of that stuff, and I'm off in 30 minutes anyways, daycare duty
20:27:46 efried ack
20:48:57 artom *snerk*
20:49:00 openstack Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj)
20:49:00 artom https://bugs.launchpad.net/nova/+bug/1836204
20:49:08 artom I guess it hates blacks
20:49:47 artom *shakes head* I'm really sorry
22:57:32 efried dansmith: Do you feel we need a bp/spec for claim_for_instance?
22:58:09 efried there should be no API, db, object, conf, upgrade, or doc impacts
#openstack-nova - 2019-07-16
00:27:04 openstackgerrit Merged openstack/nova master: doc: Add links to novaclient contributor guide https://review.opendev.org/669028
00:46:31 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Make the '--config-drive' be a boolean option https://review.opendev.org/653683
02:23:22 openstackgerrit Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935
02:25:59 openstackgerrit Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790
03:05:44 openstackgerrit Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935
04:46:47 openstackgerrit Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790
05:09:31 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix a unit test for pre_live_migration method https://review.opendev.org/670948
06:27:21 openstackgerrit Alex Xu proposed openstack/nova master: Move the allocation retrieving early https://review.opendev.org/670783
06:27:21 openstackgerrit Alex Xu proposed openstack/nova master: Add the virt driver interface for claim and unclaim the devices https://review.opendev.org/670782
06:27:22 openstackgerrit Alex Xu proposed openstack/nova master: Claim/unclaim the deivces when rebuilding the instance https://review.opendev.org/670785
06:27:22 openstackgerrit Alex Xu proposed openstack/nova master: Claim and unclaim deivces when spawn and destory the instance https://review.opendev.org/670784
06:27:23 openstackgerrit Alex Xu proposed openstack/nova master: Populate the existing mediated devices in the libvirt device manager https://review.opendev.org/670787
06:27:23 openstackgerrit Alex Xu proposed openstack/nova master: Claim/unclaim the devices when resizing instance https://review.opendev.org/670786
06:38:54 openstackgerrit Yongli He proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765
07:06:34 openstackgerrit melanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.opendev.org/507486
07:51:39 openstackgerrit Lee Yarwood proposed openstack/nova stable/stein: Add 'path' query parameter to console access url https://review.opendev.org/670972
08:28:20 tssurya takashin: are you around ?
08:29:33 takashin tssurya: yes.
08:29:43 tssurya you review comment here: https://review.opendev.org/#/c/666792/2/novaclient/tests/unit/v2/test_shell.py@4277
08:29:51 tssurya can I ask why 74 was added there in the first place ?
08:30:13 tssurya could you respond to the question when you have time ? thanks in advance
08:30:27 tssurya your*
08:30:41 takashin tssurya: okay
08:38:57 tssurya @cores: reviews for https://review.opendev.org/#/c/645611/ would be appreciated, its in the runways. Thanks in advance
09:13:52 openstackgerrit Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/666792
09:25:26 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: doc: Clarify versioned wrapped method https://review.opendev.org/670989
09:37:04 slaweq sean-k-mooney: hi, can You help me with one bug?
09:37:25 openstack Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New]
09:37:25 slaweq sean-k-mooney: https://bugs.launchpad.net/neutron/+bug/1836642
10:15:54 openstackgerrit YumengBao proposed openstack/nova master: add cyborg tempest job https://review.opendev.org/670999
10:47:26 openstackgerrit Merged openstack/python-novaclient master: doc: Clarify versioned wrapped method https://review.opendev.org/670989
10:48:42 sean-k-mooney slaweq: its posible that that is just a result of the limited resouces we have in the gate
10:49:48 slaweq sean-k-mooney: yes, it's possible but it is causing many failures various jobs :/
10:51:34 gibi tssurya: left comments in https://review.opendev.org/#/c/645611/
10:51:50 sean-k-mooney the first two reuest are long in http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_16_704979
10:52:13 sean-k-mooney that one is 16 second and the previous one was 1.4
10:52:29 sean-k-mooney but the request after that are 2ms
10:55:47 openstackgerrit Merged openstack/nova master: Add Python 3 Train unit tests https://review.opendev.org/669437
10:58:03 sean-k-mooney slaweq: it look like that was the first request on that worker thread and it had to initalise the db connection andproably some other setup stuff
10:59:42 sean-k-mooney slaweq: we can see the request is recived here http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_00_523076
11:00:32 slaweq sean-k-mooney: yes, and response is send here: http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_16_704979

Earlier   Later