| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 20:07:09 | artom | efried, heh, a lazy way of saying "something unrelated I can't be arsed to debug" :P | |
| 20:07:22 | artom | So it'll do the GET requests itself | |
| 20:07:27 | efried | there is no placement client, my point is that this isn't tempest calling something in nova that's talking to placement and getting a conflict. | |
| 20:07:39 | efried | this is tempest talking directly to placement | |
| 20:07:49 | artom | And that's bad? | |
| 20:08:14 | efried | it means there's likely an opportunity to harden the tempest code so this failure doesn't happen anymore. | |
| 20:08:59 | artom | *facepalm*, oh just got what you're saying | |
| 20:09:11 | artom | Tempest shouldn't be deleting allocations itself, Nova should be doing it | |
| 20:09:47 | artom | I think that response is from Nova though | |
| 20:11:29 | efried | mmm, yes, it looks like you're right. | |
| 20:11:54 | artom | That test is really confusing | |
| 20:13:16 | efried | if the error is coming from nova, it means we have a bug in nova. | |
| 20:16:08 | artom | Well, looks like tempest is creating a server, not waiting for it to become ACTIVE, then immediately deleting it | |
| 20:16:33 | artom | So I suspect that makes Nova hit a race between the build process and the delete request | |
| 20:16:49 | cdent | artom: that sounds right | |
| 20:17:00 | artom | Short term that can be "fixed" by making Tempest wait | |
| 20:17:22 | artom | Longer term I guess we'll need to stick a lock in Nova somewhere | |
| 20:18:49 | cdent | placement log entries are near http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-placement-api.txt.gz#_Jul_15_17_27_35_283720 | |
| 20:20:03 | cdent | handling for the req id starts here http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_33_968264 | |
| 20:20:18 | artom | Actually from the little I know about placement it should handle those kinds of races with the generation thing, right? | |
| 20:20:27 | artom | So maybe Nova just needs to handle Placement error better | |
| 20:20:28 | efried | the generation thing is exactly what's happening here | |
| 20:20:55 | efried | alloc deletion was specifically set up so that you couldn't race deletion with some other consumer op | |
| 20:21:24 | efried | so this guard is doing exactly what it's supposed to, and it's the overarching operation that's broken. As you say, trying to delete while creating. | |
| 20:21:30 | cdent | "instance disappeared during build" http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_34_806925 | |
| 20:21:49 | efried | only question is where to fix it. | |
| 20:22:24 | efried | it would be possible to redrive the alloc delete until it works. But that's still potentially racy if the other op is trying to create the alloc at the same time. | |
| 20:23:24 | efried | there's a couple races here actually. | |
| 20:23:29 | artom | @instance_state(ACTIVE) for delete :D | |
| 20:23:46 | efried | The other one is if the deletion goes through before the allocation is even created | |
| 20:25:12 | efried | it will not raise an exception (it returns False, but the caller doesn't check that) so we'll end up with a leaked allocation | |
| 20:26:15 | cdent | efried, artom if you end up creating a bug about this, please let me know what it is so I can follow along, tomorrow | |
| 20:27:29 | artom | efried, mind doing it? You have a better grasp of that stuff, and I'm off in 30 minutes anyways, daycare duty | |
| 20:27:46 | efried | ack | |
| 20:48:57 | artom | *snerk* | |
| 20:49:00 | openstack | Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj) | |
| 20:49:00 | artom | https://bugs.launchpad.net/nova/+bug/1836204 | |
| 20:49:08 | artom | I guess it hates blacks | |
| 20:49:47 | artom | *shakes head* I'm really sorry | |
| 22:57:32 | efried | dansmith: Do you feel we need a bp/spec for claim_for_instance? | |
| 22:58:09 | efried | there should be no API, db, object, conf, upgrade, or doc impacts | |
| #openstack-nova - 2019-07-16 | |||
| 00:27:04 | openstackgerrit | Merged openstack/nova master: doc: Add links to novaclient contributor guide https://review.opendev.org/669028 | |
| 00:46:31 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Make the '--config-drive' be a boolean option https://review.opendev.org/653683 | |
| 02:23:22 | openstackgerrit | Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935 | |
| 02:25:59 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790 | |
| 03:05:44 | openstackgerrit | Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935 | |
| 04:46:47 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790 | |
| 05:09:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix a unit test for pre_live_migration method https://review.opendev.org/670948 | |
| 06:27:21 | openstackgerrit | Alex Xu proposed openstack/nova master: Move the allocation retrieving early https://review.opendev.org/670783 | |
| 06:27:21 | openstackgerrit | Alex Xu proposed openstack/nova master: Add the virt driver interface for claim and unclaim the devices https://review.opendev.org/670782 | |
| 06:27:22 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim/unclaim the deivces when rebuilding the instance https://review.opendev.org/670785 | |
| 06:27:22 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim and unclaim deivces when spawn and destory the instance https://review.opendev.org/670784 | |
| 06:27:23 | openstackgerrit | Alex Xu proposed openstack/nova master: Populate the existing mediated devices in the libvirt device manager https://review.opendev.org/670787 | |
| 06:27:23 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim/unclaim the devices when resizing instance https://review.opendev.org/670786 | |
| 06:38:54 | openstackgerrit | Yongli He proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765 | |
| 07:06:34 | openstackgerrit | melanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.opendev.org/507486 | |
| 07:51:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: Add 'path' query parameter to console access url https://review.opendev.org/670972 | |
| 08:28:20 | tssurya | takashin: are you around ? | |
| 08:29:33 | takashin | tssurya: yes. | |
| 08:29:43 | tssurya | you review comment here: https://review.opendev.org/#/c/666792/2/novaclient/tests/unit/v2/test_shell.py@4277 | |
| 08:29:51 | tssurya | can I ask why 74 was added there in the first place ? | |
| 08:30:13 | tssurya | could you respond to the question when you have time ? thanks in advance | |
| 08:30:27 | tssurya | your* | |
| 08:30:41 | takashin | tssurya: okay | |
| 08:38:57 | tssurya | @cores: reviews for https://review.opendev.org/#/c/645611/ would be appreciated, its in the runways. Thanks in advance | |
| 09:13:52 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/666792 | |
| 09:25:26 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: doc: Clarify versioned wrapped method https://review.opendev.org/670989 | |
| 09:37:04 | slaweq | sean-k-mooney: hi, can You help me with one bug? | |
| 09:37:25 | openstack | Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New] | |
| 09:37:25 | slaweq | sean-k-mooney: https://bugs.launchpad.net/neutron/+bug/1836642 | |
| 10:15:54 | openstackgerrit | YumengBao proposed openstack/nova master: add cyborg tempest job https://review.opendev.org/670999 | |
| 10:47:26 | openstackgerrit | Merged openstack/python-novaclient master: doc: Clarify versioned wrapped method https://review.opendev.org/670989 | |
| 10:48:42 | sean-k-mooney | slaweq: its posible that that is just a result of the limited resouces we have in the gate | |
| 10:49:48 | slaweq | sean-k-mooney: yes, it's possible but it is causing many failures various jobs :/ | |
| 10:51:34 | gibi | tssurya: left comments in https://review.opendev.org/#/c/645611/ | |
| 10:51:50 | sean-k-mooney | the first two reuest are long in http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_16_704979 | |
| 10:52:13 | sean-k-mooney | that one is 16 second and the previous one was 1.4 | |
| 10:52:29 | sean-k-mooney | but the request after that are 2ms | |
| 10:55:47 | openstackgerrit | Merged openstack/nova master: Add Python 3 Train unit tests https://review.opendev.org/669437 | |
| 10:58:03 | sean-k-mooney | slaweq: it look like that was the first request on that worker thread and it had to initalise the db connection andproably some other setup stuff | |
| 10:59:42 | sean-k-mooney | slaweq: we can see the request is recived here http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_00_523076 | |
| 11:00:32 | slaweq | sean-k-mooney: yes, and response is send here: http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_16_704979 | |
| 11:00:43 | slaweq | it took 16 seconds to send it back | |
| 11:01:15 | slaweq | after 10 seconds this "cloud-init" like script in cirros is failing and instance don't have SSH key configured thus test fails :/ | |
| 11:01:51 | sean-k-mooney | yes but if you look at the pids it was setting up the db connection on pid 9232 which is the pid that serviced the request | |
| 11:02:17 | slaweq | sean-k-mooney: and what is strange for me, is that only this one request was so slow | |
| 11:02:20 | sean-k-mooney | so im guessing it was delayed becasue the api worker hadn to do some intal setup | |
| 11:02:39 | slaweq | why all other requests around this are much faster? | |
| 11:03:17 | sean-k-mooney | becasue they dont have to inisllise the wsgi app and set up the db connections | |
| 11:03:23 | slaweq | sean-k-mooney: but don't You think it is bug if nova needs sometimes 16 seconds to send public-key? | |
| 11:03:30 | sean-k-mooney | they can reuse all the work that has been done | |
| 11:04:12 | sean-k-mooney | i dont think it actully too 16 second i think it was waiting for the database to return with the reponce | |
| 11:04:32 | slaweq | from user (VM) PoV it was 16 seconds | |
| 11:04:58 | sean-k-mooney | sure | |
| 11:05:32 | sean-k-mooney | but as i said it look like this is because its the frist api request recived by that worker | |
| 11:06:28 | sean-k-mooney | 16 seconds is not really that un resonable given the load that the gate vms are under. we can see if there is a way to speed it up but i dont think this would be an issue in production | |
| 11:09:04 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Get rid of args to RBDDriver.__init__() https://review.opendev.org/668564 | |
| 11:09:37 | slaweq | sean-k-mooney: the problem is that this is an issue in the gate :/ | |
| 11:10:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Move consts from neutronv2/api to constants module https://review.opendev.org/668945 | |
| 11:10:21 | sean-k-mooney | slaweq: yep which we may not be able to fix in nova. im goint to look at the other logs and then ill look at the code | |