| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 20:20:55 | efried | alloc deletion was specifically set up so that you couldn't race deletion with some other consumer op | |
| 20:21:24 | efried | so this guard is doing exactly what it's supposed to, and it's the overarching operation that's broken. As you say, trying to delete while creating. | |
| 20:21:30 | cdent | "instance disappeared during build" http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_34_806925 | |
| 20:21:49 | efried | only question is where to fix it. | |
| 20:22:24 | efried | it would be possible to redrive the alloc delete until it works. But that's still potentially racy if the other op is trying to create the alloc at the same time. | |
| 20:23:24 | efried | there's a couple races here actually. | |
| 20:23:29 | artom | @instance_state(ACTIVE) for delete :D | |
| 20:23:46 | efried | The other one is if the deletion goes through before the allocation is even created | |
| 20:25:12 | efried | it will not raise an exception (it returns False, but the caller doesn't check that) so we'll end up with a leaked allocation | |
| 20:26:15 | cdent | efried, artom if you end up creating a bug about this, please let me know what it is so I can follow along, tomorrow | |
| 20:27:29 | artom | efried, mind doing it? You have a better grasp of that stuff, and I'm off in 30 minutes anyways, daycare duty | |
| 20:27:46 | efried | ack | |
| 20:48:57 | artom | *snerk* | |
| 20:49:00 | artom | https://bugs.launchpad.net/nova/+bug/1836204 | |
| 20:49:00 | openstack | Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj) | |
| 20:49:08 | artom | I guess it hates blacks | |
| 20:49:47 | artom | *shakes head* I'm really sorry | |
| 22:57:32 | efried | dansmith: Do you feel we need a bp/spec for claim_for_instance? | |
| 22:58:09 | efried | there should be no API, db, object, conf, upgrade, or doc impacts | |
| #openstack-nova - 2019-07-16 | |||
| 00:27:04 | openstackgerrit | Merged openstack/nova master: doc: Add links to novaclient contributor guide https://review.opendev.org/669028 | |
| 00:46:31 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Make the '--config-drive' be a boolean option https://review.opendev.org/653683 | |
| 02:23:22 | openstackgerrit | Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935 | |
| 02:25:59 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790 | |
| 03:05:44 | openstackgerrit | Ming Jin proposed openstack/nova master: Nova handle empty vcpu_pin_set scenario https://review.opendev.org/670935 | |
| 04:46:47 | openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.75 - show server topology https://review.opendev.org/670790 | |
| 05:09:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix a unit test for pre_live_migration method https://review.opendev.org/670948 | |
| 06:27:21 | openstackgerrit | Alex Xu proposed openstack/nova master: Add the virt driver interface for claim and unclaim the devices https://review.opendev.org/670782 | |
| 06:27:21 | openstackgerrit | Alex Xu proposed openstack/nova master: Move the allocation retrieving early https://review.opendev.org/670783 | |
| 06:27:22 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim and unclaim deivces when spawn and destory the instance https://review.opendev.org/670784 | |
| 06:27:22 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim/unclaim the deivces when rebuilding the instance https://review.opendev.org/670785 | |
| 06:27:23 | openstackgerrit | Alex Xu proposed openstack/nova master: Claim/unclaim the devices when resizing instance https://review.opendev.org/670786 | |
| 06:27:23 | openstackgerrit | Alex Xu proposed openstack/nova master: Populate the existing mediated devices in the libvirt device manager https://review.opendev.org/670787 | |
| 06:38:54 | openstackgerrit | Yongli He proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765 | |
| 07:06:34 | openstackgerrit | melanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.opendev.org/507486 | |
| 07:51:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: Add 'path' query parameter to console access url https://review.opendev.org/670972 | |
| 08:28:20 | tssurya | takashin: are you around ? | |
| 08:29:33 | takashin | tssurya: yes. | |
| 08:29:43 | tssurya | you review comment here: https://review.opendev.org/#/c/666792/2/novaclient/tests/unit/v2/test_shell.py@4277 | |
| 08:29:51 | tssurya | can I ask why 74 was added there in the first place ? | |
| 08:30:13 | tssurya | could you respond to the question when you have time ? thanks in advance | |
| 08:30:27 | tssurya | your* | |
| 08:30:41 | takashin | tssurya: okay | |
| 08:38:57 | tssurya | @cores: reviews for https://review.opendev.org/#/c/645611/ would be appreciated, its in the runways. Thanks in advance | |
| 09:13:52 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/666792 | |
| 09:25:26 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: doc: Clarify versioned wrapped method https://review.opendev.org/670989 | |
| 09:37:04 | slaweq | sean-k-mooney: hi, can You help me with one bug? | |
| 09:37:25 | slaweq | sean-k-mooney: https://bugs.launchpad.net/neutron/+bug/1836642 | |
| 09:37:25 | openstack | Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New] | |
| 10:15:54 | openstackgerrit | YumengBao proposed openstack/nova master: add cyborg tempest job https://review.opendev.org/670999 | |
| 10:47:26 | openstackgerrit | Merged openstack/python-novaclient master: doc: Clarify versioned wrapped method https://review.opendev.org/670989 | |
| 10:48:42 | sean-k-mooney | slaweq: its posible that that is just a result of the limited resouces we have in the gate | |
| 10:49:48 | slaweq | sean-k-mooney: yes, it's possible but it is causing many failures various jobs :/ | |
| 10:51:34 | gibi | tssurya: left comments in https://review.opendev.org/#/c/645611/ | |
| 10:51:50 | sean-k-mooney | the first two reuest are long in http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_16_704979 | |
| 10:52:13 | sean-k-mooney | that one is 16 second and the previous one was 1.4 | |
| 10:52:29 | sean-k-mooney | but the request after that are 2ms | |
| 10:55:47 | openstackgerrit | Merged openstack/nova master: Add Python 3 Train unit tests https://review.opendev.org/669437 | |
| 10:58:03 | sean-k-mooney | slaweq: it look like that was the first request on that worker thread and it had to initalise the db connection andproably some other setup stuff | |
| 10:59:42 | sean-k-mooney | slaweq: we can see the request is recived here http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_00_523076 | |
| 11:00:32 | slaweq | sean-k-mooney: yes, and response is send here: http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-n-api-meta.txt.gz#_Jul_11_23_43_16_704979 | |
| 11:00:43 | slaweq | it took 16 seconds to send it back | |
| 11:01:15 | slaweq | after 10 seconds this "cloud-init" like script in cirros is failing and instance don't have SSH key configured thus test fails :/ | |
| 11:01:51 | sean-k-mooney | yes but if you look at the pids it was setting up the db connection on pid 9232 which is the pid that serviced the request | |
| 11:02:17 | slaweq | sean-k-mooney: and what is strange for me, is that only this one request was so slow | |
| 11:02:20 | sean-k-mooney | so im guessing it was delayed becasue the api worker hadn to do some intal setup | |
| 11:02:39 | slaweq | why all other requests around this are much faster? | |
| 11:03:17 | sean-k-mooney | becasue they dont have to inisllise the wsgi app and set up the db connections | |
| 11:03:23 | slaweq | sean-k-mooney: but don't You think it is bug if nova needs sometimes 16 seconds to send public-key? | |
| 11:03:30 | sean-k-mooney | they can reuse all the work that has been done | |
| 11:04:12 | sean-k-mooney | i dont think it actully too 16 second i think it was waiting for the database to return with the reponce | |
| 11:04:32 | slaweq | from user (VM) PoV it was 16 seconds | |
| 11:04:58 | sean-k-mooney | sure | |
| 11:05:32 | sean-k-mooney | but as i said it look like this is because its the frist api request recived by that worker | |
| 11:06:28 | sean-k-mooney | 16 seconds is not really that un resonable given the load that the gate vms are under. we can see if there is a way to speed it up but i dont think this would be an issue in production | |
| 11:09:04 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Get rid of args to RBDDriver.__init__() https://review.opendev.org/668564 | |
| 11:09:37 | slaweq | sean-k-mooney: the problem is that this is an issue in the gate :/ | |
| 11:10:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Move consts from neutronv2/api to constants module https://review.opendev.org/668945 | |
| 11:10:21 | sean-k-mooney | slaweq: yep which we may not be able to fix in nova. im goint to look at the other logs and then ill look at the code | |
| 11:10:51 | sean-k-mooney | but its like saying the ovs neutorn agenet does not wire up new port imediatly after teh agent is restart | |
| 11:10:52 | slaweq | sean-k-mooney: thx | |
| 11:11:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Translatable output strings in heal allocation https://review.opendev.org/668925 | |
| 11:11:56 | sean-k-mooney | we know it take a whlie for it to recompute everything because it has a lot of work to do on start up. maybe we tweak something or have devstack ping the metadta api | |
| 11:12:23 | sean-k-mooney | e.g. before tempest runs to make sure this setup is done before a test hits it | |
| 11:12:45 | slaweq | sean-k-mooney: or other "gate only workaround" could be to patch cirros image and retry those calls if it fails first time | |
| 11:14:16 | cdent | "gate only workaround"-- | |
| 11:19:26 | sean-k-mooney | looking at the logs more closely the reponces that are 2ms are all cached | |
| 11:21:05 | sean-k-mooney | uncached responces for new intanstances are between 300-1000ms generally with the 16000 ms and similar long resonces also doing addtional work | |
| 11:23:05 | sean-k-mooney | by addtional work i mean it has to aquire a lock and call nova.context.get_or_set_cached_cell_and_set_connections before it then generates the metadata | |
| 11:23:21 | sean-k-mooney | or in the case of the first request also set up the db connection | |
| 11:25:30 | sean-k-mooney | ok so when its not cached it has to call this | |
| 11:25:32 | sean-k-mooney | https://github.com/openstack/nova/blob/af4cd78e8537d8f0933e30598066cedac5844866/nova/context.py#L355-L375 | |
| 11:26:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use neutron contants in cmd/manage.py https://review.opendev.org/668946 | |
| 11:26:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add 'resource_request' to neutronv2/constants https://review.opendev.org/668947 | |
| 11:26:22 | sean-k-mooney | which set up both the db connection and rabbitmq connection if its not already cached | |
| 11:35:46 | sean-k-mooney | ok so that is invoked when teh context target_cell context manager is used to lookup the instace by its uuid here https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L680 | |
| 11:35:51 | slaweq | sean-k-mooney: thx for looking into this | |
| 11:37:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use the safe get_binding_profile https://review.opendev.org/669817 | |
| 11:49:48 | gibi | efried: if you have time, could you look back to https://review.opendev.org/#/c/666857 ? | |
| 11:55:34 | sean-k-mooney | slaweq: im not 100% certin but the metadata api request that took 16 seconds may have been waiting on neutron for 12 second to respond with the security groups for the instance http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/controller/logs/screen-q-svc.txt.gz#_Jul_11_23_43_16_660751 | |
| 12:01:29 | sean-k-mooney | slaweq: actully the device id of the port matches the instance id of the metadata request so yes its was waiting for neuton to return the security group info | |