| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-15 | |||
| 02:54:20 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522 | |
| 05:57:31 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Microversion 2.80: Add user_id/project_id to migration-list API https://review.opendev.org/675023 | |
| 06:58:52 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635 | |
| 07:35:55 | gibi | good morning nova | |
| 07:41:50 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635 | |
| 07:48:59 | gibi | stephenfin: just to be sure you also got the message I got a mail from the summit organizers that the project updates will not be recoreded. Instead they would like to get an etherpad with main points. | |
| 08:02:17 | openstack | Launchpad bug 1648501 in OpenStack Compute (nova) "providing different imageRef when using block_device_mapping (image -> volume) " [Low,Confirmed] | |
| 08:02:17 | jkulik | Hi, regarding https://bugs.launchpad.net/nova/+bug/1648501 I see a problem with code assuming instance.image_ref being None equals boot-from-volume as mentioned here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3371-L3372 | |
| 08:03:46 | jkulik | Because even if imageRef is set to the same values as in the block-device-mapping, instance.image_ref will then be set even though it's boot-from-volume. | |
| 08:05:02 | jkulik | As the bug mentions, the cli already forbids providing imageRef and block-device-mapping at the same time. But the API accepts it. Shouldn't this be made consistent? | |
| 09:54:56 | SonPham | Hi. I'm working with nova and horizon. I traced code form Horizon on Button start-instance: Horizon call NovaClient: novaclient(request).servers.start(instance_id) | |
| 09:55:26 | SonPham | def start(self, server): | |
| 09:55:26 | SonPham | and I check python client server start is: | |
| 09:55:53 | SonPham | and i think it call to /nova/nova/copmute/apy.py / def start() | |
| 09:56:03 | SonPham | how it work? | |
| 10:05:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: zuul: Make functional job inherit from openstack parents https://review.opendev.org/688425 | |
| 10:05:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Stop testing Python 2 https://review.opendev.org/687954 | |
| 10:05:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unused 'nova-dsvm-base' job https://review.opendev.org/688389 | |
| 10:05:59 | gibi | SonPham: the python client calls the nova-api service via HTTP (REST). The server create request is handed by https://github.com/openstack/nova/blob/63fb66e39a2590f00541f36d94e31372c2fe82ee/nova/api/openstack/compute/servers.py#L598 | |
| 10:09:16 | gibi | jkulik: I agree it feels strange that you can provide an imageRef POST /servers but at that same time you are server is not booted from that image but booted from a volume with different content | |
| 10:09:46 | gibi | jkulik: so I think the nova-api can reject such situation | |
| 10:16:18 | jkulik | for the vmware driver in queens at least, we get a volume and an ephemeral disk, because it seems to only take into account instance.image_ref when creating the ephemeral one | |
| 10:16:45 | jkulik | (still boots from the volume, though) | |
| 10:17:52 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Implementing graceful shutdown. https://review.opendev.org/666245 | |
| 10:18:56 | SonPham | gibi server.py call to nova.api and this code call to compute/api.py/def start() ?? : self.compute_api.start(context, instance) | |
| 11:41:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387 | |
| 11:53:57 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688 | |
| 12:07:17 | gibi | dansmith: when you are up. the runway link in the channel topic need some low and as far as I remember you can have op in this channel to fix it | |
| 12:07:26 | gibi | s/low/love/ | |
| 12:09:40 | efried | brinzhang_: you around? | |
| 12:43:23 | slaweq | efried: gibi: hi | |
| 12:43:33 | slaweq | did You saw failure like https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a28/681671/10/check/neutron-tempest-dvr/a281c2b/testr_results.html.gz before? | |
| 12:43:34 | gibi | slaweq: hi! | |
| 12:43:46 | slaweq | I saw it at least 3-4 times during last week | |
| 12:44:21 | slaweq | maybe it's already known issue than I will not report new bug | |
| 12:44:49 | gibi | slaweq: I haven't seen it before. But it seems like a race condition between the build and the delete | |
| 12:45:16 | slaweq | gibi: ok, I will report new bug than | |
| 12:45:23 | slaweq | it's more for placement, right? | |
| 12:45:47 | gibi | slaweq: thanks. it is for nova. As nova calls placement with a wrong generation | |
| 12:45:54 | efried | was melwitt working on that? | |
| 12:46:05 | stephenfin | gibi: I saw that, yeah. If we've slides, that ought to do the trick | |
| 12:46:22 | slaweq | gibi: ok, thx | |
| 12:47:46 | gibi | efried: I searched gerrit now but I haven't found a patch | |
| 12:47:54 | gibi | stephenfin: OK, cool | |
| 12:47:56 | efried | yeah, me neither, hm... | |
| 12:48:55 | efried | I was sure there was a bug at least... | |
| 12:50:35 | openstack | Launchpad bug 1836754 in OpenStack Compute (nova) "Conflict when deleting allocations for an instance that hasn't finished building" [Medium,Confirmed] | |
| 12:50:35 | efried | gibi, slaweq: does this look right? https://bugs.launchpad.net/nova/+bug/1836754 | |
| 12:50:50 | slaweq | efried: yes, I just found it now :) | |
| 12:51:47 | slaweq | efried: thx for looking for it | |
| 12:51:51 | gibi | efried: yeah, down in the comments there is another failure when the compute host is deleted | |
| 12:53:06 | efried | gibi: this is another case where it would be useful to be able to distinguish between consumer and provider conflicts | |
| 12:53:31 | efried | We don't care if we get a provider conflict here, but if we get a consumer conflict that's bad. | |
| 12:53:47 | gibi | efried: noted. I still have a TODO to make progress on that | |
| 12:54:29 | efried | we could solve this with a retry, but without making that ^ distinction, I'm not sure it's the right thing to do. | |
| 12:54:47 | efried | ...or that it's better than just using an old (generation-less) microversion to drop the allocs. | |
| 12:55:03 | efried | ...or using the DELETE route which IIRC doesn't do generations at all. | |
| 12:55:13 | efried | (because no payload). | |
| 12:55:28 | efried | (and we don't put that stuff in headers) | |
| 12:57:22 | gibi | I think we need to retry on consumer conflict as well. This bug basically means that the server delete codepatch racing with the server create codepath. The end user want's the server to be deleted so even if the create codepath updated the server allocation we need to delete the updated allocation | |
| 12:58:06 | gibi | so if we want to delete that allocation in every case, then we can even call DELETE without the generation checking | |
| 12:58:31 | efried | If we can be sure we're in the instance delete flow, I agree with you. | |
| 12:59:14 | efried | Meaning we can't just go hard at the report.py level; we have to {call a different method | send a specific flag} indicating we want to force it. | |
| 12:59:56 | efried | because in the general case if we're deleting allocations and something changes the consumer, it doesn't necessarily mean we want to proceed. | |
| 13:00:06 | gibi | efried: you are right | |
| 13:00:26 | efried | Though I'm not sure how we could race e.g. a resize and a migrate | |
| 13:00:29 | gibi | efried: we need to be careful and only force the delete from the server delete codepath | |
| 13:01:23 | efried | btw I checked the placement logs in slaweq's repro and they don't include the message for the 409, so I can't tell whether it was indeed a provider conflict. | |
| 13:08:53 | gibi | efried: in the placement log here is the conflict Oct 14 14:45:26.823202 | |
| 13:09:36 | gibi | efried: around that there is multiple PUT request for consumer 5b5b12dc | |
| 13:10:26 | gibi | efried: nova-api doing a local delete for server 5b5b12dc | |
| 13:12:38 | gibi | efried: so at least slaweq's repor is a race between a server create and a server delete | |
| 13:14:37 | efried | mm. So we could fix this with a hard delete -- but what actually worries me is the reverse problem. | |
| 13:15:01 | efried | what if the delete happens first, and then the create comes in? We would have leaked allocations. | |
| 13:17:29 | efried | we should have some way for the delete to abort the create... | |
| 13:17:53 | efried | But I guess that's another problem for another day. | |
| 13:28:39 | mriedem | https://zuul.opendev.org/t/openstack/build/4da3c44dcbcd4ed7aa04a8dcaa19c011/log/logs/grenade.sh.txt.gz#35366 | |
| 13:28:39 | mriedem | stephenfin: since you care about the py2 droppage, grenade jobs are failing on some weird package things | |
| 13:28:51 | mriedem | that's with py2 on the old (train) side and py3 on the new (ussuri) side | |
| 13:29:04 | mriedem | ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. | |
| 13:29:35 | mriedem | gibi: efried: i'm pretty sure melwitt brought up that bug the other day (the local delete conflict), | |
| 13:29:42 | mriedem | did someone report a bug for it? | |
| 13:30:19 | openstack | Launchpad bug 1836754 in OpenStack Compute (nova) "Conflict when deleting allocations for an instance that hasn't finished building" [Medium,Confirmed] | |
| 13:30:19 | mriedem | looks like artom already opened one for the same thing https://bugs.launchpad.net/nova/+bug/1836754 | |
| 13:30:25 | gibi | mriedem: yes, that one | |
| 13:31:53 | mriedem | last time i looked we just need a retry in the local delete case | |
| 13:32:10 | gibi | mriedem: yeah, that was my understanding above as well | |
| 13:33:02 | mriedem | "what if the delete happens first, and then the create comes in? We would have leaked allocations." | |
| 13:33:04 | mriedem | yeah that is a problem | |
| 13:33:12 | mriedem | since the create will just re-create the consumer and allocations in placement | |
| 13:33:27 | mriedem | hear me out, SOFT DELETE-ABLE CONSUMERS! | |
| 13:35:06 | mriedem | efried: i'd think we could trap that case in conductor once the response comes from the scheduler, | |
| 13:35:42 | mriedem | b/c we'll check if the build request has been deleted in the interim during scheduling and if so halt the build process - at that point we could cleanup allocations (do we not already?) | |
| 13:38:56 | dansmith | gibi: got it | |
| 13:39:07 | gibi | dansmith: thanks! | |
| 13:41:25 | efried | mriedem: we might already, yeah. | |
| 13:42:55 | mriedem | gibi: efried: i might work up a functional test to recreate it since unit tests aren't going to cut it for that kind of interaction | |
| 13:43:32 | efried | cool | |
| 13:44:08 | efried | brinzhang: yt? | |
| 13:44:13 | gibi | mriedem: cool, sorry I'm busy today - tomorrow so I did not bite | |