Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-14
23:54:54 openstackgerrit sean mooney proposed openstack/nova master: block rebuild when numa toplogy changed https://review.opendev.org/687957
#openstack-nova - 2019-10-15
01:04:51 openstackgerrit Arthur Dayne proposed openstack/nova master: libvirt:volume:Disallow AIO=native when no 'O_DIRECT' is available https://review.opendev.org/682772
02:03:16 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add minor version [21] to test_versions https://review.opendev.org/688599
02:05:01 openstackgerrit Huachang Wang proposed openstack/nova master: cleanup to objects.fields https://review.opendev.org/688600
02:22:56 openstackgerrit Huachang Wang proposed openstack/nova master: Set instance CPU policy to 'share' when 'hw_cpu_policy==share' https://review.opendev.org/688603
02:54:19 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341
02:54:19 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029
02:54:20 openstackgerrit Dustin Cowles proposed openstack/nova master: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522
05:57:31 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Microversion 2.80: Add user_id/project_id to migration-list API https://review.opendev.org/675023
06:58:52 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635
07:35:55 gibi good morning nova
07:41:50 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635
07:48:59 gibi stephenfin: just to be sure you also got the message I got a mail from the summit organizers that the project updates will not be recoreded. Instead they would like to get an etherpad with main points.
08:02:17 jkulik Hi, regarding https://bugs.launchpad.net/nova/+bug/1648501 I see a problem with code assuming instance.image_ref being None equals boot-from-volume as mentioned here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3371-L3372
08:02:17 openstack Launchpad bug 1648501 in OpenStack Compute (nova) "providing different imageRef when using block_device_mapping (image -> volume) " [Low,Confirmed]
08:03:46 jkulik Because even if imageRef is set to the same values as in the block-device-mapping, instance.image_ref will then be set even though it's boot-from-volume.
08:05:02 jkulik As the bug mentions, the cli already forbids providing imageRef and block-device-mapping at the same time. But the API accepts it. Shouldn't this be made consistent?
09:54:56 SonPham Hi. I'm working with nova and horizon. I traced code form Horizon on Button start-instance: Horizon call NovaClient: novaclient(request).servers.start(instance_id)
09:55:26 SonPham and I check python client server start is:
09:55:26 SonPham def start(self, server):
09:55:53 SonPham and i think it call to /nova/nova/copmute/apy.py / def start()
09:56:03 SonPham how it work?
10:05:15 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove unused 'nova-dsvm-base' job https://review.opendev.org/688389
10:05:15 openstackgerrit Stephen Finucane proposed openstack/nova master: Stop testing Python 2 https://review.opendev.org/687954
10:05:15 openstackgerrit Stephen Finucane proposed openstack/nova master: zuul: Make functional job inherit from openstack parents https://review.opendev.org/688425
10:05:59 gibi SonPham: the python client calls the nova-api service via HTTP (REST). The server create request is handed by https://github.com/openstack/nova/blob/63fb66e39a2590f00541f36d94e31372c2fe82ee/nova/api/openstack/compute/servers.py#L598
10:09:16 gibi jkulik: I agree it feels strange that you can provide an imageRef POST /servers but at that same time you are server is not booted from that image but booted from a volume with different content
10:09:46 gibi jkulik: so I think the nova-api can reject such situation
10:16:18 jkulik for the vmware driver in queens at least, we get a volume and an ephemeral disk, because it seems to only take into account instance.image_ref when creating the ephemeral one
10:16:45 jkulik (still boots from the volume, though)
10:17:52 openstackgerrit Martin Midolesov proposed openstack/nova master: Implementing graceful shutdown. https://review.opendev.org/666245
10:18:56 SonPham gibi server.py call to nova.api and this code call to compute/api.py/def start() ?? : self.compute_api.start(context, instance)
11:41:52 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387
11:53:57 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
12:07:17 gibi dansmith: when you are up. the runway link in the channel topic need some low and as far as I remember you can have op in this channel to fix it
12:07:26 gibi s/low/love/
12:09:40 efried brinzhang_: you around?
12:43:23 slaweq efried: gibi: hi
12:43:33 slaweq did You saw failure like https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a28/681671/10/check/neutron-tempest-dvr/a281c2b/testr_results.html.gz before?
12:43:34 gibi slaweq: hi!
12:43:46 slaweq I saw it at least 3-4 times during last week
12:44:21 slaweq maybe it's already known issue than I will not report new bug
12:44:49 gibi slaweq: I haven't seen it before. But it seems like a race condition between the build and the delete
12:45:16 slaweq gibi: ok, I will report new bug than
12:45:23 slaweq it's more for placement, right?
12:45:47 gibi slaweq: thanks. it is for nova. As nova calls placement with a wrong generation
12:45:54 efried was melwitt working on that?
12:46:05 stephenfin gibi: I saw that, yeah. If we've slides, that ought to do the trick
12:46:22 slaweq gibi: ok, thx
12:47:46 gibi efried: I searched gerrit now but I haven't found a patch
12:47:54 gibi stephenfin: OK, cool
12:47:56 efried yeah, me neither, hm...
12:48:55 efried I was sure there was a bug at least...
12:50:35 efried gibi, slaweq: does this look right? https://bugs.launchpad.net/nova/+bug/1836754
12:50:35 openstack Launchpad bug 1836754 in OpenStack Compute (nova) "Conflict when deleting allocations for an instance that hasn't finished building" [Medium,Confirmed]
12:50:50 slaweq efried: yes, I just found it now :)
12:51:47 slaweq efried: thx for looking for it
12:51:51 gibi efried: yeah, down in the comments there is another failure when the compute host is deleted
12:53:06 efried gibi: this is another case where it would be useful to be able to distinguish between consumer and provider conflicts
12:53:31 efried We don't care if we get a provider conflict here, but if we get a consumer conflict that's bad.
12:53:47 gibi efried: noted. I still have a TODO to make progress on that
12:54:29 efried we could solve this with a retry, but without making that ^ distinction, I'm not sure it's the right thing to do.
12:54:47 efried ...or that it's better than just using an old (generation-less) microversion to drop the allocs.
12:55:03 efried ...or using the DELETE route which IIRC doesn't do generations at all.
12:55:13 efried (because no payload).
12:55:28 efried (and we don't put that stuff in headers)
12:57:22 gibi I think we need to retry on consumer conflict as well. This bug basically means that the server delete codepatch racing with the server create codepath. The end user want's the server to be deleted so even if the create codepath updated the server allocation we need to delete the updated allocation
12:58:06 gibi so if we want to delete that allocation in every case, then we can even call DELETE without the generation checking
12:58:31 efried If we can be sure we're in the instance delete flow, I agree with you.
12:59:14 efried Meaning we can't just go hard at the report.py level; we have to {call a different method | send a specific flag} indicating we want to force it.
12:59:56 efried because in the general case if we're deleting allocations and something changes the consumer, it doesn't necessarily mean we want to proceed.
13:00:06 gibi efried: you are right
13:00:26 efried Though I'm not sure how we could race e.g. a resize and a migrate
13:00:29 gibi efried: we need to be careful and only force the delete from the server delete codepath
13:01:23 efried btw I checked the placement logs in slaweq's repro and they don't include the message for the 409, so I can't tell whether it was indeed a provider conflict.
13:08:53 gibi efried: in the placement log here is the conflict Oct 14 14:45:26.823202
13:09:36 gibi efried: around that there is multiple PUT request for consumer 5b5b12dc
13:10:26 gibi efried: nova-api doing a local delete for server 5b5b12dc
13:12:38 gibi efried: so at least slaweq's repor is a race between a server create and a server delete
13:14:37 efried mm. So we could fix this with a hard delete -- but what actually worries me is the reverse problem.
13:15:01 efried what if the delete happens first, and then the create comes in? We would have leaked allocations.
13:17:29 efried we should have some way for the delete to abort the create...
13:17:53 efried But I guess that's another problem for another day.
13:28:39 mriedem stephenfin: since you care about the py2 droppage, grenade jobs are failing on some weird package things
13:28:39 mriedem https://zuul.opendev.org/t/openstack/build/4da3c44dcbcd4ed7aa04a8dcaa19c011/log/logs/grenade.sh.txt.gz#35366
13:28:51 mriedem that's with py2 on the old (train) side and py3 on the new (ussuri) side
13:29:04 mriedem ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
13:29:35 mriedem gibi: efried: i'm pretty sure melwitt brought up that bug the other day (the local delete conflict),
13:29:42 mriedem did someone report a bug for it?
13:30:19 mriedem looks like artom already opened one for the same thing https://bugs.launchpad.net/nova/+bug/1836754
13:30:19 openstack Launchpad bug 1836754 in OpenStack Compute (nova) "Conflict when deleting allocations for an instance that hasn't finished building" [Medium,Confirmed]
13:30:25 gibi mriedem: yes, that one
13:31:53 mriedem last time i looked we just need a retry in the local delete case
13:32:10 gibi mriedem: yeah, that was my understanding above as well
13:33:02 mriedem "what if the delete happens first, and then the create comes in? We would have leaked allocations."
13:33:04 mriedem yeah that is a problem
13:33:12 mriedem since the create will just re-create the consumer and allocations in placement
13:33:27 mriedem hear me out, SOFT DELETE-ABLE CONSUMERS!
13:35:06 mriedem efried: i'd think we could trap that case in conductor once the response comes from the scheduler,
13:35:42 mriedem b/c we'll check if the build request has been deleted in the interim during scheduling and if so halt the build process - at that point we could cleanup allocations (do we not already?)

Earlier   Later