Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-11
20:09:58 ildikov and then we need to check whether attachment_complete happens there or not
20:10:44 mriedem yes my patch does'nt fix it
20:10:51 mriedem i think i know what needs to happen
20:10:54 mriedem but making coffee first
20:13:00 ildikov fair enough
20:31:08 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
20:31:09 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
20:38:55 _ix More high praise if you're interested. I think that newish Pike feature in disabling nodes saved me about half a day of troubleshooting.
20:39:06 _ix Thank you!
20:41:04 mriedem thanks for the feedback
20:41:15 mriedem always nice to know when people actually "like" something
20:42:43 cdent 👍
20:43:42 mriedem dtantsur|afk: do you have a run on https://review.openstack.org/#/c/508555/ yet?
20:44:06 mriedem TheJulia: do you know where ^ is being tested to see if it unblocks the ironic multinode job?
20:44:50 mriedem dtantsur|afk: TheJulia: nevermind, found it https://review.openstack.org/#/c/527075/
20:45:11 TheJulia :)
20:45:29 mriedem is ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode the job to care about?
20:51:48 TheJulia mriedem: yes
20:56:54 mriedem dansmith: comments in john's rebalance patch - https://review.openstack.org/#/c/508555/ - i think it looks better, but we could maybe drop the new object query method too - i think there is an existing one that could be re-used in there
21:00:15 dansmith mriedem: yep, looks right
21:06:41 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Update and complete volume attachments during resize https://review.openstack.org/527228
21:06:55 mriedem smatzek: ildikov: ^ should do it for resize
21:08:06 mriedem smatzek: the rebuild fix will be different, so do you have a separate bug for that?
21:08:09 mriedem with links to the logs
21:08:41 smatzek I don't, but I'll open one.
21:09:28 mriedem dansmith: ok, ironic multinode job passed and given dmitry and john are gone, i could update quick and get another run, or just wait for john to do it - but i think he's probably on paternity
21:09:56 mriedem maybe it's fine to wait another day assuming this has been broken since pike
21:10:13 dansmith yeah I think he is.. up to you, I'm around to +2 if you want, or I can do it
21:10:16 mriedem TheJulia: is this a latent bug or regression in pike?
21:10:18 dansmith if you update I think jaypipes and I can get it
21:10:23 mriedem dansmith: ok i'll update quick
21:12:30 TheJulia mriedem: don't have enough context to say either way
21:13:15 ildikov mriedem: so resize is using a different flow and not calling bdm.attach(), right?
21:13:26 mriedem ildikov: correct
21:14:02 mriedem doing it the way i'm doing it is a lot easier than trying to re-plumb how things work for connecting the volume during a resize
21:14:24 mriedem since in the old flow, we never needed to call bdm.attach because the volume was already attached
21:14:41 mriedem the new flow is all about the attachment though, so we have to do things differently
21:15:20 mriedem we could call bdm.attach and pass do_driver_attach=False, but then we have to do that before calling driver.finish_migration and it seems wrong to 'complete' the attachment before we've actually connected it on the host
21:16:39 smatzek mriedem: I can open a bug for the rebuild one if you want. If you know there are changes needed there and want a bug to work them. However, I'm not convinced there is a bug in rebuild itself after digging through the code this afternoon.
21:20:15 smatzek Trove does this with cinderclient before calling nova rebuild, and at this point the device/mountpoint is /dev/na, which means it was lost previous to rebuild. https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L1369-L1371
21:20:53 smatzek this probably needs more digging on my part to see what was done in the test suite to that instance before getting to this point.
21:21:24 ildikov mriedem: ok, that makes sense, I don't have a Devstack running at the moment, will set up one and check these flows as I always have to re-understand the whole thing if I just try to follow this code by reading...
21:22:08 mriedem smatzek: ok i'll leave rebuild up to you
21:22:28 smatzek mriedem: I need to take off to get the kids from school and run a few errands so I'll dig into it later.
21:22:37 mriedem ildikov: we definitely have a todo to add a test to tempest for resize with a volume attached (non-bfv)
21:24:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Re-use existing ComputeNode on ironic rebalance https://review.openstack.org/508555
21:24:08 mriedem dansmith: ^ i'll recheck the ironic canary patch
21:24:11 openstackgerrit Merged openstack/nova master: Address nits from service create/destroy notification review https://review.openstack.org/523162
21:24:16 openstackgerrit Merged openstack/nova master: [placement] Fix API reference for microversion 1.14 https://review.openstack.org/526973
21:24:22 openstackgerrit Merged openstack/nova stable/pike: Mention API behavior change when over quota limit https://review.openstack.org/525511
21:24:34 openstackgerrit Merged openstack/nova stable/pike: Downgrade log for keystone verify client fail https://review.openstack.org/525475
21:25:26 mriedem all he wanted was a pepsi https://www.youtube.com/watch?v=aYItTxqTc38
21:28:05 ildikov just another regular Monday :)
21:28:12 _ix Quick question ... can I assume that that ports should be bound to the host that... hosts the instance?
21:28:26 openstackgerrit Merged openstack/nova stable/pike: Fix ValueError if invalid max_rows passed to db purge https://review.openstack.org/525628
21:28:29 mriedem _ix: unless it's migrating
21:29:13 _ix mriedem: Where should I be looking for problems if this isn't necessarily the case?
21:29:29 _ix That is, I just instantiated, and they seem to be bound incorrectly.
21:29:36 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
21:29:37 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
21:30:00 dansmith mriedem: cool
21:30:17 mriedem _ix: nova-compute logs
21:30:18 mriedem ?
21:30:41 mriedem _ix: maybe check to see if the instance was rescheduled during the create to another host and the ports weren't cleaned up properly
21:30:58 _ix I've been seeing a bit of that, so that might make sense.
21:31:20 mriedem _ix: maybe check https://review.openstack.org/#/c/520248/
21:31:22 _ix No ports appear to be working on this host that it's supposed to be living on, so, there are a litany of issues. Thanks.
21:31:49 mriedem _ix: that likely means we hit that compute, created a port that couldn't be bound, and we rescheduled to another compute host
21:32:08 _ix !
21:32:09 mriedem if the instance built on the next host, it would be using the 2nd port but we orphaned the 1st port
21:32:11 _ix I just saw that error today.
21:32:28 _ix Cool.
21:39:17 mriedem i've removed zkvm from nova-ci for now
21:57:09 mriedem never seen a live migration failure like this before http://logs.openstack.org/07/516707/23/check/legacy-tempest-dsvm-multinode-live-migration/d9089b2/logs/libvirt/qemu/instance-0000000a.txt.gz
22:03:17 mriedem edleafe: so for prep_resize and unshelve_instance, https://review.openstack.org/#/c/516707/23/nova/compute/rpcapi.py - the filter_properties dict contains a limits key, is that getting converted to a dict via scheduler utils?
22:05:05 openstackgerrit Merged openstack/nova stable/pike: Vzstorage: synchronize volume connect https://review.openstack.org/524688
22:09:13 mriedem edleafe: ok so i think the only remaining thing is there are some unnecessary conversions for the limits object in conductor manager in https://review.openstack.org/#/c/516707/ still - do you want to remove those or should i just do that quick?
22:12:17 edleafe mriedem: I can do it
22:12:46 edleafe mriedem: re: the filter properties dict key, I'm tracing the logic backwards
22:15:52 edleafe mriedem: ok, yeah, I don't see any paths that the filter_properties could contain a SchedulerLimits object
22:16:19 mriedem yeah i think it's handled in the 4 cases that matter
22:16:27 mriedem it's just different in two of them which is why i had to dig
22:33:17 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
22:33:18 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
22:33:18 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
22:33:19 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
22:33:26 edleafe mriedem: addressed your comments ^^
22:33:29 mriedem on it
22:41:17 openstackgerrit Merged openstack/nova stable/pike: Add regression test for rebuilding a volume-backed server https://review.openstack.org/525632
22:42:44 mriedem edleafe: +2 on that bottom patch
22:42:46 mriedem thanks
22:44:39 mriedem and the one after it
22:44:55 mriedem jaypipes: dansmith: https://review.openstack.org/#/c/516707/ and the one after it have my +2
22:46:21 edleafe mriedem: kewl
22:46:40 mriedem +2 on cdent's cache headers patch too https://review.openstack.org/#/c/521640/
22:48:50 jaypipes mriedem: cool.
22:53:29 ildikov mriedem: BTW as for resize, do we need to do the same magic with finish_revert_resize too?
22:55:21 mriedem ildikov: maybe
22:55:46 mriedem likely need a test for confirm and revert a resize with an instance that has a volume attached
22:56:15 ildikov mriedem: as we're calling _terminate_volume_connections in revert_resize too which basically that same story just backwards

Earlier   Later