| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 13:56:30 | mriedem | unshelve should be handled here https://review.openstack.org/#/c/583715/3/nova/conductor/manager.py@806 | |
| 13:57:16 | mriedem | i could update the functional test in that change to run a shelve/unshelve routine to make sure DISK_GB still doesn't show up in placement allocations | |
| 13:58:07 | pooja_jadhav | mriedem: cold/live migrate (both trying to test) | |
| 13:58:27 | mriedem | damn i have to rebase this series because https://review.openstack.org/#/c/583646/ was rebased | |
| 13:58:35 | mriedem | pooja_jadhav: https://docs.openstack.org/devstack/latest/guides/multinode-lab.html | |
| 13:58:38 | mriedem | if you're using devstack | |
| 13:59:52 | pooja_jadhav | mriedem: thanks, will check and let u know if the queries :) | |
| 14:00:04 | pooja_jadhav | yes, I am using devstack | |
| 14:00:26 | mriedem | that doc might be old | |
| 14:04:59 | pooja_jadhav | mriedem: Also tested resize and evacuate, but no allocation records get added for DISK_GB. | |
| 14:06:03 | pooja_jadhav | mriedem: But for evacuate, hypervisor-stats showing incorrect incorrect usages. | |
| 14:06:15 | openstackgerrit | Boxiang Zhu proposed openstack/nova stable/pike: Fix "instance snap min disk size err after resize instance" https://review.openstack.org/584770 | |
| 14:08:52 | mriedem | pooja_jadhav: we already know that hypervisor-stats has latent bugs for reporting disk based on the flavor and not if the server is volume-backed | |
| 14:09:08 | mriedem | that's what https://review.openstack.org/#/c/583646/ shows | |
| 14:09:15 | pooja_jadhav | mriedem: Ohk | |
| 14:10:26 | pooja_jadhav | mriedem: Have tested live/cold migrate then? for allocation of DISK_GB or not? | |
| 14:11:13 | mriedem | pooja_jadhav: https://review.openstack.org/#/c/583715/ functionally tests that for cold migration | |
| 14:11:26 | mriedem | https://review.openstack.org/#/c/583715/3/nova/tests/functional/test_servers.py | |
| 14:12:00 | mriedem | i was saying i could add a shelve/unshelve wrinkle to that test as well since you said you saw allocations for DISK_GB in placement after unshelve | |
| 14:13:07 | mriedem | but i need to rebase this series now | |
| 14:14:35 | pooja_jadhav | mriedem: thanks :) | |
| 14:24:13 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 14:56:35 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove usage of migrateToURI{2} APIs https://review.openstack.org/567258 | |
| 14:59:26 | mriedem | efried: hmm, looks like the retrying package might not be thread safe http://logs.openstack.org/03/583203/2/check/tempest-full-py3/0b5111a/controller/logs/screen-n-cond-cell1.txt.gz#_Jul_23_13_35_01_491350 | |
| 14:59:47 | mriedem | got a thread switch there which looks like it caused a failure during RT._update | |
| 15:00:05 | mriedem | well, maybe not | |
| 15:00:16 | mriedem | could just be throwing me b/c it's in the trace b/c of the decorator | |
| 15:00:57 | mriedem | hit a messaging timeout, maybe just a slow node | |
| 15:01:22 | mriedem | yeah | |
| 15:01:22 | efried | mriedem: afaik, retrying doesn't use threads at all. Are you saying it has a global or something? | |
| 15:01:22 | mriedem | Jul 23 13:35:01.252889 ubuntu-xenial-packethost-us-west-1-0000896864 nova-compute[22210]: WARNING nova.servicegroup.drivers.db [None req-93269558-9418-452a-b391-8766a59984a5 tempest-TestVolumeSwap-747104316 tempest-TestVolumeSwap-747104316] Lost connection to nova-conductor for reporting service status.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 06f920f1d628445b877ff4100980047e | |
| 15:01:32 | mriedem | efried: no i think it just threw me in the stacktrace | |
| 15:01:36 | efried | ight | |
| 15:02:54 | sean-k-mooney | QQ are the functional tests safe to run with devstack stacked? | |
| 15:03:12 | mriedem | nova tox -e functional? | |
| 15:03:14 | mriedem | they run in a venv so yes | |
| 15:03:40 | sean-k-mooney | ok cool was just wondering if i would get any failures due to haveing opentack running | |
| 15:15:30 | stephenfin | mriedem: RE: numa-aware-vswitches and move operations. Rebuilds don't run through the scheduler and don't make claims so I ignore that kind of rebuild operation, right? | |
| 15:15:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Report 0 root_gb in resource tracker if instance is bfv. https://review.openstack.org/584204 | |
| 15:15:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715 | |
| 15:15:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739 | |
| 15:15:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add shelve/unshelve wrinkle to volume-backed disk func test https://review.openstack.org/584931 | |
| 15:15:48 | mriedem | pooja_jadhav: ^ added a shelve/unshelve functional test to the end of the series and that shows things working as designed | |
| 15:16:10 | sean-k-mooney | stephenfin: rebuilds should still go through the scheuler | |
| 15:16:14 | mriedem | stephenfin: depends, what does the RUN_ON_REBUILD flag say for NUMATopologyFilter? | |
| 15:16:21 | mriedem | sean-k-mooney: not in all cases, and not for all filters | |
| 15:16:32 | mriedem | stephenfin: and to be clear, you're talking about rebuild, not evacuate, yes? | |
| 15:16:46 | mriedem | stephenfin: class NUMATopologyFilter(filters.BaseHostFilter): | |
| 15:16:46 | mriedem | """Filter on requested NUMA topology.""" | |
| 15:16:46 | mriedem | RUN_ON_REBUILD = True | |
| 15:17:03 | stephenfin | mriedem: It's True and yes | |
| 15:17:18 | mriedem | stephenfin: so then you need to put the stuff in the request spec if we're going through the scheduler during rebuild b/c the image changes | |
| 15:17:29 | mriedem | because image_props.get('hw_cpu_policy')] and image_props.get('hw_cpu_thread_policy')] | |
| 15:17:34 | mriedem | if the image changes, the numa stuff can change | |
| 15:17:51 | mriedem | stephenfin: rebuild isn't a move, but we have to validate the new image | |
| 15:18:17 | mriedem | *validate the new image still works on the current host | |
| 15:19:12 | sean-k-mooney | mriedem: a rebuilt instance could land on a different host then the instace was on previously though right. its effectivly a new boot with the same port/volumes as the original instance | |
| 15:19:34 | mriedem | sean-k-mooney: no | |
| 15:19:44 | mriedem | sean-k-mooney: are you talking about evacuate? | |
| 15:20:16 | sean-k-mooney | no i taught if we rebuilt and the previous host was nolonger valid we would schdule to another host that was | |
| 15:20:23 | sean-k-mooney | i gues we just fail to rebuild | |
| 15:21:19 | mriedem | you get a novalidhost | |
| 15:21:27 | mriedem | if the rebuild + new image doesn't fit on the current host | |
| 15:21:41 | mriedem | instance goes to error state and you have to fix it | |
| 15:22:19 | mriedem | bauzas: ping https://review.openstack.org/#/c/583715/ | |
| 15:23:10 | mriedem | bauzas: also ping https://review.openstack.org/#/c/581813/ - that's the nova-status check for request spec migrations which means if we get that in rocky, we can drop the reqspec migrations and compat code in stein | |
| 15:23:15 | mriedem | which is something i desperatly want to do | |
| 15:23:21 | sean-k-mooney | mriedem: right i was assuming we would hit the retry fileter and try another host but i guess not. in my mind i was assuming we did the host affinity for rebuil via a weigher rather then filtering to just the current host | |
| 15:23:21 | mriedem | that and spellcheck | |
| 15:23:43 | mriedem | sean-k-mooney: we pass a hint to the scheduler telling it to filter on the existing instance host | |
| 15:24:42 | sean-k-mooney | mriedem: cool ill look into how that work more closely in the future. good to know | |
| 15:26:06 | mriedem | stephenfin: so i assume you might have 2 patches: 1 to add the physnet and tunneled plumbing to the nova.network.model.Network object and populate it during _build_network_info_model, and then another change to call some utils method in the various conductor spots to populate ReuqestSpec.network_metadata using instance.info_cache.network_info | |
| 15:26:34 | stephenfin | mriedem: That's exactly what I have, albeit rolled into one patch | |
| 15:26:38 | mriedem | stephenfin: btw, this change is doing the same call spots in conductor that you'll need https://review.openstack.org/#/c/583715/ | |
| 15:26:57 | stephenfin | mriedem: However, are you sure about the "scheduler getting called" bit. Comments/logic in the code suggest otherwise mriedem: Are you sure. Your comments here say something else https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L901-L916 | |
| 15:27:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L920 | |
| 15:27:25 | stephenfin | By rebuild I'm referring to the 'openstack server rebuild $NAME" command | |
| 15:27:56 | mriedem | note the "*unless* a new image # was specified" | |
| 15:28:09 | stephenfin | Riiiight, missed that | |
| 15:28:10 | mriedem | the rebuild_instance method in conductor manager is totally coupled to how the api code works | |
| 15:28:31 | stephenfin | So, what do I do in the "no new instance" case | |
| 15:28:35 | stephenfin | Happily carry on? | |
| 15:28:41 | mriedem | no request spec | |
| 15:28:41 | mriedem | ? | |
| 15:28:49 | sean-k-mooney | mriedem: even if the same image is passed we should proably reschdule as the image metadata could have changed | |
| 15:28:50 | mriedem | no new image? | |
| 15:29:15 | mriedem | stephenfin: if the image doesn't change, we don't call the scheduler | |
| 15:30:04 | mriedem | sean-k-mooney: yeah, maybe, which would be new logic in the api to detect that since we persist the original image properties on the instance system_metadata i think | |
| 15:30:44 | stephenfin | mriedem: Right, so I can't recalculate NUMA affinity for any network changes. That's OK, yeah? | |
| 15:30:50 | dansmith | mriedem: right, and I think we said that we shouldn't refuse a rebuild because an image has changed metadata since the user then has very little recourse | |
| 15:31:29 | mriedem | dansmith: i don't remember talking about image meta changing on the original image, but ok | |
| 15:31:34 | dansmith | I do | |
| 15:31:42 | sean-k-mooney | mriedem: hum well if we are using the cached copy it should safe but if we pull them down from glance then we might violate one of the image requests or not be able to supprot one of them | |
| 15:31:46 | mriedem | stephenfin: yes because we assume the network affinity on the same host for the same image hasn't changed | |
| 15:32:35 | stephenfin | mriedem: Sweet | |
| 15:32:43 | stephenfin | https://imgflip.com/i/2ejmw4 | |
| 15:34:15 | sean-k-mooney | dansmith: mriedem do we also use the embeded flavor on rebuild? changes to image meta on rebuild is logically the same as changes to flavour extra specs so we should proably have the same behavior | |
| 15:34:26 | dansmith | we do | |
| 15:35:06 | dansmith | I'm not actually sure we do use the embedded _image_ properties for rebuild, but I'm saying I don't think that's a problem | |
| 15:35:16 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3172 | |