| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 13:53:34 | tssurya | dansmith: ack | |
| 13:53:53 | pooja_jadhav | dansmith, mriedem: I have 2 nodes, I am facing issue for migrate API. Can u tell me what are the pre requisites? | |
| 13:54:08 | mriedem | pooja_jadhav: cold migrate? | |
| 13:54:45 | mriedem | unless the nodes are on the same shared storage pool you have to configure ssh access for the nova user between the nodes | |
| 13:55:02 | pooja_jadhav | mriedem, dansmith: for unshelve, there is issue (DISK_GB record is added after unshelve the instance). | |
| 13:55:13 | mriedem | pooja_jadhav: added where? | |
| 13:55:22 | mriedem | to placement? | |
| 13:55:22 | dansmith | mriedem: you have a patch for that right? | |
| 13:55:25 | mriedem | dansmith: yeah | |
| 13:55:45 | pooja_jadhav | mriedem: yes, in the allocation table | |
| 13:56:02 | pooja_jadhav | can I provide the details, I have tested? | |
| 13:56:30 | mriedem | unshelve should be handled here https://review.openstack.org/#/c/583715/3/nova/conductor/manager.py@806 | |
| 13:57:16 | mriedem | i could update the functional test in that change to run a shelve/unshelve routine to make sure DISK_GB still doesn't show up in placement allocations | |
| 13:58:07 | pooja_jadhav | mriedem: cold/live migrate (both trying to test) | |
| 13:58:27 | mriedem | damn i have to rebase this series because https://review.openstack.org/#/c/583646/ was rebased | |
| 13:58:35 | mriedem | pooja_jadhav: https://docs.openstack.org/devstack/latest/guides/multinode-lab.html | |
| 13:58:38 | mriedem | if you're using devstack | |
| 13:59:52 | pooja_jadhav | mriedem: thanks, will check and let u know if the queries :) | |
| 14:00:04 | pooja_jadhav | yes, I am using devstack | |
| 14:00:26 | mriedem | that doc might be old | |
| 14:04:59 | pooja_jadhav | mriedem: Also tested resize and evacuate, but no allocation records get added for DISK_GB. | |
| 14:06:03 | pooja_jadhav | mriedem: But for evacuate, hypervisor-stats showing incorrect incorrect usages. | |
| 14:06:15 | openstackgerrit | Boxiang Zhu proposed openstack/nova stable/pike: Fix "instance snap min disk size err after resize instance" https://review.openstack.org/584770 | |
| 14:08:52 | mriedem | pooja_jadhav: we already know that hypervisor-stats has latent bugs for reporting disk based on the flavor and not if the server is volume-backed | |
| 14:09:08 | mriedem | that's what https://review.openstack.org/#/c/583646/ shows | |
| 14:09:15 | pooja_jadhav | mriedem: Ohk | |
| 14:10:26 | pooja_jadhav | mriedem: Have tested live/cold migrate then? for allocation of DISK_GB or not? | |
| 14:11:13 | mriedem | pooja_jadhav: https://review.openstack.org/#/c/583715/ functionally tests that for cold migration | |
| 14:11:26 | mriedem | https://review.openstack.org/#/c/583715/3/nova/tests/functional/test_servers.py | |
| 14:12:00 | mriedem | i was saying i could add a shelve/unshelve wrinkle to that test as well since you said you saw allocations for DISK_GB in placement after unshelve | |
| 14:13:07 | mriedem | but i need to rebase this series now | |
| 14:14:35 | pooja_jadhav | mriedem: thanks :) | |
| 14:24:13 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 14:56:35 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove usage of migrateToURI{2} APIs https://review.openstack.org/567258 | |
| 14:59:26 | mriedem | efried: hmm, looks like the retrying package might not be thread safe http://logs.openstack.org/03/583203/2/check/tempest-full-py3/0b5111a/controller/logs/screen-n-cond-cell1.txt.gz#_Jul_23_13_35_01_491350 | |
| 14:59:47 | mriedem | got a thread switch there which looks like it caused a failure during RT._update | |
| 15:00:05 | mriedem | well, maybe not | |
| 15:00:16 | mriedem | could just be throwing me b/c it's in the trace b/c of the decorator | |
| 15:00:57 | mriedem | hit a messaging timeout, maybe just a slow node | |
| 15:01:22 | mriedem | Jul 23 13:35:01.252889 ubuntu-xenial-packethost-us-west-1-0000896864 nova-compute[22210]: WARNING nova.servicegroup.drivers.db [None req-93269558-9418-452a-b391-8766a59984a5 tempest-TestVolumeSwap-747104316 tempest-TestVolumeSwap-747104316] Lost connection to nova-conductor for reporting service status.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 06f920f1d628445b877ff4100980047e | |
| 15:01:22 | efried | mriedem: afaik, retrying doesn't use threads at all. Are you saying it has a global or something? | |
| 15:01:22 | mriedem | yeah | |
| 15:01:32 | mriedem | efried: no i think it just threw me in the stacktrace | |
| 15:01:36 | efried | ight | |
| 15:02:54 | sean-k-mooney | QQ are the functional tests safe to run with devstack stacked? | |
| 15:03:12 | mriedem | nova tox -e functional? | |
| 15:03:14 | mriedem | they run in a venv so yes | |
| 15:03:40 | sean-k-mooney | ok cool was just wondering if i would get any failures due to haveing opentack running | |
| 15:15:30 | stephenfin | mriedem: RE: numa-aware-vswitches and move operations. Rebuilds don't run through the scheduler and don't make claims so I ignore that kind of rebuild operation, right? | |
| 15:15:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Report 0 root_gb in resource tracker if instance is bfv. https://review.openstack.org/584204 | |
| 15:15:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715 | |
| 15:15:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739 | |
| 15:15:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add shelve/unshelve wrinkle to volume-backed disk func test https://review.openstack.org/584931 | |
| 15:15:48 | mriedem | pooja_jadhav: ^ added a shelve/unshelve functional test to the end of the series and that shows things working as designed | |
| 15:16:10 | sean-k-mooney | stephenfin: rebuilds should still go through the scheuler | |
| 15:16:14 | mriedem | stephenfin: depends, what does the RUN_ON_REBUILD flag say for NUMATopologyFilter? | |
| 15:16:21 | mriedem | sean-k-mooney: not in all cases, and not for all filters | |
| 15:16:32 | mriedem | stephenfin: and to be clear, you're talking about rebuild, not evacuate, yes? | |
| 15:16:46 | mriedem | RUN_ON_REBUILD = True | |
| 15:16:46 | mriedem | """Filter on requested NUMA topology.""" | |
| 15:16:46 | mriedem | stephenfin: class NUMATopologyFilter(filters.BaseHostFilter): | |
| 15:17:03 | stephenfin | mriedem: It's True and yes | |
| 15:17:18 | mriedem | stephenfin: so then you need to put the stuff in the request spec if we're going through the scheduler during rebuild b/c the image changes | |
| 15:17:29 | mriedem | because image_props.get('hw_cpu_policy')] and image_props.get('hw_cpu_thread_policy')] | |
| 15:17:34 | mriedem | if the image changes, the numa stuff can change | |
| 15:17:51 | mriedem | stephenfin: rebuild isn't a move, but we have to validate the new image | |
| 15:18:17 | mriedem | *validate the new image still works on the current host | |
| 15:19:12 | sean-k-mooney | mriedem: a rebuilt instance could land on a different host then the instace was on previously though right. its effectivly a new boot with the same port/volumes as the original instance | |
| 15:19:34 | mriedem | sean-k-mooney: no | |
| 15:19:44 | mriedem | sean-k-mooney: are you talking about evacuate? | |
| 15:20:16 | sean-k-mooney | no i taught if we rebuilt and the previous host was nolonger valid we would schdule to another host that was | |
| 15:20:23 | sean-k-mooney | i gues we just fail to rebuild | |
| 15:21:19 | mriedem | you get a novalidhost | |
| 15:21:27 | mriedem | if the rebuild + new image doesn't fit on the current host | |
| 15:21:41 | mriedem | instance goes to error state and you have to fix it | |
| 15:22:19 | mriedem | bauzas: ping https://review.openstack.org/#/c/583715/ | |
| 15:23:10 | mriedem | bauzas: also ping https://review.openstack.org/#/c/581813/ - that's the nova-status check for request spec migrations which means if we get that in rocky, we can drop the reqspec migrations and compat code in stein | |
| 15:23:15 | mriedem | which is something i desperatly want to do | |
| 15:23:21 | mriedem | that and spellcheck | |
| 15:23:21 | sean-k-mooney | mriedem: right i was assuming we would hit the retry fileter and try another host but i guess not. in my mind i was assuming we did the host affinity for rebuil via a weigher rather then filtering to just the current host | |
| 15:23:43 | mriedem | sean-k-mooney: we pass a hint to the scheduler telling it to filter on the existing instance host | |
| 15:24:42 | sean-k-mooney | mriedem: cool ill look into how that work more closely in the future. good to know | |
| 15:26:06 | mriedem | stephenfin: so i assume you might have 2 patches: 1 to add the physnet and tunneled plumbing to the nova.network.model.Network object and populate it during _build_network_info_model, and then another change to call some utils method in the various conductor spots to populate ReuqestSpec.network_metadata using instance.info_cache.network_info | |
| 15:26:34 | stephenfin | mriedem: That's exactly what I have, albeit rolled into one patch | |
| 15:26:38 | mriedem | stephenfin: btw, this change is doing the same call spots in conductor that you'll need https://review.openstack.org/#/c/583715/ | |
| 15:26:57 | stephenfin | mriedem: However, are you sure about the "scheduler getting called" bit. Comments/logic in the code suggest otherwise mriedem: Are you sure. Your comments here say something else https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L901-L916 | |
| 15:27:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L920 | |
| 15:27:25 | stephenfin | By rebuild I'm referring to the 'openstack server rebuild $NAME" command | |
| 15:27:56 | mriedem | note the "*unless* a new image # was specified" | |
| 15:28:09 | stephenfin | Riiiight, missed that | |
| 15:28:10 | mriedem | the rebuild_instance method in conductor manager is totally coupled to how the api code works | |
| 15:28:31 | stephenfin | So, what do I do in the "no new instance" case | |
| 15:28:35 | stephenfin | Happily carry on? | |
| 15:28:41 | mriedem | ? | |
| 15:28:41 | mriedem | no request spec | |
| 15:28:49 | sean-k-mooney | mriedem: even if the same image is passed we should proably reschdule as the image metadata could have changed | |
| 15:28:50 | mriedem | no new image? | |
| 15:29:15 | mriedem | stephenfin: if the image doesn't change, we don't call the scheduler | |
| 15:30:04 | mriedem | sean-k-mooney: yeah, maybe, which would be new logic in the api to detect that since we persist the original image properties on the instance system_metadata i think | |
| 15:30:44 | stephenfin | mriedem: Right, so I can't recalculate NUMA affinity for any network changes. That's OK, yeah? | |