| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 13:48:36 | mriedem | given our current merge rates | |
| 13:48:38 | dansmith | oye | |
| 13:48:53 | mriedem | it's taken a week to land code right now | |
| 13:49:07 | dansmith | yarp | |
| 13:49:16 | mriedem | so i'm really only mostly interested in pushing things across the line this week that is already ready to go and has had a lot of review already | |
| 13:49:18 | mriedem | personally, | |
| 13:49:33 | mriedem | plus i'm going to have a shitty week with some personal stuff going on so won't be fully available | |
| 13:50:15 | pooja_jadhav | dansmith: hello | |
| 13:50:26 | dansmith | mriedem: ack | |
| 13:50:29 | dansmith | pooja_jadhav: hi | |
| 13:50:41 | efried | Scheduler meeting in ten minutes in #openstack-meeting-alt | |
| 13:50:52 | tssurya | mriedem: ack thanks a lot for all the time an review doen on this spec (if I haven't yet thanked you) | |
| 13:51:21 | tssurya | dansmith: thanks to you too for all the help and reviews | |
| 13:51:43 | pooja_jadhav | dansmith: Actually, gone through your patch, https://review.openstack.org/#/c/580720/. Now testing the move operations (resize, unshelve, evacuate and migrate). | |
| 13:52:54 | openstackgerrit | Dan Smith proposed openstack/nova master: Online data migration for queued_for_delete flag https://review.openstack.org/584504 | |
| 13:53:21 | dansmith | tssurya: mriedem: fixed py35 issue in the test ^ | |
| 13:53:23 | mriedem | pooja_jadhav: are there problems or you're just letting us know you're testing? | |
| 13:53:32 | mriedem | pooja_jadhav: also, thanks for testing this stuff | |
| 13:53:34 | tssurya | dansmith: ack | |
| 13:53:53 | pooja_jadhav | dansmith, mriedem: I have 2 nodes, I am facing issue for migrate API. Can u tell me what are the pre requisites? | |
| 13:54:08 | mriedem | pooja_jadhav: cold migrate? | |
| 13:54:45 | mriedem | unless the nodes are on the same shared storage pool you have to configure ssh access for the nova user between the nodes | |
| 13:55:02 | pooja_jadhav | mriedem, dansmith: for unshelve, there is issue (DISK_GB record is added after unshelve the instance). | |
| 13:55:13 | mriedem | pooja_jadhav: added where? | |
| 13:55:22 | mriedem | to placement? | |
| 13:55:22 | dansmith | mriedem: you have a patch for that right? | |
| 13:55:25 | mriedem | dansmith: yeah | |
| 13:55:45 | pooja_jadhav | mriedem: yes, in the allocation table | |
| 13:56:02 | pooja_jadhav | can I provide the details, I have tested? | |
| 13:56:30 | mriedem | unshelve should be handled here https://review.openstack.org/#/c/583715/3/nova/conductor/manager.py@806 | |
| 13:57:16 | mriedem | i could update the functional test in that change to run a shelve/unshelve routine to make sure DISK_GB still doesn't show up in placement allocations | |
| 13:58:07 | pooja_jadhav | mriedem: cold/live migrate (both trying to test) | |
| 13:58:27 | mriedem | damn i have to rebase this series because https://review.openstack.org/#/c/583646/ was rebased | |
| 13:58:35 | mriedem | pooja_jadhav: https://docs.openstack.org/devstack/latest/guides/multinode-lab.html | |
| 13:58:38 | mriedem | if you're using devstack | |
| 13:59:52 | pooja_jadhav | mriedem: thanks, will check and let u know if the queries :) | |
| 14:00:04 | pooja_jadhav | yes, I am using devstack | |
| 14:00:26 | mriedem | that doc might be old | |
| 14:04:59 | pooja_jadhav | mriedem: Also tested resize and evacuate, but no allocation records get added for DISK_GB. | |
| 14:06:03 | pooja_jadhav | mriedem: But for evacuate, hypervisor-stats showing incorrect incorrect usages. | |
| 14:06:15 | openstackgerrit | Boxiang Zhu proposed openstack/nova stable/pike: Fix "instance snap min disk size err after resize instance" https://review.openstack.org/584770 | |
| 14:08:52 | mriedem | pooja_jadhav: we already know that hypervisor-stats has latent bugs for reporting disk based on the flavor and not if the server is volume-backed | |
| 14:09:08 | mriedem | that's what https://review.openstack.org/#/c/583646/ shows | |
| 14:09:15 | pooja_jadhav | mriedem: Ohk | |
| 14:10:26 | pooja_jadhav | mriedem: Have tested live/cold migrate then? for allocation of DISK_GB or not? | |
| 14:11:13 | mriedem | pooja_jadhav: https://review.openstack.org/#/c/583715/ functionally tests that for cold migration | |
| 14:11:26 | mriedem | https://review.openstack.org/#/c/583715/3/nova/tests/functional/test_servers.py | |
| 14:12:00 | mriedem | i was saying i could add a shelve/unshelve wrinkle to that test as well since you said you saw allocations for DISK_GB in placement after unshelve | |
| 14:13:07 | mriedem | but i need to rebase this series now | |
| 14:14:35 | pooja_jadhav | mriedem: thanks :) | |
| 14:24:13 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 14:56:35 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Remove usage of migrateToURI{2} APIs https://review.openstack.org/567258 | |
| 14:59:26 | mriedem | efried: hmm, looks like the retrying package might not be thread safe http://logs.openstack.org/03/583203/2/check/tempest-full-py3/0b5111a/controller/logs/screen-n-cond-cell1.txt.gz#_Jul_23_13_35_01_491350 | |
| 14:59:47 | mriedem | got a thread switch there which looks like it caused a failure during RT._update | |
| 15:00:05 | mriedem | well, maybe not | |
| 15:00:16 | mriedem | could just be throwing me b/c it's in the trace b/c of the decorator | |
| 15:00:57 | mriedem | hit a messaging timeout, maybe just a slow node | |
| 15:01:22 | mriedem | Jul 23 13:35:01.252889 ubuntu-xenial-packethost-us-west-1-0000896864 nova-compute[22210]: WARNING nova.servicegroup.drivers.db [None req-93269558-9418-452a-b391-8766a59984a5 tempest-TestVolumeSwap-747104316 tempest-TestVolumeSwap-747104316] Lost connection to nova-conductor for reporting service status.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 06f920f1d628445b877ff4100980047e | |
| 15:01:22 | efried | mriedem: afaik, retrying doesn't use threads at all. Are you saying it has a global or something? | |
| 15:01:22 | mriedem | yeah | |
| 15:01:32 | mriedem | efried: no i think it just threw me in the stacktrace | |
| 15:01:36 | efried | ight | |
| 15:02:54 | sean-k-mooney | QQ are the functional tests safe to run with devstack stacked? | |
| 15:03:12 | mriedem | nova tox -e functional? | |
| 15:03:14 | mriedem | they run in a venv so yes | |
| 15:03:40 | sean-k-mooney | ok cool was just wondering if i would get any failures due to haveing opentack running | |
| 15:15:30 | stephenfin | mriedem: RE: numa-aware-vswitches and move operations. Rebuilds don't run through the scheduler and don't make claims so I ignore that kind of rebuild operation, right? | |
| 15:15:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Report 0 root_gb in resource tracker if instance is bfv. https://review.openstack.org/584204 | |
| 15:15:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715 | |
| 15:15:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739 | |
| 15:15:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add shelve/unshelve wrinkle to volume-backed disk func test https://review.openstack.org/584931 | |
| 15:15:48 | mriedem | pooja_jadhav: ^ added a shelve/unshelve functional test to the end of the series and that shows things working as designed | |
| 15:16:10 | sean-k-mooney | stephenfin: rebuilds should still go through the scheuler | |
| 15:16:14 | mriedem | stephenfin: depends, what does the RUN_ON_REBUILD flag say for NUMATopologyFilter? | |
| 15:16:21 | mriedem | sean-k-mooney: not in all cases, and not for all filters | |
| 15:16:32 | mriedem | stephenfin: and to be clear, you're talking about rebuild, not evacuate, yes? | |
| 15:16:46 | mriedem | RUN_ON_REBUILD = True | |
| 15:16:46 | mriedem | """Filter on requested NUMA topology.""" | |
| 15:16:46 | mriedem | stephenfin: class NUMATopologyFilter(filters.BaseHostFilter): | |
| 15:17:03 | stephenfin | mriedem: It's True and yes | |
| 15:17:18 | mriedem | stephenfin: so then you need to put the stuff in the request spec if we're going through the scheduler during rebuild b/c the image changes | |
| 15:17:29 | mriedem | because image_props.get('hw_cpu_policy')] and image_props.get('hw_cpu_thread_policy')] | |
| 15:17:34 | mriedem | if the image changes, the numa stuff can change | |
| 15:17:51 | mriedem | stephenfin: rebuild isn't a move, but we have to validate the new image | |
| 15:18:17 | mriedem | *validate the new image still works on the current host | |
| 15:19:12 | sean-k-mooney | mriedem: a rebuilt instance could land on a different host then the instace was on previously though right. its effectivly a new boot with the same port/volumes as the original instance | |
| 15:19:34 | mriedem | sean-k-mooney: no | |
| 15:19:44 | mriedem | sean-k-mooney: are you talking about evacuate? | |
| 15:20:16 | sean-k-mooney | no i taught if we rebuilt and the previous host was nolonger valid we would schdule to another host that was | |
| 15:20:23 | sean-k-mooney | i gues we just fail to rebuild | |
| 15:21:19 | mriedem | you get a novalidhost | |
| 15:21:27 | mriedem | if the rebuild + new image doesn't fit on the current host | |
| 15:21:41 | mriedem | instance goes to error state and you have to fix it | |
| 15:22:19 | mriedem | bauzas: ping https://review.openstack.org/#/c/583715/ | |
| 15:23:10 | mriedem | bauzas: also ping https://review.openstack.org/#/c/581813/ - that's the nova-status check for request spec migrations which means if we get that in rocky, we can drop the reqspec migrations and compat code in stein | |
| 15:23:15 | mriedem | which is something i desperatly want to do | |
| 15:23:21 | mriedem | that and spellcheck | |
| 15:23:21 | sean-k-mooney | mriedem: right i was assuming we would hit the retry fileter and try another host but i guess not. in my mind i was assuming we did the host affinity for rebuil via a weigher rather then filtering to just the current host | |
| 15:23:43 | mriedem | sean-k-mooney: we pass a hint to the scheduler telling it to filter on the existing instance host | |
| 15:24:42 | sean-k-mooney | mriedem: cool ill look into how that work more closely in the future. good to know | |