| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-14 | |||
| 11:32:17 | sean-k-mooney | cold migrate and resize would both triger this. live migrate who knows | |
| 11:32:22 | mdbooth | logan-: At this point if you were a RH customer I'd ask you for sosreports. | |
| 11:32:54 | mdbooth | Basically I'd want to see full logs from the initial error which caused the root disk to be deleted. | |
| 11:33:32 | mdbooth | The subsequent error after the root disk has been deleted isn't all that interesting. | |
| 11:34:03 | sean-k-mooney | mdbooth: in theory this would alow apply to any missmatch in image backend right e.g. any combinantion lvm or image or rbd where the backend changes. | |
| 11:34:32 | mdbooth | sean-k-mooney: Yes. The code explicitly assumes that the layouts are identical on source and dest. | |
| 11:35:30 | sean-k-mooney | so this seams like something we should add to the host state object and then check in the scheuler in the future | |
| 11:35:44 | logan- | Ok I can pull more logs this afternoon. What would help? nova-compute in debug on both source & dest I’m guessing? | |
| 11:36:23 | mdbooth | logan-: Yep, specifically for the resize operation itself, not just the subsequent failure to boot | |
| 11:36:31 | logan- | Got it | |
| 11:36:35 | logan- | Will do. Thanks | |
| 11:40:41 | mdbooth | logan-: If you look in the instance directory on the destination you should find the root disk there | |
| 11:40:51 | mdbooth | logan-: Also, you should be able to revert resize | |
| 11:41:40 | mdbooth | I don't think we've actually deleted the root disk here, it's just that the dest is looking for it in the wrong place. | |
| 11:42:43 | sean-k-mooney | we might have deleted it on the source node | |
| 11:43:51 | sean-k-mooney | or do we not do that until we confirm and your guessing on revert it tries to find it using the location form the dest | |
| 11:45:09 | mdbooth | Not until confirm | |
| 11:45:20 | mdbooth | Revert will use the copy which remains on the source | |
| 11:46:29 | mdbooth | So actually I think this is just a case of extremely bad user experience. The operation isn't supported, but we could warn earlier rather than going down in flames. I don't believe any data has been lost. | |
| 11:47:03 | mdbooth | logan-: If ^^^ is correct, no need for those logs, btw. | |
| 11:56:42 | openstackgerrit | Merged openstack/nova stable/rocky: Set defult value of num_nvme_discover_tries=5 https://review.openstack.org/608683 | |
| 12:00:03 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Allow driver to specify switch&port for faster lookup https://review.openstack.org/617695 | |
| 12:52:40 | openstackgerrit | Merged openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | |
| 13:08:40 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:19:20 | logan- | updated the bug with additional details mdbooth | |
| 13:19:51 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | |
| 13:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | |
| 13:23:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 13:23:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 13:26:09 | mdbooth | logan-: Can you retry the revert after setting instance state to 'resize_confirm' ? | |
| 13:32:18 | ivaat | hi i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | |
| 13:32:18 | openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | |
| 13:42:59 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:47:25 | logan- | mdbooth: I guess that will require database shenanigans? "openstack server set: error: argument --state: invalid choice: u'resize_confirm' (choose from 'active', 'error')" | |
| 13:50:51 | mriedem | logan-: heh, reset state won't allow you to set it back to VERIFY_RESIZE | |
| 13:50:58 | mriedem | so yes, db surgery | |
| 13:51:12 | logan- | ok i'll give it a shot | |
| 13:51:39 | mriedem | because you need to get to this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3506 | |
| 13:53:56 | ivaat | hi! i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | |
| 13:53:56 | openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | |
| 13:54:04 | ivaat | i am sure it is bug | |
| 13:58:36 | mriedem | ivaat: see my comment in the bug, it's not a bug, unless you can investigate and prove otherwise | |
| 13:58:41 | mriedem | it looks like a misconfiguration | |
| 13:59:15 | ivaat | mriedem, well i attached nova.conf as well. tried user with logging in with same user. works | |
| 14:00:04 | ivaat | if it looks like misconfiguration then could you point out what part in config? i have gone trough config several times. followed official doc | |
| 14:00:38 | mriedem | see my comment on the bug | |
| 14:00:51 | ivaat | bug seems to be deleted | |
| 14:01:57 | ivaat | sry going directly with bug link i can see it now | |
| 14:02:49 | ivaat | mriedem, wtf. i just confirmed i have done this several times.. just take your time when helping. conf was attached to see what is wrong | |
| 14:03:36 | logan- | i tried hacking instances.vm_state='resized' and now it says "Instance has not been resized." so I guess there is some other state fields that need to be updated to hack it back to a revertable state | |
| 14:05:22 | mriedem | ivaat: i looked at the paste in the bug, i didn't see [keystone_authtoken] in there | |
| 14:07:31 | mriedem | logan-: so what did you do? vm was in error and you fixed the disk and tried to confirm again but that failed b/c it was in error rather than resized status? | |
| 14:07:37 | ivaat | at moment when i installed was no guide to add keystone_authtoken | |
| 14:07:55 | mriedem | ivaat: i don't know what install guide you were looking at | |
| 14:08:02 | mriedem | maybe something very old | |
| 14:09:12 | ivaat | alright, i will test it out | |
| 14:09:16 | ivaat | thx for now | |
| 14:10:15 | aspiers[m] | mriedem: does it make sense to aim for a quick hallway discussion about SEV while we're here? sean-k-mooney too if you're around | |
| 14:10:35 | ivaat | even nova conf is missing such keyword.. since it is section it should have been there | |
| 14:10:48 | ivaat | i think i got previous version conf | |
| 14:10:59 | logan- | mriedem: ya, resize -> error -> unable to revert resize due to error state | |
| 14:11:20 | sean-k-mooney | aspiers[m]: im not in berlin unfortunetly but im on irc | |
| 14:12:05 | aspiers[m] | sean-k-mooney: OK thanks. We can continue on the review of course, but feel free to ping me here too if you prefer | |
| 14:12:35 | logan- | So looking at how to revert the resize and return it back to the source node where the disk is still present in /var/lib/nova/instances/<uuid>_resize/* | |
| 14:12:36 | sean-k-mooney | if im needed i could dial in to a googel hangout but im sure there are enough people to sync up with to figure things out there without me | |
| 14:12:53 | aspiers[m] | I'm currently liasing with our SEV engineers and AMD's to get more clarity around the questions you guys raised | |
| 14:15:08 | aspiers[m] | guessing a hangout won't be needed but will certainly let you know if I'm wrong | |
| 14:15:36 | sean-k-mooney | aspiers[m]: ok just read your responces. if we can just document what will work and what wont then im more or less happy. | |
| 14:16:11 | sean-k-mooney | aspiers[m]: im also fine with an fix constatin + vm ram + video ram | |
| 14:16:19 | aspiers[m] | sean-k-mooney: absolutely - next patch set should give much more clarity over what is expected to work vs. not | |
| 14:16:25 | sean-k-mooney | but if we go that way i would make the constant a config value | |
| 14:16:56 | aspiers[m] | OK that sounds like a good idea as a short-term workaround until qemu exposes that calculation | |
| 14:17:42 | aspiers[m] | so you mean a fixed configurable value for the overhead per VM, right? so <hard_limit> would be calculated as flavor ram plus that value | |
| 14:17:54 | sean-k-mooney | the issue is qemu cant really know this up front as it depends on several things that can cahgne at runtime and from version to version | |
| 14:18:18 | aspiers[m] | possibly with an adjustment if guest video ram is specified via extra specs or image props | |
| 14:18:43 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 14:18:52 | sean-k-mooney | aspiers[m]: ya it could be a config value for the over head or a flavor extraspec | |
| 14:19:29 | sean-k-mooney | just some way to reduce or increase the 512 without code change if we mesuer it is too big or two small later | |
| 14:19:56 | sean-k-mooney | so it would be ram + video ram + a configurable constat overhead | |
| 14:20:40 | sean-k-mooney | that said i think we do this somewhere already | |
| 14:23:44 | sean-k-mooney | aspiers[m]: we support setting the hardlimit in https://github.com/openstack/nova/blob/88951ca98e1b286b58aa1ad94f9af40b8260c01f/nova/virt/libvirt/config.py#L2102-L2133 and i belive we use it for realtime instances | |
| 14:25:00 | aspiers[m] | sean-k-mooney: ah yeah, I guess probably for the same reason (memory pinning) | |
| 14:25:57 | sean-k-mooney | aspiers[m]: yes we use it so we can memlock the memory for realtime instaces. also all hugepage backed instace will also have the hugepages memlocked which we use for ovs-dpdk so it can do dma into the guest memory | |
| 14:29:32 | sean-k-mooney | im trying not to be angry about the fact that apparently we only use this in unit tests.... http://codesearch.openstack.org/?q=hard_limit&i=nope&files=&repos=nova | |
| 14:30:05 | aspiers[m] | haha :) | |
| 14:31:13 | sean-k-mooney | aspiers[m]: well the code is there for you to use... | |
| 14:34:37 | aspiers[m] | indeed. I don't think the actual coding is gonna be too hard | |
| 15:21:53 | mriedem | dansmith: so belmiro gave me another issue to worry about, for which he's going to open a bug. they drop the allocation candidates limit down to 10 to make scheduling faster, but they also at times disable a large number of computes in a cell for $reasons (mnaser also does that for load balancing the cloud); the problem then is b/c of the low alloc candidates limit, placement gives back 10 disabled computes and gets novalidhost | |
| 15:22:08 | mriedem | to workaround it he set reserved = total for inventory in placement on the disabled nodes, | |
| 15:22:12 | mriedem | but that sucks for hundreds of vms | |
| 15:22:20 | dansmith | ouch | |
| 15:22:24 | mriedem | i suggested a pre-request placement filter to somehow filter out disabled nodes | |
| 15:22:40 | mriedem | maybe forbidden trait/aggregate, idk | |
| 15:22:48 | dansmith | hmm | |
| 15:22:59 | dansmith | they disable compute or disable a cell? | |
| 15:23:04 | dansmith | *computes | |
| 15:23:08 | sean-k-mooney | mriedem: i was going to say yes tag with a trait and use forbindin traits but then we are using placmeent to sotre metadata | |
| 15:24:34 | dansmith | mriedem: I think the right thing to do is to make placement results better so they don't need such a tiny result set to mitigate performance issues | |
| 15:24:42 | dansmith | but, regardless of what that number is, the same problem could happen | |