| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-14 | |||
| 11:34:32 | mdbooth | sean-k-mooney: Yes. The code explicitly assumes that the layouts are identical on source and dest. | |
| 11:35:30 | sean-k-mooney | so this seams like something we should add to the host state object and then check in the scheuler in the future | |
| 11:35:44 | logan- | Ok I can pull more logs this afternoon. What would help? nova-compute in debug on both source & dest I’m guessing? | |
| 11:36:23 | mdbooth | logan-: Yep, specifically for the resize operation itself, not just the subsequent failure to boot | |
| 11:36:31 | logan- | Got it | |
| 11:36:35 | logan- | Will do. Thanks | |
| 11:40:41 | mdbooth | logan-: If you look in the instance directory on the destination you should find the root disk there | |
| 11:40:51 | mdbooth | logan-: Also, you should be able to revert resize | |
| 11:41:40 | mdbooth | I don't think we've actually deleted the root disk here, it's just that the dest is looking for it in the wrong place. | |
| 11:42:43 | sean-k-mooney | we might have deleted it on the source node | |
| 11:43:51 | sean-k-mooney | or do we not do that until we confirm and your guessing on revert it tries to find it using the location form the dest | |
| 11:45:09 | mdbooth | Not until confirm | |
| 11:45:20 | mdbooth | Revert will use the copy which remains on the source | |
| 11:46:29 | mdbooth | So actually I think this is just a case of extremely bad user experience. The operation isn't supported, but we could warn earlier rather than going down in flames. I don't believe any data has been lost. | |
| 11:47:03 | mdbooth | logan-: If ^^^ is correct, no need for those logs, btw. | |
| 11:56:42 | openstackgerrit | Merged openstack/nova stable/rocky: Set defult value of num_nvme_discover_tries=5 https://review.openstack.org/608683 | |
| 12:00:03 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Allow driver to specify switch&port for faster lookup https://review.openstack.org/617695 | |
| 12:52:40 | openstackgerrit | Merged openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | |
| 13:08:40 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:19:20 | logan- | updated the bug with additional details mdbooth | |
| 13:19:51 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | |
| 13:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | |
| 13:23:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 13:23:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 13:26:09 | mdbooth | logan-: Can you retry the revert after setting instance state to 'resize_confirm' ? | |
| 13:32:18 | openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | |
| 13:32:18 | ivaat | hi i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | |
| 13:42:59 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:47:25 | logan- | mdbooth: I guess that will require database shenanigans? "openstack server set: error: argument --state: invalid choice: u'resize_confirm' (choose from 'active', 'error')" | |
| 13:50:51 | mriedem | logan-: heh, reset state won't allow you to set it back to VERIFY_RESIZE | |
| 13:50:58 | mriedem | so yes, db surgery | |
| 13:51:12 | logan- | ok i'll give it a shot | |
| 13:51:39 | mriedem | because you need to get to this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3506 | |
| 13:53:56 | openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | |
| 13:53:56 | ivaat | hi! i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | |
| 13:54:04 | ivaat | i am sure it is bug | |
| 13:58:36 | mriedem | ivaat: see my comment in the bug, it's not a bug, unless you can investigate and prove otherwise | |
| 13:58:41 | mriedem | it looks like a misconfiguration | |
| 13:59:15 | ivaat | mriedem, well i attached nova.conf as well. tried user with logging in with same user. works | |
| 14:00:04 | ivaat | if it looks like misconfiguration then could you point out what part in config? i have gone trough config several times. followed official doc | |
| 14:00:38 | mriedem | see my comment on the bug | |
| 14:00:51 | ivaat | bug seems to be deleted | |
| 14:01:57 | ivaat | sry going directly with bug link i can see it now | |
| 14:02:49 | ivaat | mriedem, wtf. i just confirmed i have done this several times.. just take your time when helping. conf was attached to see what is wrong | |
| 14:03:36 | logan- | i tried hacking instances.vm_state='resized' and now it says "Instance has not been resized." so I guess there is some other state fields that need to be updated to hack it back to a revertable state | |
| 14:05:22 | mriedem | ivaat: i looked at the paste in the bug, i didn't see [keystone_authtoken] in there | |
| 14:07:31 | mriedem | logan-: so what did you do? vm was in error and you fixed the disk and tried to confirm again but that failed b/c it was in error rather than resized status? | |
| 14:07:37 | ivaat | at moment when i installed was no guide to add keystone_authtoken | |
| 14:07:55 | mriedem | ivaat: i don't know what install guide you were looking at | |
| 14:08:02 | mriedem | maybe something very old | |
| 14:09:12 | ivaat | alright, i will test it out | |
| 14:09:16 | ivaat | thx for now | |
| 14:10:15 | aspiers[m] | mriedem: does it make sense to aim for a quick hallway discussion about SEV while we're here? sean-k-mooney too if you're around | |
| 14:10:35 | ivaat | even nova conf is missing such keyword.. since it is section it should have been there | |
| 14:10:48 | ivaat | i think i got previous version conf | |
| 14:10:59 | logan- | mriedem: ya, resize -> error -> unable to revert resize due to error state | |
| 14:11:20 | sean-k-mooney | aspiers[m]: im not in berlin unfortunetly but im on irc | |
| 14:12:05 | aspiers[m] | sean-k-mooney: OK thanks. We can continue on the review of course, but feel free to ping me here too if you prefer | |
| 14:12:35 | logan- | So looking at how to revert the resize and return it back to the source node where the disk is still present in /var/lib/nova/instances/<uuid>_resize/* | |
| 14:12:36 | sean-k-mooney | if im needed i could dial in to a googel hangout but im sure there are enough people to sync up with to figure things out there without me | |
| 14:12:53 | aspiers[m] | I'm currently liasing with our SEV engineers and AMD's to get more clarity around the questions you guys raised | |
| 14:15:08 | aspiers[m] | guessing a hangout won't be needed but will certainly let you know if I'm wrong | |
| 14:15:36 | sean-k-mooney | aspiers[m]: ok just read your responces. if we can just document what will work and what wont then im more or less happy. | |
| 14:16:11 | sean-k-mooney | aspiers[m]: im also fine with an fix constatin + vm ram + video ram | |
| 14:16:19 | aspiers[m] | sean-k-mooney: absolutely - next patch set should give much more clarity over what is expected to work vs. not | |
| 14:16:25 | sean-k-mooney | but if we go that way i would make the constant a config value | |
| 14:16:56 | aspiers[m] | OK that sounds like a good idea as a short-term workaround until qemu exposes that calculation | |
| 14:17:42 | aspiers[m] | so you mean a fixed configurable value for the overhead per VM, right? so <hard_limit> would be calculated as flavor ram plus that value | |
| 14:17:54 | sean-k-mooney | the issue is qemu cant really know this up front as it depends on several things that can cahgne at runtime and from version to version | |
| 14:18:18 | aspiers[m] | possibly with an adjustment if guest video ram is specified via extra specs or image props | |
| 14:18:43 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 14:18:52 | sean-k-mooney | aspiers[m]: ya it could be a config value for the over head or a flavor extraspec | |
| 14:19:29 | sean-k-mooney | just some way to reduce or increase the 512 without code change if we mesuer it is too big or two small later | |
| 14:19:56 | sean-k-mooney | so it would be ram + video ram + a configurable constat overhead | |
| 14:20:40 | sean-k-mooney | that said i think we do this somewhere already | |
| 14:23:44 | sean-k-mooney | aspiers[m]: we support setting the hardlimit in https://github.com/openstack/nova/blob/88951ca98e1b286b58aa1ad94f9af40b8260c01f/nova/virt/libvirt/config.py#L2102-L2133 and i belive we use it for realtime instances | |
| 14:25:00 | aspiers[m] | sean-k-mooney: ah yeah, I guess probably for the same reason (memory pinning) | |
| 14:25:57 | sean-k-mooney | aspiers[m]: yes we use it so we can memlock the memory for realtime instaces. also all hugepage backed instace will also have the hugepages memlocked which we use for ovs-dpdk so it can do dma into the guest memory | |
| 14:29:32 | sean-k-mooney | im trying not to be angry about the fact that apparently we only use this in unit tests.... http://codesearch.openstack.org/?q=hard_limit&i=nope&files=&repos=nova | |
| 14:30:05 | aspiers[m] | haha :) | |
| 14:31:13 | sean-k-mooney | aspiers[m]: well the code is there for you to use... | |
| 14:34:37 | aspiers[m] | indeed. I don't think the actual coding is gonna be too hard | |
| 15:21:53 | mriedem | dansmith: so belmiro gave me another issue to worry about, for which he's going to open a bug. they drop the allocation candidates limit down to 10 to make scheduling faster, but they also at times disable a large number of computes in a cell for $reasons (mnaser also does that for load balancing the cloud); the problem then is b/c of the low alloc candidates limit, placement gives back 10 disabled computes and gets novalidhost | |
| 15:22:08 | mriedem | to workaround it he set reserved = total for inventory in placement on the disabled nodes, | |
| 15:22:12 | mriedem | but that sucks for hundreds of vms | |
| 15:22:20 | dansmith | ouch | |
| 15:22:24 | mriedem | i suggested a pre-request placement filter to somehow filter out disabled nodes | |
| 15:22:40 | mriedem | maybe forbidden trait/aggregate, idk | |
| 15:22:48 | dansmith | hmm | |
| 15:22:59 | dansmith | they disable compute or disable a cell? | |
| 15:23:04 | dansmith | *computes | |
| 15:23:08 | sean-k-mooney | mriedem: i was going to say yes tag with a trait and use forbindin traits but then we are using placmeent to sotre metadata | |
| 15:24:34 | dansmith | mriedem: I think the right thing to do is to make placement results better so they don't need such a tiny result set to mitigate performance issues | |
| 15:24:42 | dansmith | but, regardless of what that number is, the same problem could happen | |
| 15:25:02 | dansmith | so if it's a disabled compute situation, I think having compute decorate its RP in some way is probably the thing to do there, | |
| 15:25:15 | dansmith | although it generates a lot of chatty traffic from api->compute->placement | |
| 15:25:17 | sean-k-mooney | stephenfin: bauzas by the way is there a nova cyborg session at the ptg? they demoed something that does not work upstream in the key note. im reviewing the spec again but it would be good to check in with them. | |
| 15:25:44 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 15:28:05 | mriedem | i would really want to avoid needing to make placement api calls from the pre-placement filter for sure | |